"Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents
The DOI matches its Crossref record; 49 of the 51 references it cites were found in a registry or an index.
Abstract
The past four years have seen the rise of conversational agents (CAs) in everyday life. Apple, Microsoft, Amazon, Google and Facebook have all embedded proprietary CAs within their software and, increasingly, conversation is becoming a key mode of human-computer interaction. Whilst we have long been familiar with the notion of computers that speak, the investigative concern within HCI has been upon multimodality rather than dialogue alone, and there is no sense of how such interfaces are used in everyday life. This paper reports the findings of interviews with 14 users of CAs in an effort to understand the current interactional factors affecting everyday use. We find user expectations dramatically out of step with the operation of the systems, particularly in terms of known machine intelligence, system capability and goals. Using Norman's 'gulfs of execution and evaluation' [30] we consider the implications of these findings for the design of future systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Whereas, we found that the challenges our participants experienced were more elementary.”
From Exploring How Older Adults Use a Smart Speaker–Based Voice Assistant in Their First Interactions: Qualitative Study (2021) · cites this paper as [32]
In the citing paper
“In addition, it could be more important for novice users to have an appropriate mental model and understand the system rather than an efficient user experience [32].”
From Exploring How Older Adults Use a Smart Speaker–Based Voice Assistant in Their First Interactions: Qualitative Study (2021) · cites this paper as [32]
In the citing paper
“human-AI mixed-initiative presents a generally valuable and underexplored interface paradigm.”
From Mixed-Initiative Creative Interfaces (2017) · cites this paper as [20]
In the citing paper
“Not only do these breakdowns significantly diminish user satisfaction and damage trust, they also decrease the likelihood of continued use [18, 44].”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [44]
In the citing paper
“This is because when voice assistants fail, such as by incorrectly answering a question, it derails user trust [27, 35].”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [35]
17 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
How Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808
(2010) · Sociology for Pharmacists · doi:10.4324/9780203381175_chapter_9 · Cited by 14,152
Norman, D. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,374
Thematic networks: an analytic tool for qualitative research
Attride-Stirling, J. (2001) · Qualitative Research · doi:10.1177/146879410100100307 · Cited by 5,506 · cited 2×
“The thematic network is a ‘web-like coding structure’ [2; 385] that allows for a rich understanding of the conceptual interplay occurring within the narrative.”
Background(1997) · Choice Reviews Online · doi:10.5860/choice.34-3702 · Cited by 4,887
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Licklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938 · cited 1×
“When considering human-machine dialogue from an HCI perspective, some of the earliest work can be traced back to vision statements and position papers such as JCR Licklider (1960), ‘Man-machine symbiosis’ [25] and the early work of Richard Bolt [4].”
MethodEmbodied Conversational Agents
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/2697.001.0001 · Cited by 1,799
Interaction Ritual: Essays on Face-to-face Behaviour
Goffman, E. (1972) · Medical Entomology and Zoology · Cited by 1,449
Research works printed but not linked: 14
- Boden Margaret A, 2007, Proceedings of Artificial Companions in Society: Perspectives on the Present and Future. Oxford Internet Institute.
- Davis Martin, Engines of Logic: Mathematicians and the Origin of the Computer
- Glass James R., 1999, Challenges for Spoken Dialogue Systems. In the proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding. (ASRU), CO, USA
- 10.5555/1941863 · doi:10.5555/1941863
- Heuwinkel Kerstin, Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Robnett Licklider Joseph Carl, 1960, HFE-1,(March
- Moore Roger, Spoken Language Processing: Where do we go from Here? In Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Payr Sabine, Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Preece Jenny, 2015, Interaction Design: Beyond Huma- Computer Interaction
- Pulman Stephen, Proceedings of the 2010 Workshop on Companionable Dialogue Systems. Assoc. for Computational Linguistics
- Trappl Robert, From Jeeves Jeannie to Siri, and Then? In Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Vassallo Giorgio, Semantic Computing. Sheu, Yu, Ramamoorthy, Joshi & Zadeh.(eds)
- 10.5555/1870559.1870562 · doi:10.5555/1870559.1870562
- Wilks Yorick, 2010, Amsterdam: John Benjamins.
Cited by
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
AI-based chatbots in customer service and their effects on user compliance
Adam, M., Wessel, M., & Benlian, A. (2021) · Electronic Markets · doi:10.1007/s12525-020-00414-7 · Cited by 1,195
“CAs may, for instance, provide unsuitable responses to the user requests, leading to a gap between the user ’ s expectation and the system ’ s performance (Luger and Sellen 2016; Orlowski 2017).”
BackgroundRapp, A., Curti, L., & Boldi, A. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102630 · Cited by 643
Voice Interfaces in Everyday Life
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174214 · Cited by 717
“Luger and Sellen [17], through interviews, problematise the limited functionality of existing commercially-available VUIs, stipulating that a “gulf” exists between expectant and actual capabilities of the systems.”
BackgroundInfluentialChen, Y., Jensen, S., Albert, L. J., Gupta, S., & Lee, T. (2023) · Information Systems Frontiers · doi:10.1007/s10796-022-10291-4 · Cited by 685
“As one type of AI technology, conversational agents (Luger & Sellen, 2016) such as chatbots (Schlesinger et al., 2018; Stieglitz et al., 2021), have been leveraged to assist student life in education (Goel & Polepeddi, 2016; Hwang & Chang, 2021; Kerly et al., 2007; Kerry et al., 2009; Okonkwo &…”
BackgroundChaves, A. P., & Gerosa, M. A. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1841438 · Cited by 572
“…argues that humans socially respond to computers as they respond to other people [Reeves and Nass 1996], the literature has shown that interactions with conversational agents are not quite equal to human-human interactions [Luger and Sellen 2016; Mou and Xu 2017; Shechtman and Horowitz 2003].”
BackgroundInfluentialAlmost human? A comparative case study on the social media presence of virtual influencers
Arsenyan, J., & Mirowska, A. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102694 · Cited by 566
Cooperative Heterogeneous Multi-Robot Systems: A Survey
Rizk, Y., Awad, M., & Tunstel, E. W. (2020) · ACM Computing Surveys · doi:10.1145/3303848 · Cited by 449
“…seen significant strides in recent years with the emergence of deep learning and big data, embodied by systems like Siri, Amazon Alexa, and others [129], accuracy, computational complexity, and responsiveness are still issues that need improvements before incorporating them into more complex…”
Background"Alexa is my new BFF": Social Roles, User Satisfaction, and Personification of the Amazon Echo
Purington, A., Taft, J. G., Sannon, S., Bazarova, N. N., & Taylor, S. H. (2017) · Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/3027063.3053246 · Cited by 476
“voice recognition technology and intelligent responses [9,16].”
BackgroundWhat Makes a Good Conversation?: Challenges in Designing Truly Conversational Agents
Clark, L., Pantidi, N., Cooney, O., Doyle, P., Garaialde, D., Edwards, J., Spillane, B., Gilmartin, E., Murad, C., Munteanu, C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300705 · Cited by 461
“hat conversational agents can and should be capable of performing. These expectations support the notion that an agent is a basic dialogue partner [8] lacking in humanlike conversational capabilities [20, 35, 42]. Our participants personally expressed no desire to build bonds with conversational agents (although they…”
BackgroundResultInfluentialPradhan, A., Mehta, K., & Findlater, L. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174033 · Cited by 448
How perceptions of intelligence and anthropomorphism affect adoption of personal intelligent agents
Moussawi, S., Koufaris, M., & Benbunan-Fich, R. (2021) · Electronic Markets · doi:10.1007/s12525-020-00411-w · Cited by 406
“…and satisfaction with their PIAs (Purington et al., 2017; Sannon et al., 2018; Lopatovska and Williams, 2018), experience with and expectations of PIAs (Luger and Sellen, 2016), as well as concerns around privacy (Zeng et al., 2017) and users ’ affective experiences with PIAs (Yang et al., 2019).”
BackgroundExplanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
What Makes Users Trust a Chatbot for Customer Service? An Exploratory Interview Study
Følstad, A., Nordheim, C. B., & Bjørkli, C. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01437-7_16 · Cited by 349
“Failure to do so, may lead the user to believe that the chatbot is capable of more than it actually is, which in turn may lead to frustration and reduced willingness to continue using chatbots for this purpose [14].”
BackgroundUX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025739 · Cited by 391
“Intelligent systems sometimes encourage unrealistic expectations, leading to some reluctance to use them in complex or sensitive contexts [38,55].”
BackgroundEvaluating and Informing the Design of Chatbots
Jain, M., Kumar, P., Kota, R., & Patel, S. N. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196735 · Cited by 376
Systematic Review: Trust-Building Factors and Implications for Conversational Agent Design
Rheu, M., Shin, J. Y., Peng, W., & Huh-Yoo, J. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1807710 · Cited by 360
Music, Search, and IoT: How People (Really) Use Voice Assistants
Ammari, T., Kaye, J., Tsai, J. Y., & Bentley, F. (2019) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3311956 · Cited by 356
"What can i help you with?": infrequent users' experiences of intelligent personal assistants
Cowan, B. R., Pantidi, N., Coyle, D., Morrissey, K., Clarke, P., Al-Shehri, S., Earley, D., & Bandeira, N. (2017) · Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services · doi:10.1145/3098279.3098539 · Cited by 351
“More recently Luger and Sellen [34] provided a comprehensive study of the experiences of people who use IPAs on a frequent basis (almost daily).”
BackgroundMethodInfluentialAre we preparing for a good AI society? A bibliometric review and research agenda
Fosso Wamba, S., Bawack, R. E., Guthrie, C., Queiroz, M. M., & Carillo, K. D. A. (2021) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2020.120482 · Cited by 275
Users' experiences with chatbots: findings from a questionnaire study
Følstad, A., & Brandtzaeg, P. B. (2020) · Quality and User Experience · doi:10.1007/s41233-020-00033-2 · Cited by 252
“Luger and Sellen [37] investigated user experiences with voice-based chat-bots and found that it was challenging for users to properly understand chatbot capabilities and to interact efficiently with chatbots; they suggested that play might be a useful entry-point for learning about chatbot…”
BackgroundChatbots: changing user needs and motivations
Brandtzaeg, P. B., & Følstad, A. (2018) · Interactions · doi:10.1145/3236669 · Cited by 322
Understanding the Long-Term Use of Smart Speaker Assistants
Bentley, F., Luvogt, C., Silverman, M., Wirasinghe, R., White, B., & Lottridge, D. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3264901 · Cited by 316
“Luger and Sellen [19] explored the use of phone-based smart assistants and found them to be “like a really bad PA [personal assistant].””
BackgroundCommunication Breakdowns Between Families and Alexa
Beneteau, E., Richards, O. K., Zhang, M., Kientz, J. A., Yip, J., & Hiniker, A. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300473 · Cited by 313
“Despite the “conversational” interface with conversational agents, people are not yet able to talk to technology in the same way that they talk to other people [23,34] .”
BackgroundInfluentialTalk to me: Exploring user interactions with the Amazon Alexa
Lopatovska, I., Rink, K., Knight, I., Raines, K., Cosenza, K., Williams, H., Sorsche, P., Hirsch, D., Li, Q., & Martinez, A. (2019) · Journal of Librarianship and Information Science · doi:10.1177/0961000618759414 · Cited by 298
“In trying to understand everyday uses of the IPA, Luger and Sellen (2016) interviewed 14 participants about their usage patterns, motivations and expectations of the IPAs.”
BackgroundMethodResultChatbots for customer service: user experience and motivation
Følstad, A., & Skjuve, M. (2019) · Proceedings of the 1st International Conference on Conversational User Interfaces · doi:10.1145/3342775.3342784 · Cited by 207
“While the existing literature suggests that users are often poor at assessing the capabilities of virtual agents [26, 38], our findings suggest that users of chatbots for customer service hold reasonably realistic expectations regarding the capabilities of such chatbots.”
BackgroundChatbots, Humbots, and the Quest for Artificial General Intelligence
Grudin, J., & Jacques, R. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300439 · Cited by 275
Mirbabaie, M., Stieglitz, S., & Frick, N. R. J. (2021) · Health and Technology · doi:10.1007/s12553-021-00555-5 · Cited by 241
“The importance of AI as a component of the diagnostic process has been steadily increasing since building systems became more practical [12, 13].”
BackgroundMethodMorris, R. R., Kouddous, K., Kshirsagar, R., & Schueller, S. M. (2018) · Journal of Medical Internet Research · doi:10.2196/10148 · Cited by 273
“Conversational agents are software applications that respond to users with natural language, often with the goal of helping a user complete a task [1].”
BackgroundEffectiveness of an Empathic Chatbot in Combating Adverse Effects of Social Exclusion on Mood
de Gennaro, M., Krumhuber, E. G., & Lucas, G. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.03061 · Cited by 265
Intelligent personal assistants: A systematic literature review
de Barcelos Silva, A., Gomes, M. M., da Costa, C. A., da Rosa Righi, R., Barbosa, J. L. V., Pessin, G., De Doncker, G., & Federizzi, G. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113193 · Cited by 230
Future directions for chatbot research: An interdisciplinary research agenda
Følstad, A., Araujo, T., Law, E. L.-C., Brandtzaeg, P. B., Papadopoulos, S., Reis, L., Baez, M., Laban, G., McAllister, P., Ischen, C., et al. (2021) · Computing · doi:10.1007/s00607-021-01016-7 · Cited by 247
Self-citationXiao, Z., Zhou, M. X., Liao, Q. V., Mark, G., Chi, C., Chen, W., & Yang, H. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3381804 · Cited by 185
“The general consensus is that conversational interfaces offer several advantages over traditional WIMP (Windows, Icons, Menus, and Pointers) interfaces [14, 56].”
BackgroundAn Initial Model of Trust in Chatbots for Customer Service—Findings from a Questionnaire Study
Nordheim, C. B., Følstad, A., & Bjørkli, C. A. (2019) · Interacting with Computers · doi:10.1093/iwc/iwz022 · Cited by 233
“Rather, users may experience chatbots that misinterpret their requests, that provide partial answers to their requests, and that are presented in a style that is not sufficiently adapted to the dialogue context (Luger & Sellen, 2016).”
BackgroundHow to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies
Vereschak, O., Bailly, G., & Caramiaux, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476068 · Cited by 229
Smart Home Personal Assistants: A Security and Privacy Review
Edu, J. S., Such, J. M., & Suarez-Tangil, G. (2021) · ACM Computing Surveys · doi:10.1145/3412383 · Cited by 189
“They enable a playful interaction, making their use more engaging [74].”
BackgroundZhang (2021) · doi:10.1145/3432945 · Cited by 225
“The most ideal communication approach is via natural language processing, which has been studied for decades and applied to conversational agents in CSCW [41].”
BackgroundResilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns
Ashktorab, Z., Jain, M., Liao, Q. V., & Weisz, J. D. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300484 · Cited by 224
Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good
Wang, X., Shi, W., Kim, R., Oh, Y., Yang, S., Zhang, J., & Yu, Z. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1566 · Cited by 222
“The primary goal of these agents is to facilitate taskcompletion and human-engagement in practical contexts (Luger and Sellen, 2016; Bickmore et al., 2016; Graesser et al., 2014; Yu et al., 2016b).”
BackgroundKim, S., Lee, J., & Gweon, G. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300316 · Cited by 190
Patterns for How Users Overcome Obstacles in Voice User Interfaces
Myers, C., Furqan, A., Nebolsky, J., Caro, K., & Zhu, J. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173580 · Cited by 213
“In addition to the error rates of natural language processing (NLP), users are struggling to make correct mental models [8, 3, 14].”
BackgroundMethodInfluentialTowards Designing Cooperative and Social Conversational Agents for Customer Service
Gnewuch, U., Morana, S., & Mädche, A. (2017) · AIS Electronic Library (AISeL) · Cited by 372
Kim, S., & Choudhury, A. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2021.106914 · Cited by 200
“At the same time, the most common and critical challenge our participants faced throughout the study was its limited conversational capabilities, especially when the device failed to understand user commands, which echoes prior work (Luger & Sellen, 2016).”
BackgroundThe State of Speech in HCI: Trends, Themes and Challenges
Clark, L., Doyle, P., Garaialde, D., Gilmartin, E., Schlögl, S., Edlund, J., Aylett, M., Cabral, J., Munteanu, C., Edwards, J., et al. (2019) · Interacting with Computers · doi:10.1093/iwc/iwz016 · Cited by 193
“Our review indicates that there has been some recent progress towards this (e.g. Luger & Sellen, 2016; Porcheron et al., 2017; Cowan et al., 2017).”
BackgroundResultInfluentialConceptual Metaphors Impact Perceptions of Human-AI Collaboration
Khadpe, P., Krishna, R., Fei-Fei, L., Hancock, J. T., & Bernstein, M. S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415234 · Cited by 183
“viously, interviews have established that expectations from conversational agents such as Siri, Google Assistant, and Alexa are out of sync with the actual capabilities and performance of the systems [49, 83]. So, after repeatedly hitting the agent’s capability limits, users retreat to using the agents only for menial…”
BackgroundInfluentialAt Your Service: Designing Voice Assistant Personalities to Improve Automotive User Interfaces
Braun, M., Mainz, A., Chadowitz, R., Pfleging, B., & Alt, F. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300270 · Cited by 182
The acceptance of chatbots in an enterprise context – A survey study
Brachten, F., Kißmer, T., & Stieglitz, S. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2021.102375 · Cited by 201
The User Experience of ChatGPT: Findings from a Questionnaire Study of Early Users
Skjuve, M., Følstad, A., & Brandtzaeg, P. B. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3597144 · Cited by 135
“Likewise, Luger and Sellen [21], in an interview study with voice agent users, found them to be frustrated when agents could not perform requested tasks and showed limitations in natural language understanding.”
BackgroundResultTrusting Virtual Agents: The Effect of Personality
Zhou, M. X., Mark, G., Li, J., & Yang, H. (2019) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3232077 · Cited by 176
“…technical advances have been made to support human-computer interactions in richer modalities such as voice and gestures (e.g., see Refs [22, 75]), the technologies including those used in commercial systems like Apple Siri, Amazon Alexa, and Google Now are far from being perfect [25, 60, 93, 102].”
BackgroundZhang, Z., Xu, Y., Wang, Y., Yao, B., Ritchie, D., Wu, T., Yu, M., Wang, D., & Li, T. J.-J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517479 · Cited by 174
Rzepka, C., Berger, B., & Hess, T. (2022) · Information Systems Frontiers · doi:10.1007/s10796-021-10226-5 · Cited by 146
“These findings are in line with qualitative research on the determinants of voice assistant use (Luger & Sellen, 2016; Moussawi, 2018; Rzepka, 2019).”
Abdolrahmani, A., Kuber, R., & Branham, S. M. (2018) · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3234695.3236344 · Cited by 172
Skjuve, M., Haugstveit, I. M., Følstad, A., & Brandtzaeg, P. B. (2019) · Human Technology · doi:10.17011/ht/urn.201902201607 · Cited by 138
“For instance, Luger and Sellen (2016), in an interview study of Siri, Cortana, and Google Now users, identified substantial mismatch between user expectations and system capabilities and, consequently, argued for the need for conversational systems to be open about their limitations.”
BackgroundAnthropomorphic Information Systems
Pfeuffer, N., Benlian, A., Gimpel, H., & Hinz, O. F. (2019) · Business & Information Systems Engineering · doi:10.1007/s12599-019-00599-y · Cited by 184
“User assistant/personal butler (UAS) systems, such as Microsoft’s Cortana, Apple’s Siri and Google’s Google Now have become increasingly important in the everyday life of smartphone users (Luger and Sellen 2016).”
BackgroundMethodInfluentialPersonification of the Amazon Alexa: BFF or a Mindless Companion
Lopatovska, I., & Williams, H. (2018) · Proceedings of the 2018 Conference on Human Information Interaction&Retrieval - CHIIR '18 · doi:10.1145/3176349.3176868 · Cited by 160
“Affective expressions towards Alexa might also signal users’ tendencies to test the system that they do not fully understand and that “fools” them to believe it is more “human” than it actually is [19].”
BackgroundLet's Talk About Race: Identity, Chatbots, and AI
Schlesinger, A., O'Hara, K. P., & Taylor, A. S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173889 · Cited by 138
“There have been a range of studies involving conversational agents presented at CHI that have artificial agents involved in nursing, educational settings, activism, and conflict resolution [58,66,68,80,81].”
"Alexa is a Toy": Exploring Older Adults' Reasons for Using, Limiting, and Abandoning Echo
Trajkova, M., & Martin-Hammond, A. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376760 · Cited by 154
Kim, S. (2021) · JMIR mHealth and uHealth · doi:10.2196/20427 · Cited by 153
“In addition, it could be more important for novice users to have an appropriate mental model and understand the system rather than an efficient user experience [32].”
He Is Just Like Me: A Study of the Long-Term Use of Smart Speakers by Parents and Children
Garg, R., & Sengupta, S. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3381002 · Cited by 138
Design Ideation with AI - Sketching, Thinking and Talking with Generative Machine Learning Models
Tholander, J., & Jonsson, M. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3596014 · Cited by 148
Alexa, Can I Trust You? Exploring Consumer Paths to Trust in Smart Voice-Interaction Technologies
Foehr, J., & Germelmann, C. C. (2020) · Journal of the Association for Consumer Research · doi:10.1086/707731 · Cited by 137
“…that conceptualize SVITs mainly with reference to their underlying software components or computational infrastructure — for example, natural language processing, AI, or cloud computing (e.g., Luger and Sellen 2016; Cho 2018; De Keyser et al. 2018; Lopatovska and Williams 2018; Myers et al. 2018).”
BackgroundWhat Makes a Good Conversation? Challenges in Designing Truly Conversational Agents
Clark, L., Pantidi, N., Cooney, O., Doyle, P. R., Garaialde, D., Edwards, J., Spillane, B., Murad, C., Munteanu, C., Wade, V., et al. (2019) · Cork Open Research Archive (University College Cork, Ireland) · doi:10.48550/arxiv.1901.06525 · Cited by 128
“hat conversational agents can and should be capable of performing. These expectations support the notion that an agent is a basic dialogue partner [8] lacking in humanlike conversational capabilities [20, 35, 42]. Our participants personally expressed no desire to build bonds with conversational agents (although they…”
BackgroundResultInfluentialFace Value? Exploring the Effects of Embodiment for a Group Facilitation Agent
Shamekhi, A., Liao, Q. V., Wang, D., Bellamy, R. K. E., & Erickson, T. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173965 · Cited by 124
Designing Anthropomorphic Enterprise Conversational Agents
Diederich, S., Brendel, A. B., & Kolbe, L. M. (2020) · Business & Information Systems Engineering · doi:10.1007/s12599-020-00639-y · Cited by 127
“…the human-likeness of an agent increasingly poses the risk of disappointing users and fostering feelings of uncanniness (Ben Mimoun et al. 2012; Luger and Sellen 2016), in particular when the CA does not provide meaningful responses due to its limited conversational capabilities and is thus…”
BackgroundMethodInfluentialDesigning for Workplace Reflection: A Chat and Voice-Based Conversational Agent
Kocielnik, R., Avrahami, D., Marlow, J., Lu, D., & Hsieh, G. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196784 · Cited by 119
Moussawi, S., Koufaris, M., & Benbunan-Fich, R. (2023) · European Journal of Information Systems · doi:10.1080/0960085x.2021.2018365 · Cited by 125
Cambre, J., & Kulkarni, C. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359325 · Cited by 119
“Voice assistants currently take a largely “one size fits all” approach in which each instance of a given device takes on the same voice by default; indeed, the same voice is often used across devices powered by the same company’s assistant software (e.g. Alexa-enabled or Google Assistant-enabled devices).”
BackgroundInfluentialA Review of Recent Deep Learning Approaches in Human-Centered Machine Learning
Kaluarachchi, T., Reis, A., & Nanayakkara, S. (2021) · Sensors · doi:10.3390/s21072514 · Cited by 124
“[51] conducted a set of interviews to understand people’s perception of voice assistants from tech giants.”
BackgroundMatthews, G., Hancock, P. A., Lin, J., Panganiban, A. R., Reinerman-Jones, L. E., Szalma, J. L., & Wohleber, R. W. (2021) · Personality and Individual Differences · doi:10.1016/j.paid.2020.109969 · Cited by 118
Spoken language interaction with robots: Recommendations for future research
Marge, M., Espy-Wilson, C., Ward, N., Alwan, A. A., Artzi, Y., Bansal, M., Blankenship, G., Chai, J. Y., Daumé, H., Dey, D., et al. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101255 · Cited by 172
“Of course there can be tradeoffs between attracting users to engage in the first place and enabling truly effective interaction [27].”
Background(2017) · doi:10.1145/2998181.2998298 · Cited by 131
“Others such as Luger and Sellen [26], however, paint a more challenging picture.”
BackgroundInfluentialSchöbel, S., Schmitt, A., Benner, D., Saqr, M., Janson, A., & Leimeister, J. M. (2024) · Information Systems Frontiers · doi:10.1007/s10796-023-10375-9 · Cited by 127
Understanding the Role of Privacy and Trust in Intelligent Personal Assistant Adoption
Liao, Y., Vitak, J., Kumar, P., Zimmer, M., & Kritikos, K. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-15742-5_9 · Cited by 127
Privacy and smart speakers: A multi-dimensional approach
Lutz, C., & Newlands, G. (2021) · The Information Society · doi:10.1080/01972243.2021.1897914 · Cited by 123
“…the fields of computer science and human-computer interaction (Feng, Fawaz, and Shin 2017; Geeng and Roesner 2019; Lau, Zimmerman, and Schaub 2018; Luger and Sellen 2016; Malkin et al. 2019; Zheng et al. 2018), with only limited discussion in the social sciences (Brause and Blank 2020; Pridmore…”
BackgroundExploring Conversational Search With Humans, Assistants, and Wizards
Vtyurina, A., Savenkov, D., Agichtein, E., & Clarke, C. L. A. (2017) · Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/3027063.3053175 · Cited by 117
“The most closely related work was done by Luger and Sellen [10], where 14 people were interviewed about their experience with an intelligent assistant that they use in their daily life.”
BackgroundFølstad, A., & Taylor, C. (2021) · Quality and User Experience · doi:10.1007/s41233-021-00046-5 · Cited by 97
Wang, Q., Saha, K., Gregori, E., Joyner, D., & Goel, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445645 · Cited by 120
“It is also well-established in the literature that people often have unrealistically high expectations for CAs [61, 98].”
BackgroundInfluentialLiu, M. X., Sarkar, A., Negreanu, C., Zorn, B., Williams, J., Toronto, N., & Gordon, A. D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580817 · Cited by 119
Mixed-Initiative Creative Interfaces
Deterding, S., Hook, J., Fiebrink, R., Gillies, M., Gow, J., Akten, M., Smith, G., Liapis, A., & Compton, K. (2017) · Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/3027063.3027072 · Cited by 117
“Indeed, with the current rise of AI, (semi-)autonomous systems, and conversational agents [19,20], human-AI mixed-initiative presents a generally valuable and underexplored interface paradigm.”
BackgroundFarmChat: A Conversational Agent to Answer Farmer Queries
Jain, M., Kumar, P., Bhansali, I., Liao, Q. V., Truong, K., & Patel, S. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3287048 · Cited by 117
Liao, Q. V., Mas-ud Hussain, M., Chandar, P., Davis, M., Khazaeni, Y., Crasso, M. P., Wang, D., Muller, M., Shami, N. S., & Geyer, W. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173577 · Cited by 116
“Despite the popularity of commercial chatbots (e.g. Siri), there is little empirical report of user interaction patterns except for a few qualitative studies [28, 35].”
BackgroundInfluentialSasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models
King, E., Yu, H., Lee, S., & Julien, C. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3643505 · Cited by 153
“Prior work illustrates common shortcomings of assistant devices that users might describe their goals to: namely, they struggle to operate outside of a predictable, rigid command structure [22, 34, 39] and are generally unable to deal with complex commands [50, 63].”
BackgroundChatbot ads with a human touch: A test of anthropomorphism, interactivity, and narrativity
Sun, Y., Chen, J., & Sundar, S. S. (2024) · Journal of Business Research · doi:10.1016/j.jbusres.2023.114403 · Cited by 115
How Do You Want Your Chatbot? An Exploratory Wizard-of-Oz Study with Young, Urban Indians
Medhi Thies, I., Menon, N., Magapu, S., Subramony, M., & O’Neill, J. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-67744-6_28 · Cited by 104
“Finally there is recent work that studied what users want from their conversational agents (like Siri, Cortana and Google Now), and found that user expectations are dramatically out of step with the operation of agents [28].”
Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
Ehsan, U., Saha, K., De Choudhury, M., & Riedl, M. O. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579467 · Cited by 85
Izadi, S., & Forouzanfar, M. (2024) · AI · doi:10.3390/ai5020041 · Cited by 140
“When chat-bots fail to understand user intent, provide inappropriate responses, or deliver inaccurate information, users experience frustration and dissatisfaction [67].”
BackgroundZhan, E. S., Molina, M. D., Rheu, M., & Peng, W. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2261731 · Cited by 92
“The above-described fear of AI may threaten users’ autonomy (G € ornemann & Spiekermann, 2022) and reduce users’ satisfaction with AI systems (Luger & Sellen, 2016), which potentially hinders individuals by making choices based on fear toward these technologies rather than logical reasoning (e.g.,…”
Fact-checking information from large language models can decrease headline discernment
DeVerna, M., Yan, H. Y., Yang, K.-C., & Menczer, F. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2322823121 · Cited by 146
Metatla, O., Oldfield, A., Ahmed, T., Vafeas, A., & Miglani, S. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300608 · Cited by 112
“For example, recent research uncovered a mismatch between users expectation and experience with VUIs in terms of system capability and goals [22], and highlighted the importance of accounting for perceived personality of voice agents [9].”
BackgroundShaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models
Dhillon, P. S., Molaei, S., Li, J., Golub, M., Zheng, S., & Robert, L. P. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642134 · Cited by 111
“Text quality slightly increased by 0.37 as participants progressed through conditions (p=0.01), suggesting familiarity with the interface improves performance [42].”
BackgroundInfluentialUnderstanding Affective Experiences with Conversational Agents
Yang, X., Aurisicchio, M., & Baxter, W. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300772 · Cited by 95
“Consistently with previous findings [10, 33], the dominant scenario was shown to involve basic information requests such as weather updates or news.”
BackgroundResultInfluentialThe Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems
Ziems, C., Yu, J., Wang, Y.-C., Halevy, A., & Yang, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.261 · Cited by 58
AIoT Smart Home via Autonomous LLM Agents
Rivkin, D., Hogan, F. R., Feriani, A., Konar, A., Sigal, A., Liu, X., & Dudek, G. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3471904 · Cited by 142
“However, these modern home assistants usually struggle with implicit and complex commands [16].”
Multi-Modal Repairs of Conversational Breakdowns in Task-Oriented Dialogs
Li, T. J.-J., Chen, J., Xia, H., Mitchell, T. M., & Myers, B. A. (2020) · Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3379337.3415820 · Cited by 98
“Conversational breakdowns decrease users’ satisfaction, trust, and willingness to continue using a conversational sys tem [6, 16, 25, 26, 44], and may cause users to abandon the current task [3].”
BackgroundHornung, O., & Smolnik, S. (2022) · Electronic Markets · doi:10.1007/s12525-021-00493-0 · Cited by 96
“Dissat-isfaction and frustration can only be avoided by ensuring that the (future) implementation of organizational PVAs are seamlessly integrated and work with very few errors (Luger & Sellen, 2016).”
BackgroundRheu, M., Dai, Y. H., Meng, J., & Peng, W. (2024) · Communication Research · doi:10.1177/00936502231221669 · Cited by 110
“For example, Luger and Sellen (2016) reported that users of conversational agents (CAs), such as Alexa, often encounter situations where their expectations for machine intelligence do not align with the actual capability of the CAs.”
Designing Conversational Agents: A Self-Determination Theory Approach
Yang, X., & Aurisicchio, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445445 · Cited by 81
“in [17, 29, 37, 54, 85]) which could limit the effective application of these findings in design practice (e.”
BackgroundResultInfluentialMapping Perceptions of Humanness in Intelligent Personal Assistant Interaction
Doyle, P. R., Edwards, J., Dumbleton, O., Clark, L., & Cowan, B. R. (2019) · Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services · doi:10.1145/3338286.3340116 · Cited by 100
“This humanness seems to give users expectations as to the type of capabilities that a system may have as a dialogue partner [36, 38].”
BackgroundInfluentialOn the ability of virtual agents to decrease cognitive load: an experimental study
Brachten, F., Brünker, F., Frick, N. R. J., Ross, B., & Stieglitz, S. (2020) · Information Systems and e-Business Management · doi:10.1007/s10257-020-00471-7 · Cited by 75
Toward a design theory for virtual companionship
Strohmann, T., Siemon, D., Khosrawi-Rad, B., & Robra-Bissantz, S. (2023) · Human–Computer Interaction · doi:10.1080/07370024.2022.2084620 · Cited by 80
“Research suggests that such negative perceptions of CAs are due to a mismatch between the mental model and user expectations (Luger & Sellen, 2016).”
Background
Metadata
- Authors
- Luger, E., & Sellen, A.
- Affiliations
- Microsoft Research (United Kingdom) (GB)
- Venue
- Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2016-05-07
- Pages
- 5286-5297
- OpenAlex
- W2405187948
- Semantic Scholar
- 814e62bb4f0ec6facd640fbd3e3fcbd189716d20
- Cited by
- 1,176
- References
- 51
- Citations by year
- 2026: 164 · 2025: 175 · 2024: 119 · 2023: 116 · 2022: 107 · 2021: 148 · 2020: 141 · 2019: 126 · 2018: 54 · 2017: 25 · 2016: 1
- FWCI
- 37.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- chatbots, voice assistants, everyday use, user expectations, human-computer interaction, multimodality, system capabilities, interface design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.