"Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents
The DOI matches its Crossref record; 49 of the 51 references it cites were found in a registry or an index.
Abstract
The past four years have seen the rise of conversational agents (CAs) in everyday life. Apple, Microsoft, Amazon, Google and Facebook have all embedded proprietary CAs within their software and, increasingly, conversation is becoming a key mode of human-computer interaction. Whilst we have long been familiar with the notion of computers that speak, the investigative concern within HCI has been upon multimodality rather than dialogue alone, and there is no sense of how such interfaces are used in everyday life. This paper reports the findings of interviews with 14 users of CAs in an effort to understand the current interactional factors affecting everyday use. We find user expectations dramatically out of step with the operation of the systems, particularly in terms of known machine intelligence, system capability and goals. Using Norman's 'gulfs of execution and evaluation' [30] we consider the implications of these findings for the design of future systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Whereas, we found that the challenges our participants experienced were more elementary.”
From Exploring How Older Adults Use a Smart Speaker–Based Voice Assistant in Their First Interactions: Qualitative Study (2021) · cites this paper as [32]
In the citing paper
“In addition, it could be more important for novice users to have an appropriate mental model and understand the system rather than an efficient user experience [32].”
From Exploring How Older Adults Use a Smart Speaker–Based Voice Assistant in Their First Interactions: Qualitative Study (2021) · cites this paper as [32]
In the citing paper
“human-AI mixed-initiative presents a generally valuable and underexplored interface paradigm.”
From Mixed-Initiative Creative Interfaces (2017) · cites this paper as [20]
In the citing paper
“Not only do these breakdowns significantly diminish user satisfaction and damage trust, they also decrease the likelihood of continued use [18, 44].”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [44]
In the citing paper
“This is because when voice assistants fail, such as by incorrectly answering a question, it derails user trust [27, 35].”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [35]
17 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
How Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808
(2010) · Sociology for Pharmacists · doi:10.4324/9780203381175_chapter_9 · Cited by 14,152
Norman, D. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,374
Thematic networks: an analytic tool for qualitative research
Attride-Stirling, J. (2001) · Qualitative Research · doi:10.1177/146879410100100307 · Cited by 5,506 · cited 2×
“The thematic network is a ‘web-like coding structure’ [2; 385] that allows for a rich understanding of the conceptual interplay occurring within the narrative.”
Background(1997) · Choice Reviews Online · doi:10.5860/choice.34-3702 · Cited by 4,887
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Licklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938 · cited 1×
“When considering human-machine dialogue from an HCI perspective, some of the earliest work can be traced back to vision statements and position papers such as JCR Licklider (1960), ‘Man-machine symbiosis’ [25] and the early work of Richard Bolt [4].”
MethodEmbodied Conversational Agents
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/2697.001.0001 · Cited by 1,799
Interaction Ritual: Essays on Face-to-face Behaviour
Goffman, E. (1972) · Medical Entomology and Zoology · Cited by 1,449
Research works printed but not linked: 14
- Boden Margaret A, 2007, Proceedings of Artificial Companions in Society: Perspectives on the Present and Future. Oxford Internet Institute.
- Davis Martin, Engines of Logic: Mathematicians and the Origin of the Computer
- Glass James R., 1999, Challenges for Spoken Dialogue Systems. In the proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding. (ASRU), CO, USA
- 10.5555/1941863 · doi:10.5555/1941863
- Heuwinkel Kerstin, Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Robnett Licklider Joseph Carl, 1960, HFE-1,(March
- Moore Roger, Spoken Language Processing: Where do we go from Here? In Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Payr Sabine, Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Preece Jenny, 2015, Interaction Design: Beyond Huma- Computer Interaction
- Pulman Stephen, Proceedings of the 2010 Workshop on Companionable Dialogue Systems. Assoc. for Computational Linguistics
- Trappl Robert, From Jeeves Jeannie to Siri, and Then? In Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Vassallo Giorgio, Semantic Computing. Sheu, Yu, Ramamoorthy, Joshi & Zadeh.(eds)
- 10.5555/1870559.1870562 · doi:10.5555/1870559.1870562
- Wilks Yorick, 2010, Amsterdam: John Benjamins.
Cited by
Items 301–400 of 1,007
Designing the Future of Personal Fashion
Vaccaro, K., Agarwalla, T., Shivakumar, S., & Kumar, R. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174201 · Cited by 35
“Conducting needfinding with chatbots using a wizard-of-oz approach is also important because users often misunderstand chatbot capabilities and their expectations can be “dramatically out of step" with system capabilities [27, 28].”
MethodSugumar, M., & Chandra, S. (2021) · Journal of International Technology and Information Management · doi:10.58729/1941-6679.1501 · Cited by 34
“Because of difficulties in mapping the chatbots' capabilities, users are unable to discern the chatbot capabilities (Jain et al., 2018; Luger & Sellen, 2016).”
BackgroundZhou, A. Y., Tsai, W. S., & Men, L. R. (2024) · International Journal of Business Communication · doi:10.1177/23294884241229223 · Cited by 43
“Luger and Sellen (2016) reported that user expectations were often out of step with the operation of the systems, resulting in frustration.”
Chatbots for Data Collection in Surveys: A Comparison of Four Theory-Based Interview Probes
Jacobsen, R. M., Cox, S. R., Griggio, C. F., & van Berkel, N. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714128 · Cited by 41
“…information has been explored in a variety of contexts, since conversational interfaces possess several advantages over traditional WIMP (Windows, Icons, Menus and Pointers) interfaces [76, 113], for instance, they do not lock users into a fixed path when users provide diverse requests [113].”
Measuring User Experience in Chatbots: An Approach to Interpersonal Communication Competence
Skjuve, M. B., & Brandtzæg, P. B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-17705-8_10 · Cited by 41
Maharjan, R., Doherty, K., Rohani, D. A., Bækgaard, P., & Bardram, J. E. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3484508 · Cited by 29
“And inally, studies have shown that users often establish high expectations for CAs [52], anticipating emotional exchanges, relationship building, and human-to-human like conversations [16], which often leads to a drastic decline in use when CAs cannot meet users’ expectations [16, 52].”
BackgroundThe Effect of Audiences on the User Experience with Conversational Interfaces in Physical Spaces
Candello, H., Pinhanez, C., Pichiliani, M., Cavalin, P., Figueiredo, F., Vasconcelos, M., & Do Carmo, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300320 · Cited by 33
Rapp, A., Boldi, A., Curti, L., Perrucci, A., & Simeoni, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580995 · Cited by 19
“When chatbots do not meet these expectations, the user may experience dissatisfaction or frustration [54, 100].”
BackgroundInfluentialMusic in the digital age: commodity, community, communion
Cross, I. R. M. (2023) · AI & Society · doi:10.1007/s00146-023-01670-9 · Cited by 37
“In general, as Luger and Sellen (2016) put it in the title of their paper, interacting with a conversational agent is all too often “Like having a really bad PA”.”
Managing Uncertainty in Time Expressions for Virtual Assistants
Rong, X., Fourney, A., Brewer, R. N., Morris, M. R., & Bennett, P. N. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025674 · Cited by 30
“Whether issuing commands or retrieving information, users are encouraged, by convention and instruction [18, 37], to”
BackgroundAylett, M. P., Sutton, S. J., & Vazquez-Alvarez, Y. (2019) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3342775.3342806 · Cited by 33
Weber, P. P., & Ludwig, T. G. (2020) · Mensch und Computer (MuC) · doi:10.1145/3404983.3405513 · Cited by 52
“The o igin of oda CA goe back o he chatbot ELIZA, often referenced as the first CA [2, 17, 26, 35, 41], from the year 1966 [50].”
BackgroundMethodMishra, A. A., & Shukla, A. (2020) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-64849-7_24 · Cited by 36
“…conceptualize SVITs mainly with reference to their underlying software components or computational infrastructure, for example, natural language processing, arti fi cial intelligence (AI), or cloud computing (e.g., Luger and Sellen 2016; Cho 2018; Lopatovska and Williams 2018; Myers et al. 2018).”
BackgroundRobinson, N., & Kavanagh, D. J. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102700 · Cited by 35
LipIO: Enabling Lips as both Input and Output Surface
Jingu, A., Tanaka, Y., & Lopes, P. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580775 · Cited by 33
User preferences and willingness to pay for in-vehicle assistance
Mihale-Wilson, C., Zibuschka, J., & Hinz, O. F. (2019) · Electronic Markets · doi:10.1007/s12525-019-00330-5 · Cited by 34
“Cowan et al. (2017), Luger and Sellen (2016) and Wulf et al. (2014) for instance document that not only frequent but also infrequent users of intelligent personal assistants such as Siri or Google lament the current technology performance and issues with trust, data privacy, and transparency (Ram…”
BackgroundA Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures
Baughan, A., Wang, X., Liu, A., Mercurio, A., Chen, J., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581152 · Cited by 32
“Sometimes after a failure, users will try to reformulate, simplify, or hyper-enunciate their commands as a way to continue using the device [31, 35, 43, 61].”
BackgroundResultInfluentialEmotional responses to human values in technology: The case of conversational agents
Görnemann, E., & Spiekermann, S. (2022) · Human-Computer Interaction · doi:10.1080/07370024.2022.2136094 · Cited by 35
“A number of studies found that gaps between user expectations and actual user experience diminish user satisfaction (Luger & Sellen, 2016; Nguyen & Sidorova, 2017; Zamora, 2017).”
Arakawa, R., Lehman, J. F., & Goel, M. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3699759 · Cited by 45
Differences in Interactions with a Conversational Agent
Oh, Y. H., Chung, K., & Ju, D. Y. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17093189 · Cited by 28
“Thirdly, elderly users need to be informed about the agent’s capabilities [9].”
BackgroundInfluentialHeuristic Evaluation of COVID-19 Chatbots
Höhn, S., & Bongard-Blanchy, K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-68288-0_9 · Cited by 32
“chatbot context [9, 21], stretching to user interviews to analyse user needs and expectations [12].”
BackgroundKiro: A Design Fiction to Explore Social Conversation with Voice Assistants
Ringfort-Felner, R., Laschke, M., Sadeghian, S., & Hassenzahl, M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492852 · Cited by 42
“There is increased interest within HCI to explore how people use and perceive VAs (e.g., [3, 19, 46, 66]).”
BackgroundReference model of next-generation digital personal assistant: integrating proactive behavior
Meurisch, C., Ionescu, M.-D., Schmidt, B., & Mühlhäuser, M. (2017) · ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) · doi:10.1145/3123024.3123145 · Cited by 35
“All participants confirm the findings of [5] and limit their usage to information search (e.”
BackgroundInfluentialLong-Term Value of Social Robots through the Eyes of Expert Users
Dereshev, D., Kirk, D., Matsumura, K., & Maeda, T. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300896 · Cited by 31
“Kiseleva et al. suggested that satisfaction with IPAs is related to task completion and effort to complete it [37], while Luger and Sellen related user satisfaction with user expectations of IPAs, pointing out that users have poor mental models of how IPAswork [42].”
BackgroundCommunityBots: Creating and Evaluating A Multi-Agent Chatbot Platform for Public Input Elicitation
Jiang, Z., Rashik, M., Panchal, K., Jasim, M., Sarvghad, A., Riahi, P., DeWitt, E., Thurber, F., & Mahyar, N. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579469 · Cited by 40
“Chatbots are designed to simulate an intelligent conversation with one or multiple humans via textual or auditory methods [75].”
BackgroundThunström, A. O., Carlsen, H. K., Ali, L., Larson, T., Hellström, A., & Steingrímsson, S. (2024) · JMIR Human Factors · doi:10.2196/54581 · Cited by 33
“In a study by Luger and Sellen [45], the authors found that higher expectations of the system lead to a higher risk of disappointment and lower scores: this could possibly explain why female participants were more agitated as their expectations might have been higher [45], however, we have no data…”
BackgroundMental Models and Home Virtual Assistants (HVAs)
Cho, J. (2018) · Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3170427.3180286 · Cited by 29
“There are some studies of peoples’ perceptions and use of VAs [8,12], but little work has looked at users’ mental models for what kinds of interactions they think the devices are capable of.”
BackgroundChatbots as Part of Digital Government Service Provision – A User Perspective
Abbas, N., Følstad, A., & Bjørkli, C. A. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25581-6_5 · Cited by 39
“At the same time, the open character of chatbot interaction may cause users to hold unrealistic expectations of chatbot capabilities (Luger & Sellen, 2016), which in turn may impact perceptions negatively (Zamora, 2017).”
BackgroundUser recommendations for intelligent personal assistants
Lopatovska, I., Griffin, A. L., Gallagher, K., Ballingall, C., Rock, C., & Velazquez, M. (2020) · Journal of Librarianship and Information Science · doi:10.1177/0961000619841107 · Cited by 27
“…is not perfect: long, sustained, coherent, on-topic conversations with IPAs, resembling human conversational patterns, are not supported (Guo et al., 2017), leading to frustration, dis-appointment, and discontinued use (Bopp, 2018; Cowan et al., 2017; Kiseleva et al., 2016; Luger and Sellen 2016).”
BackgroundMethodInfluentialDetermining the causes of user frustration in the case of conversational chatbots
Chhabra, S., Kaushal, V., & Girija, S. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2362956 · Cited by 31
“Thus, users have exhibited frustration and thereafter discontinuance post-adoption of the chatbot (Luger and Sellen 2016).”
Kim, J. S., & Baek, T. H. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2429647 · Cited by 33
“Users are more likely to engage with chatbots that actively participate in conversations and provide relevant responses (Luger and Sellen 2016).”
Enhancing EFL Speaking Skills with AI-Powered Word Guessing: A Comparison of Human and AI Partners
Pituxcoosuvarn, M., 谷村, 緑., Murakami, Y., & White, J. S. (2025) · Information · doi:10.3390/info16060427 · Cited by 31
Mozafari, N., Weiger, W. H., & Hammerschmidt, M. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.355 · Cited by 38
“On the other hand, research has shown that there seems to be a mismatch of user expectations towards chatbot performance and actual performance [16].”
BackgroundDesigning VUIs for Social Assistance Robots for People with Dementia
Striegl, J., Gollasch, D., Loitsch, C., & Weber, G. (2021) · Mensch und Computer (MuC) · doi:10.1145/3473856.3473887 · Cited by 31
A Mixed-Method Analysis of Text and Audio Search Interfaces with Varying Task Complexity
Vtyurina, A., Clarke, C. L. A., Law, E., Trippas, J. R., & Bota, H. (2020) · Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval · doi:10.1145/3409256.3409822 · Cited by 29
Commanding and Re-Dictation: Developing Eyes-Free Voice-Based Interaction for Editing Dictated Text
Ghosh, D., Liu, C., Zhao, S., & Hara, K. (2020) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3390889 · Cited by 29
“Luger and Sellen [18] had noted the presence of these “conversational-style” elements in users’ conversations with their conversational agents.”
BackgroundPoser, M., Küstermann, G. C., Tavanapour, N., & Bittner, E. A. C. (2022) · Information Systems Frontiers · doi:10.1007/s10796-022-10265-6 · Cited by 35
“Mis-interpretation of user input, incorrect responses, and tedious interactions often fail to meet users’ high expectations of conversations with CAs (Luger & Sellen, 2016).”
BackgroundChasing the Chatbots: Directions for Interaction and Design Research
Piccolo, L. S. G., Mensio, M., & Alani, H. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-17705-8_14 · Cited by 28
Potdevin, D., Clavel, C., & Sabouret, N. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2021.102612 · Cited by 28
When Users Assist the Voice Assistants: From Supervision to Failure Resolution
Lahoual, D., & Frejus, M. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3299053 · Cited by 28
Bowden, K. K., Wu, J., Cui, W. Q., Juraska, J., Harrison, V., Schwarzmann, B., Santer, N. D., Whittaker, S., & Walker, M. (2019) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3342775.3342792 · Cited by 36
“For example [28] explored user reactions to Alexa, finding that participants often had inflated expectations of system capability leading to unsuccessful interactions and system errors.”
BackgroundSun, X., Liu, Y., de Wit, J., Bosch, J. A., & Li, Z. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650837 · Cited by 36
Investigating meta-intents: user interaction preferences in conversational recommender systems
Ma, Y., & Ziegler, J. (2024) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-024-09411-3 · Cited by 29
“A considerable number of users are reluctant to be strongly guided by a system in their search and decision process and may prefer to explore the available options themselves by using, for example, search or filtering techniques (Luger and Sellen 2016; Kleemann et al. 2022).”
Alexa, We Need to Talk: A Data Literacy Approach on Voice Assistants
Pins, D., Jakobi, T., Boden, A., Alizadeh, F., & Wulf, V. (2021) · Designing Interactive Systems Conference · doi:10.1145/3461778.3462001 · Cited by 34
“With VAs becoming common in private households, recent work has shown that users often only have superficial awareness of how a VA works [36] and how the collected data is being used [34].”
BackgroundAn Evaluation of Visual Embodiment for Voice Assistants on Smart Displays
Bonfert, M., Zargham, N., Saade, F., Porzel, R., & Malaka, R. (2021) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3469595.3469611 · Cited by 37
“Hwang et al. [22] categorized three distinct characteristics of bodily display, subordinate attitude, and sexualization to investigate the reflections of gender stereotypes toward women in female-voiced VAs.”
BackgroundInfluentialAnatomy of a Digital Assistant
Wellsandt, S., Hribernik, K., & Thoben, K. (2021) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-85910-7_34 · Cited by 32
Han, S., Liu, M., Pan, Z., Cai, Y. C., & Shao, P. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-022-00311-4 · Cited by 32
“Our observation of the students’ high expectations of the chatbot seems to keep with what a considerable body of prior research suggests; people often set unrealistically high expectations of conversational agents (Luger & Sellen, 2016; Wang et al., 2021b; Zamora, 2017).”
BackgroundStreetWise: Smart Speakers vs Human Help in Public Slum Settings
Pearson, J., Robinson, S., Reitmaier, T., Jones, M., Ahire, S., Joshi, A., Sahoo, D., Maravi, N., & Bhikne, B. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300326 · Cited by 28
Blair, J., & Abdullah, S. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3432194 · Cited by 35
“While prior studies found screen-based responses unfavorable as compared to voice responses [24], participants in this study preferred to have screenbased responses or have voice responses paired with supplemental information on their device screens.”
BackgroundResultInfluentialChin, J., Desai, S., Lin, S. (.-H., & Mejia, S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637365 · Cited by 27
“Luger & Sellen [48], in interviews with 14 participants, found that users began interactions with VAs with a high expectation of their intelligence but were disappointed due to the VAs’ limitation in understanding context and frequent functional errors.”
BackgroundAlshibly, H. H., Alwreikat, A., Ghobrial, R. A., & Abuaddous, M. Y. (2024) · Cogent Business & Management · doi:10.1080/23311975.2024.2387196 · Cited by 31
“…human-like conversations and interact with humans through the installed voice or textual interfaces (Dignum & Dignum, 2020; li & Zhang, 2023; luger & sellen, 2016). these systems are based on natural language processing (nlP) and artificial intelligence (aI) technologies to interact with…”
Liu, C., Su, M., Xiang, Y., Huang, Y., Yang, Y., Zhang, K., & Fan, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714228 · Cited by 36
“However, older adults often prefer utilizing shorter, less complex sentences and frequently rely on ambiguous expressions [13, 52, 86, 107].”
Lee, S., Cho, M., & Lee, S. (2020) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3411840 · Cited by 38
“Luger and Sellen [30] conducted a comprehensive study to understand what factors affect everyday use of dialogue-only CA systems.”
BackgroundResultInfluentialAlizadeh, F., Stevens, G., & Esau, M. (2021) · i-com · doi:10.1515/icom-2021-0009 · Cited by 38
User Expectations of Conversational Chatbots Based on Online Reviews
Svikhnushina, E., Placinta, A., & Pu, P. (2021) · Designing Interactive Systems Conference · doi:10.1145/3461778.3462125 · Cited by 35
“This is in line with prior research that found that playful interactions serve as an entry point even for functional personal assistants such as Siri or Cortana [26] and constitute an appealing feature for chatbots in general [5, 19, 29, 39].”
BackgroundResultKarkera, Y., Tandukar, B., Chandra, S., & Martin-Hammond, A. M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581561 · Cited by 32
“From this work, we have gained a better understanding of usage patterns [10, 53], needs, and challenges of VAs [23, 26, 44], but also the possibilities of these devices to support individuals with disabilities [2, 3, 14, 52], children [9], and those with varied mobility or abilities [2, 3, 12, 17, 56].”
BackgroundKlein, A. M., Deutschländer, J., Kölln, K., Rauschenberger, M., & Escalona, M. J. (2023) · Journal of Software Evolution and Process · doi:10.1002/smr.2618 · Cited by 29
“19 Although they are integrated into every-day life (e.g., Mobile via smartphones and At home via smart speakers), challenges such as data security and privacy concerns, 20 voice command recognition, 4,21 context-sensitivity, 9,22 and human-like conversation 5 remain difficult.”
Conversational Agents for Recipe Recommendation
Barko-Sherif, S., Elsweiler, D., & Harvey, M. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377967 · Cited by 30
Ayobi, A., Hughes, J., Duckworth, C. J., Dylag, J. J., Gordon James, S., Marshall, P., Guy, M., Kumaran, A., Chapman, A., Boniface, M., et al. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581424 · Cited by 26
How Personal Experience and Technical Knowledge Affect Using Conversational Agents
Chen, M.-L., & Wang, H.-C. (2018) · Companion Proceedings of the 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3180308.3180362 · Cited by 26
“Previous studies have shown that both frequent [2] or infrequent [1] users can over-or under-expect the capabilities of CA.”
BackgroundSocial identity in trusting artificial intelligence agents: Evidence from lab and online experiments
Sun, Y., Xu, C., & Xu, H. (2024) · Managerial and Decision Economics · doi:10.1002/mde.4361 · Cited by 28
SQLucid: Grounding Natural Language Database Queries with Interactive Explanations
Tian, Y., Kummerfeld, J. K., Li, T. J.-J., & Zhang, T. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676368 · Cited by 38
The Value-Sensitive Conversational Agent Co-Design Framework
Sadek, M., Calvo, R., & Mougenot, C. (2023) · International journal of human computer interactions · doi:10.1080/10447318.2024.2426737 · Cited by 26
“While experts recommended aligning expectations early regarding AI capabilities (Luger & Sellen, 2016; Amershi et al., 2019), we additionally recommend using shared intended values to create expectations around supported use-cases and underlying relevant values (as done in Activity 1 of the VSCA…”
Usable and Acceptable Response Delays of Conversational Agents in Automotive User Interfaces
Funk, M., Cunningham, C., Kanver, D., Saikalis, C., & Pansare, R. (2020) · International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI) · doi:10.1145/3409120.3410651 · Cited by 27
“Considering the use of a GUI, research has demanded [13], but not concluded on whether using a VUI should always be accompanied by a GUI.”
BackgroundMethodInfluentialMulti-Agent Voice Assistants: An Investigation of User Experience
Zargham, N., Bonfert, M., Porzel, R., Döring, T., & Malaka, R. (2021) · International Conference on Mobile and Ubiquitous Multimedia (MUM) · doi:10.1145/3490632.3490662 · Cited by 32
Exploring Smart Speaker User Experience for People Who Stammer
Bleakley, A., Rough, D. J., Roper, A., Lindsay, S., Porcheron, M., Lee, M., Nicholson, S. A., Cowan, B. R., & Clark, L. (2022) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3544823 · Cited by 31
“Prior work has demonstrated the reluctance of people to engage with speech interfaces in public settings [14, 33], due to a sense of social embarrassment or awkwardness [1, 14].”
BackgroundA need for trust in conversational interface research
Edwards, J., & Sanoubari, E. (2019) · Proceedings of the 1st International Conference on Conversational User Interfaces · doi:10.1145/3342775.3342809 · Cited by 22
“Other user studies have revealed a conceptualization of trust that relates instead to a belief in the abilities and competency of the interface [16].”
BackgroundGiving Voice to Silent Data: Designing with Personal Music Listening History
Wirfs-Brock, J., Mennicken, S., & Thom, J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376493 · Cited by 32
Speech diversity and speech interfaces: considering an inclusive future through stammering
Clark, L., Cowan, B. R., Roper, A., Lindsay, S., & Sheers, O. (2020) · Proceedings of the 2nd Conference on Conversational User Interfaces · doi:10.1145/3405755.3406139 · Cited by 25
“…the need to consider the potential gulf of expectation due to the humanness of such systems versus their actual functionality [8, 16, 19], the need to learn how to interact effectively [16] as well as how social and multiparty contexts impact the type of interactions we have with IPAs [8, 10, 24].”
BackgroundMitigating Boredom Using An Empathetic Conversational Agent
Samrose, S., Anbarasu, K., Joshi, A., & Mishra, T. (2020) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3383652.3423905 · Cited by 28
“While CAs have become a key mode of human-computer interaction, the difference between user expectation and actual interaction experience is still quite large [32].”
BackgroundUser Simulation for Evaluating Information Access Systems
Balog, K., & Zhai, C. (2023) · arXiv · doi:10.48550/arxiv.2306.08550 · Cited by 26
Iniesto, F., Coughlan, T., Lister, K., Devine, P., Freear, N., Greenwood, R., Holmes, W., Kenny, I., McLeod, K., & Tudor, R. (2023) · ACM Transactions on Accessible Computing · doi:10.1145/3568166 · Cited by 25
“Table 1 shows that many of the studies used questionnaires with task-driven or free iteration with the CUI, [12, 15, 33, 40, 41, 43, 45] 3 Free interaction [14, 34, 40, 41] 4 Direct observation [15, 40, 41] 6 Interview [34, 40] direct observation, or included interviews in their methodologies.”
BackgroundChatbots to Support Children in Coping with Online Threats: Socio-technical Requirements
Piccolo, L. S. G., Troullinou, P., & Alani, H. (2021) · Designing Interactive Systems Conference 2021 · doi:10.1145/3461778.3462114 · Cited by 25
“While some of these studies address specifically voice-based agents [12, 19, 23, 28], others focus on textual chatbots [24, 31, 32].”
BackgroundVtyurina, A. (2019) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3295750.3298976 · Cited by 27
Should robots be polite? Expectations about politeness in human–robot interaction
Lumer, E., & Buschmeier, H. (2023) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1242127 · Cited by 29
Prompting an Embodied AI Agent: How Embodiment and Multimodal Signaling Affects Prompting Behaviour
Zhang, T., Yeung, C. A., Aurelia, E., Onishi, Y., Chulpongsatorn, N., Li, J., & Tang, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713110 · Cited by 28
“Yet, interactions with these agents are often limited to rigid, turn-based exchanges, where users issue voice commands, and agents respond verbally [37].”
Framing human-AI collaboration in language education
Stockwell, G., & Wang, Y. (2025) · Technology in Language Teaching & Learning · doi:10.29140/tltl.v7n4.103824 · Cited by 25
Conversational Agents for Mental Health and Wellbeing
Callejas, Z., & Griol, D. (2021) · Logic, argumentation & reasoning · doi:10.1007/978-3-030-61438-6_11 · Cited by 29
“Luger and Sellen [74] present an interesting study of user expectations of conversational agents (in the general personal assistant context, not specifically related to mental health).”
PEACE: A Model of Key Social and Emotional Qualities of Conversational Chatbots
Svikhnushina, E., & Pu, P. (2022) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3531064 · Cited by 30
“Entertainment and humor form a common “entry point” to chatbot technology, even for the functional chatbots operating in domain-specific settings [44, 46].”
BackgroundResultInfluentialRuane, E., Faure, T., Smith, R., Bean, D., Carson-Berndsen, J., & Ventresque, A. (2018) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3180308.3180373 · Cited by 25
“Previous work around chatbot quality has focused on user quality of experience and has shown that users have high expectations and low tolerance for poor quality [3].”
BackgroundLinnemann, G. A., & Jucks, R. (2018) · Interacting with Computers · doi:10.1093/iwc/iwy005 · Cited by 26
How Virtual Personal Assistants Influence Children’s Communication
Bylieva, D. S., Bekiroğulları, Z., Lobatyuk, V. V., & Nam, T. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-65857-1_12 · Cited by 27
Key Qualities of Conversational Chatbots – the PEACE Model
Svikhnushina, E., & Pu, P. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450643 · Cited by 28
“chatbots operating in domain-specific settings [31, 33].”
BackgroundInfluentialAnthis, J. R., Pauketat, J. V. T., Ladak, A., & Manoli, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713329 · Cited by 24
“While anthropomorphism is often associated with positive HCI outcomes through increased trust and engagement, it can lead to unrealistic expectations [108] and the well-known “uncanny valley” effect [163].”
Kokorelias, K. M., Grigorovich, A., Harris, M. T., Rehman, U., Ritchie, L., Levy, A., Denecke, K., & McMurray, J. (2024) · Journal of Medical Internet Research · doi:10.2196/57258 · Cited by 26
Kaniwa, Y., Kuribayashi, M., Kayukawa, S., Sato, D., Takagi, H., Asakawa, C., & Morishima, S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3676492 · Cited by 32
Poser, M., Singh, S., & Bittner, E. A. C. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.144 · Cited by 28
“Furthermore, customers’ high expectations toward CAs’ conversational capabilities lead to breakdowns [35].”
BackgroundAge-Related Differences in Preferences for Using Voice Assistants
Gollasch, D., & Weber, G. (2021) · Mensch und Computer (MuC) · doi:10.1145/3473856.3473889 · Cited by 29
Zargham, N., Alexandrovsky, D., Erich, J., Wenig, N., & Malaka, R. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3519843 · Cited by 35
Chan, S., Li, J., Yao, B., Mahmood, A., Huang, C., Jimison, H. B., Mynatt, E. D., & Wang, D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757442 · Cited by 27
Voice-based Conversational UX Studies and Design
Reeves, S., Porcheron, M., Fischer, J. E., Candello, H. C. D. S. P., McMillan, D., McGregor, M., Moore, R. J., Sikveland, R. O., Taylor, A. S., Velkovska, J., et al. (2018) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3170427.3170619 · Cited by 30
“Furthermore, the work lends itself to myriad research approaches, including ethnomethodological and conversation analytic work [7,10,11,12,17], interviews [4], and Wizard of Oz and technical probe approaches [5].”
BackgroundVitro: Designing a Voice Assistant for the Scientific Lab Workplace
Cambre, J., Liu, Y., Taylor, R. E., & Kulkarni, C. E. (2019) · Designing Interactive Systems Conference · doi:10.1145/3322276.3322298 · Cited by 30
“This has led some researchers to question whether voice assistants have any compelling utility in everyday life [33].”
BackgroundImproving the Usability of Voice User Interfaces: A New Set of Ergonomic Criteria
Nowacki, C., Gordeeva, A., & Lizé, A.-H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49760-6_8 · Cited by 28
Anthis, J. R., Pauketat, J. V. T., Ladak, A., & Manoli, A. (2024) · arXiv · doi:10.48550/arxiv.2407.08867 · Cited by 24
“While anthropomorphism is often associated with positive HCI outcomes through increased trust and engagement, it can lead to unrealistic expectations [108] and the well-known “uncanny valley” effect [163].”
Understanding the Needs and Challenges of Using Conversational Agents for Deaf Older Adults
Blair, J., & Abdullah, S. (2019) · Companion Publication of the 2019 Conference on Computer Supported Cooperative Work and Social Computing · doi:10.1145/3311957.3359487 · Cited by 23
“Previous CA studies have focused on the daily use [1, 3, 16] and challenges of voice interfaces across general audiences [8, 9, 13], as well as the barriers associated with other voice technologies for DHH users [2, 5, 6].”
BackgroundFrom explanations to shared understandings of AI
Nicenboim, I. P., Giaccardi, E., & Redström, J. (2022) · Proceedings of DRS · doi:10.21606/drs.2022.773 · Cited by 24
Robe, P., Kuttal, S. K., AuBuchon, J., & Hart, J. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549127 · Cited by 27
Conversational Agents: Acting on the Wave of Research and Development
Jacques, R., Følstad, A., Gerber, E. M., Grudin, J. T., Luger, E. A., Monroy‐Hernández, A., & Wang, D. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3299034 · Cited by 23
“[7] By 2018, intelligent agents had appreciative users, but progressing beyond basic tasks was challenging [15].”
BackgroundSelf-citationSun, L., Fu, S., Yao, B., Lu, Y., Li, W., Gu, H., Gesi, J., Huang, J., Luo, C., & Wang, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.21501 · Cited by 25
“Evaluating the usability of agentic AI systems, especially those involving conversational assistants, presents unique methodological challenges [7, 11, 46].”
Intentions, Meanings, and Whys
Barth, F. J., Candello, H. C. D. S. P., Cavalin, P., & Pinhanez, C. S. (2020) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3405755.3406128 · Cited by 22
Metadata
- Authors
- Luger, E., & Sellen, A.
- Affiliations
- Microsoft Research (United Kingdom) (GB)
- Venue
- Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2016-05-07
- Pages
- 5286-5297
- OpenAlex
- W2405187948
- Semantic Scholar
- 814e62bb4f0ec6facd640fbd3e3fcbd189716d20
- Cited by
- 1,176
- References
- 51
- Citations by year
- 2026: 164 · 2025: 175 · 2024: 119 · 2023: 116 · 2022: 107 · 2021: 148 · 2020: 141 · 2019: 126 · 2018: 54 · 2017: 25 · 2016: 1
- FWCI
- 37.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- chatbots, voice assistants, everyday use, user expectations, human-computer interaction, multimodality, system capabilities, interface design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.