"Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents
The DOI matches its Crossref record; 49 of the 51 references it cites were found in a registry or an index.
Abstract
The past four years have seen the rise of conversational agents (CAs) in everyday life. Apple, Microsoft, Amazon, Google and Facebook have all embedded proprietary CAs within their software and, increasingly, conversation is becoming a key mode of human-computer interaction. Whilst we have long been familiar with the notion of computers that speak, the investigative concern within HCI has been upon multimodality rather than dialogue alone, and there is no sense of how such interfaces are used in everyday life. This paper reports the findings of interviews with 14 users of CAs in an effort to understand the current interactional factors affecting everyday use. We find user expectations dramatically out of step with the operation of the systems, particularly in terms of known machine intelligence, system capability and goals. Using Norman's 'gulfs of execution and evaluation' [30] we consider the implications of these findings for the design of future systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Whereas, we found that the challenges our participants experienced were more elementary.”
From Exploring How Older Adults Use a Smart Speaker–Based Voice Assistant in Their First Interactions: Qualitative Study (2021) · cites this paper as [32]
In the citing paper
“In addition, it could be more important for novice users to have an appropriate mental model and understand the system rather than an efficient user experience [32].”
From Exploring How Older Adults Use a Smart Speaker–Based Voice Assistant in Their First Interactions: Qualitative Study (2021) · cites this paper as [32]
In the citing paper
“human-AI mixed-initiative presents a generally valuable and underexplored interface paradigm.”
From Mixed-Initiative Creative Interfaces (2017) · cites this paper as [20]
In the citing paper
“Not only do these breakdowns significantly diminish user satisfaction and damage trust, they also decrease the likelihood of continued use [18, 44].”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [44]
In the citing paper
“This is because when voice assistants fail, such as by incorrectly answering a question, it derails user trust [27, 35].”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [35]
17 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
How Many Interviews Are Enough?: An Experiment with Data Saturation and Variability
Guest, G., Bunce, A., & Johnson, L. (2006) · Field Methods · doi:10.1177/1525822x05279903 · Cited by 20,808
(2010) · Sociology for Pharmacists · doi:10.4324/9780203381175_chapter_9 · Cited by 14,152
Norman, D. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,374
Thematic networks: an analytic tool for qualitative research
Attride-Stirling, J. (2001) · Qualitative Research · doi:10.1177/146879410100100307 · Cited by 5,506 · cited 2×
“The thematic network is a ‘web-like coding structure’ [2; 385] that allows for a rich understanding of the conceptual interplay occurring within the narrative.”
Background(1997) · Choice Reviews Online · doi:10.5860/choice.34-3702 · Cited by 4,887
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Licklider, J. C. R. (1960) · IRE Transactions on Human Factors in Electronics · doi:10.1109/thfe2.1960.4503259 · Cited by 1,938 · cited 1×
“When considering human-machine dialogue from an HCI perspective, some of the earliest work can be traced back to vision statements and position papers such as JCR Licklider (1960), ‘Man-machine symbiosis’ [25] and the early work of Richard Bolt [4].”
MethodEmbodied Conversational Agents
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/2697.001.0001 · Cited by 1,799
Interaction Ritual: Essays on Face-to-face Behaviour
Goffman, E. (1972) · Medical Entomology and Zoology · Cited by 1,449
Research works printed but not linked: 14
- Boden Margaret A, 2007, Proceedings of Artificial Companions in Society: Perspectives on the Present and Future. Oxford Internet Institute.
- Davis Martin, Engines of Logic: Mathematicians and the Origin of the Computer
- Glass James R., 1999, Challenges for Spoken Dialogue Systems. In the proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding. (ASRU), CO, USA
- 10.5555/1941863 · doi:10.5555/1941863
- Heuwinkel Kerstin, Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Robnett Licklider Joseph Carl, 1960, HFE-1,(March
- Moore Roger, Spoken Language Processing: Where do we go from Here? In Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Payr Sabine, Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Preece Jenny, 2015, Interaction Design: Beyond Huma- Computer Interaction
- Pulman Stephen, Proceedings of the 2010 Workshop on Companionable Dialogue Systems. Assoc. for Computational Linguistics
- Trappl Robert, From Jeeves Jeannie to Siri, and Then? In Your Virtual Butler: The Making-of. Robert Trappl (ed)
- Vassallo Giorgio, Semantic Computing. Sheu, Yu, Ramamoorthy, Joshi & Zadeh.(eds)
- 10.5555/1870559.1870562 · doi:10.5555/1870559.1870562
- Wilks Yorick, 2010, Amsterdam: John Benjamins.
Cited by
Items 401–500 of 1,007
The Effects of System Initiative during Conversational Collaborative Search
Avula, S., Choi, B., & Arguello, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512913 · Cited by 32
“, usability and willingness to collaborate with the system) [35, 37].”
BackgroundPan, L., Yu, C., Li, J., Huang, T. Y., Bi, X., & Shi, Y. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517459 · Cited by 30
Moldt, J.-A., Wietek, T. F., Mamlouk, A. M., & Herrmann‐Werner, A. (2022) · Digital Health · doi:10.1177/20552076221139092 · Cited by 24
“Dissatisfaction can be generated in the user if these expectations are not met.(56) As stated by Mori et al.”
BackgroundHaghighat, P., Nguyen, T. Q., Valizadeh, M., Arvan, M., Parde, N., Kim, M., & Jeong, H. (2023) · Applied Ergonomics · doi:10.1016/j.apergo.2023.103969 · Cited by 25
Haoyue, L. L., & Cho, H. (2024) · Universal Access in the Information Society · doi:10.1007/s10209-023-01087-7 · Cited by 30
Hands-free but not Eyes-free: A Usability Evaluation of Siri while Driving
Larsen, H. H., Scheel, A. N., Bogers, T., & Larsen, B. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377962 · Cited by 25
“These tasks were partly based on earlier work on IPA usability [24] and voice search log analysis [9], who found that voice queries with narrow information needs were more common for IPA use.”
MethodCybersex with human- and machine-cued partners: Gratifications, shortcomings, and tensions.
Banks, J., & Van Ouytsel, J. (2020) · Technology, Mind, and Behavior · doi:10.1037/tmb0000008 · Cited by 14
“…agents in terms of machine intelligence, capabilities of systems, and goals for their use that are often not fulfilled by those agents (Luger & Sellen, 2016), especially with respect to specific expectations for contingency—the variable alignment between a speaker’s actions and a…”
BackgroundNatural language goal understanding for smart home environments
Noura, M., Heil, S., & Gaedke, M. (2020) · International Conference on the Internet of Things (IoT) · doi:10.1145/3410992.3410996 · Cited by 25
“However, with the rise of modern voice-based user interfaces such as Amazon Echo and Google Home, the way of controlling smart devices is shifting from a controller-centric model to a people-centric model to make smart environments more natural for end-users [12].”
BackgroundExamining Humanness as a Metaphor to Design Voice User Interfaces
Desai, S., Dubiel, M., & Leiva, L. A. (2024) · ACM Conversational User Interfaces 2024 · doi:10.1145/3640794.3665535 · Cited by 21
“These folk theories stem from the discrepancy between what the users expect based on the metaphors provided to them and the actual experience of using CUIs [48], also explained by the concept of Norman’s “gulf of execution” [60].”
BackgroundChang, R.-C., Natalie, R., Xu, W., Yap, J. Z. F., & Guo, A. (2025) · International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663547.3746319 · Cited by 26
“When interacting with humans, such assessments of ability are often made through conversational cues and context, but the absence of analogous signals in AI systems makes it difficult for users to form accurate mental models of the system’s capabilities [65].”
Generational differences in trust in digital assistants
Noah, B., & Sethumadhavan, A. (2019) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1071181319631029 · Cited by 23
“Research on digital assistants indicates that there is a large gap between user expectations and actual experience, and this can lead to either disuse or misuse (Luger & Sellen, 2016).”
BackgroundBuilding and Designing Expressive Speech Synthesis
Aylett, M. P., Clark, L., Cowan, B. R., & Torre, I. (2021) · ACM eBooks · doi:10.1145/3477322.3477329 · Cited by 22
Smart Speakers for Inclusion: How Can Intelligent Virtual Assistants Really Assist Everybody?
Sciarretta, E., & Alimenti, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78462-1_6 · Cited by 22
Rebalancing Worker Initiative and AI Initiative in Future Work: Four Task Dimensions
He, J., Piorkowski, D., Muller, M. J., Brimijoin, K., Houde, S., & Weisz, J. D. (2023) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3596671.3598572 · Cited by 25
Estimating Conversational Styles in Conversational Microtask Crowdsourcing
Qiu, S., Gadiraju, U., & Bozzon, A. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392837 · Cited by 25
Conversational Futures: Emancipating Conversational Interactions for Futures Worth Wanting
Lee, M., Noortman, R., Zaga, C., Starke, A. D., Huisman, G., & Andersen, K. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445244 · Cited by 29
“Nevertheless, CUIs usually cannot handle more complex interactions [43], and many people expect CUIs only to have utilitarian request-response interactions with us [15] like customer service agents [53].”
BackgroundTurner, S., Pattnaik, N., Nurse, J. R. C., & Li, S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555159 · Cited by 23
“However, users’ mental models of home IoT devices have been shown to be inaccurate [40], and this lack of understanding, coupled with a lack of interfaces, can make it harder to understand how the devices should be secured.”
Background“I Didn’t Catch That, But I’ll Try My Best”: Anticipatory Error Handling in a Voice Controlled Game
Zargham, N., Pfau, J., Schnackenberg, T., & Malaka, R. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502115 · Cited by 20
Is Alexa like a computer? A search engine? A friend? A silly child? Yes.
Desai, S., & Twidale, M. (2022) · Proceedings of the 4th Conference on Conversational User Interfaces · doi:10.1145/3543829.3544535 · Cited by 20
Designing AI Personalities: Enhancing Human-Agent Interaction Through Thoughtful Persona Design
Zargham, N., Dubiel, M., Desai, S., Mildner, T., & Belz, H.-J. (2024) · International Conference on Mobile and Ubiquitous Multimedia (MUM) · doi:10.1145/3701571.3701608 · Cited by 26
“Studies suggest that many conversational agents fail to meet user expectations as effective interlocutors [11, 12, 21, 24, 31, 40].”
BackgroundPorcheron, M., Fischer, J. E., & Valstar, M. (2020) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3405755.3406168 · Cited by 20
“…(VUIs), from smartphone assistants (e.g. Siri) through to smart speakers (e.g. Amazon Echo) have led to a recent resurgence of research in the HCI (e.g. [10, 16]) and CSCW (e.g. [12, 15, 17]) communities that examines the design and use of novel technologies such as ‘natural language’ interfaces.”
BackgroundEliciting Spoken Interruptions to Inform Proactive Speech Agent Design
Edwards, J., Janssen, C. P., Gould, S. J. J., & Cowan, B. R. (2021) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3469595.3469618 · Cited by 22
Automated information retrieval and services of graduate school using chatbot system
Rukhiran, M., & Netinant, P. (2022) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v12i5.pp5330-5338 · Cited by 26
“Additionally, chatbot application tools represent an excellent opportunity for helping human emotion responses [21] and precisely interactive information retrieval [22], [23].”
Ehtesham-Ul-Haque, M., D’Rozario, J., Adnin, R., Utshaw, F. T., Tasneem, F., Shefa, I. J., & Islam, A. B. M. A. A. (2023) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101168 · Cited by 22
Let’s Talk Games: An Expert Exploration of Speech Interaction with NPCs
Zargham, N., Friehs, M. A., Tonini, L., Alexandrovsky, D., Ruthven, E. G., Nacke, L. E., & Malaka, R. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2338666 · Cited by 29
“Studies have shown that users tend to push the limits of VUIs by posing diverse questions, often surpassing the agent’s abilities (Lovato & Piper, 2015; Luger & Sellen, 2016).”
Exploring AI Problem Formulation with Children via Teachable Machines
Dwivedi, U., Elsayed-Ali, S., Bonsignore, E., & Kacorri, H. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642692 · Cited by 23
“While having a human-like name and being able to listen/speak is at some level anthropomorphism [70], these features are standard even in everyday devices and appliances that are not considered social robots ( e.g. our smartphones).”
BackgroundExamining Children's Perceptions of Intelligent Media Characters
Xu, Y., Thomas, T., Li, Z., Chan, M., Lin, G. C., & Moore, K. (2024) · International Journal of Child-Computer Interaction · doi:10.1016/j.ijcci.2024.100700 · Cited by 20
Igwe-Nmaju, C., Gbaja, C., & Ikeh, C. O. (2023) · International Journal of Science and Research Archive · doi:10.30574/ijsra.2023.10.2.1042 · Cited by 23
Relationships to “conversational” machines: Living with voice assistants at home
Velkovska, J., Zouinar, M., Veyrier, C.-A., & Carey-Libbrecht, L. (2020) · Réseaux · doi:10.3917/res.220.0047 · Cited by 23
"Human, All Too Human": NOAA Weather Radio and the Emotional Impact of Synthetic Voices
Scott, K. M., Ashby, S., & Hanna, J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376338 · Cited by 23
“While current evidence is typically focused on individual consumption through interactive virtual assistants [21, 11] or chatbots [23], the present study examines user reactions “in the wild” and in the quite novel and unexplored context of long-format radio content.”
BackgroundCan digital personal assistants persuade people to exercise?
Paay, J., Kjeldskov, J., Papachristos, E., Hansen, K. M., Jørgensen, T. B., & Overgaard, K. L. (2020) · Behaviour and Information Technology · doi:10.1080/0144929x.2020.1814412 · Cited by 27
Guo, J., Guo, J., Yang, C., Wu, Y., & Sun, L. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445129 · Cited by 26
“Otherwise, breakdowns would deplete user’s willingness to interact [19, 33], and they may refuse to reveal more information and are more likely to leave the conversation.”
BackgroundTea, Earl Grey, Hot: Designing Speech Interactions from the Imagined Ideal of Star Trek
Axtell, B., & Munteanu, C. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445640 · Cited by 21
“replicating human-to-human conversation is not desired or helpful [16,31].”
BackgroundDigital ethnography of home use of digital personal assistants
Paay, J., Kjeldskov, J., Hansen, K. M., Jørgensen, T. B., & Overgaard, K. L. (2020) · Behaviour and Information Technology · doi:10.1080/0144929x.2020.1834620 · Cited by 22
“…for many years, including the use and usability of VUIs (Myers et al. 2018), user expectation and user experience of them (Goulati and Szostak 2011; Luger and Sellen 2016), improved accessibility for people with physical impairments (Corbett and Weber 2016), and design guidelines for speech…”
BackgroundMethodInfluentialHu, Y., Xiao, Y., Hua, Y., Fan, Y. W., & Li, F. (2024) · Behavioral Sciences · doi:10.3390/bs14121182 · Cited by 23
How Can a Chatbot Support Human Resource Management? Exploring the Operational Interplay
Taule, T., Følstad, A., & Fostervold, K. I. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-94890-0_5 · Cited by 23
“Alexa, Do You Want to Build a Snowman?” Characterizing Playful Requests to Conversational Agents
Shani, C., Libov, A., Tolmach, S., Lewin-Eytan, L., Maarek, Y., & Shahaf, D. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3519870 · Cited by 30
Humanizing Chatbots for Interactive Marketing
Tsai, W. S., & Chuan, C. (2023) · Springer eBooks · doi:10.1007/978-3-031-14961-0_12 · Cited by 22
Understanding In-Situ Programming for Smart Home Automation
Liu, X., Shi, Y., Yu, C., Cheng, G., Yang, T., Liang, C., & Shi, Y. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3596254 · Cited by 30
“Although we unified the system capacity introduction and the system behaviors through the studies, participants reported that they had different expectations for the system depending on different technical backgrounds and their previous experiences [93].”
BackgroundLind, M., & Dickel, S. (2023) · Convergence The International Journal of Research into New Media Technologies · doi:10.1177/13548565231192100 · Cited by 25
“…open to speculation whether the marketing managers of this commercial have taken into account the potentially involuntary perlocutionary effect of necessarily disappointing customers whose expectations cannot be met by the reality of interacting with these devices (e.g. Luger and Sellen, 2016).”
Human-Computer Interaction: A Systematic Review
Jain, P., Gupta, V. K., Tiwari, H., Shukla, A., Pandey, P., & Gupta, A. (2023) · International Conference on Advanced Computing & Communication Technologies (ICACCTech) · doi:10.1109/icacctech61146.2023.00015 · Cited by 19
“Good UX design is critical in ensuring that users enjoy using the technology and can accomplish their goals quickly and efficiently [3].”
BackgroundCoimagining the Future of Voice Assistants with Cultural Sensitivity
Seaborn, K., Sawa, Y., & Watanabe, M. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/3238737 · Cited by 22
The Personality Dimensions GPT-3 Expresses During Human-Chatbot Interactions
Kovačević, N., Holz, C., Groß, M. H., & Wampfler, R. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3659626 · Cited by 27
“Conveying personality in conversational agents makes the interaction engaging and believable, lets the user anthropomorphize the agents [40, 66], enhances the user engagement [88] and user experience [91], and increases user acceptance of the agent through a high level of personalization [98].”
Chen, Q., Gong, Y. Y., Keil, M., Liu, S., & Lu, Y. (2024) · MIS Quarterly · doi:10.25300/misq/2023/17525 · Cited by 20
LLM content moderation and user satisfaction: evidence from response refusals in Chatbot Arena
Pasch, S. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2565668 · Cited by 28
“After applying this filtering criterion, our final dataset consists of 49,938 conversation pairs, each with a single user prompt and two corresponding model responses.”
Nicky: Toward a Virtual Assistant for Test and Measurement Instrument Recommendations
Kincaid, R. H., & Pollock, G. (2017) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc.2017.11 · Cited by 19
Frame Analysis of Voice Interaction Gameplay
Allison, F., Newn, J., Smith, W., Carter, M., & Gibbs, M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300623 · Cited by 19
“Despite these aspirations, however, currentvoice-controlledsystemsaremoreoftencharacterised astroublesome[30],disappointing[23]orembarrassing[7].”
Zhao, Y., Jaber, R., McMillan, D., & Munteanu, C. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502036 · Cited by 22
Shin, J. Y., & Huh-Yoo, J. (2020) · Proceedings of the 14th EAI International Conference on Pervasive Computing Technologies for Healthcare · doi:10.1145/3421937.3422024 · Cited by 18
“Examples included the revelation of system intelligence, the creation of humorous engagement, and supported ongoing user engagement through the agent design [34].”
BackgroundMethodInfluentialArtificial intelligence and higher education: A systematic review
Behera, A., Trivedi, P., Patra, S., & Makeni, C. (2025) · American Journal of STEM Education · doi:10.32674/0x43s107 · Cited by 20
Elwahsh, S., Stern, N., Singh, A., & Ayobi, A. (2025) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3719160.3736615 · Cited by 24
“Mhm...” – Conversational Strategies For Product Search Assistants
Papenmeier, A., Frummet, A., & Kern, D. (2022) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3498366.3505809 · Cited by 20
Zhou, Z., Li, Z., Zhang, Y., & Sun, L. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2093773 · Cited by 22
“However, breakdown repair strategies, such as confirmation and repeat, do not work sufficiently effectively because they do not provide sufficient transparency of the agent status (Beneteau et al., 2019), cause of errors (Li et al., 2020, p.), or affordance (Luger & Sellen, 2016).”
BackgroundPhinnemore, R., Reza, M., Lewis, B., Mahadevan, K., Wang, B. K., Annett, M., & Wigdor, D. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581346 · Cited by 21
“These attributes may contribute to users assigning anthropomorphic characteristics to voice assistants that are implemented within a conversational agent, computer, or media [41, 56] or having emotional interactions with voice assistants [72].”
BackgroundOrthmann, B., Leite, I., Bresin, R., & Torre, I. (2023) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3611655 · Cited by 22
“However, Natural Language Processing and spoken dialogue systems are advancing at a much slower rate than synthetic voices [9, 33, 35, 63], so we are currently at risk of deploying high-end, human-sounding voices on robots that cannot live up to the expectations that these voices aford.”
BackgroundDo, H. J., Kong, H., Tetali, P., Karahalios, K. G., & Bailey, B. P. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610192 · Cited by 23
A Study of Usage and Usability of Intelligent Personal Assistants in Denmark
Bogers, T., Al-Basri, A. A. A., Rytlig, C. O., Møller, M. E. B., Rasmussen, M. J., Michelsen, N. K. B., & Jørgensen, S. G. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-15742-5_7 · Cited by 21
Exploring Interaction with Remote Autonomous Systems using Conversational Agents
Robb, D. A., Lopes, J., Padilla, S., Laskov, A., Garcia, F. J. C., Liu, X., Willners, J. S., Valeyrie, N., Lohan, K. S., Lane, D. M., et al. (2019) · Designing Interactive Systems Conference · doi:10.1145/3322276.3322318 · Cited by 20
“This is an example of a gap between system context and user context, a phenomenon noted in CAs from other domains [25, 26, 42].”
BackgroundExploring the Uncanny Valley Theory in the Constructs of a Virtual Assistant Personality
García, M. P., & López, S. S. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-29516-5_76 · Cited by 22
The Effect of Embodied Anthropomorphism of Personal Assistants on User Perceptions
Schneiders, E., Papachristos, E., & van Berkel, N. (2021) · Australian Conference on Human-Computer Interaction (OzCHI) · doi:10.1145/3520495.3520503 · Cited by 18
PROs in the wild: Assessing the validity of patient reported outcomes in an electronic registry
Cabitza, F., Dui, L. G., & Banfi, G. (2019) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2019.01.009 · Cited by 18
Cognitive plausibility in voice-based AI health counselors
Kannampallil, T., Smyth, J. M., Jones, S., Payne, P., & Ma, J. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0278-7 · Cited by 18
Svikhnushina, E., & Pu, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.13883 · Cited by 19
“Many previous studies pointed out the importance of naturalness, i.e. human-like qualities, in this technology [6, 7, 8, 9, 10, 11].”
BackgroundResultInfluentialA Conversational Interface for Self-screening for ADHD in Adults
Håvik, R., Wake, J. D., Flobak, E., Lundervold, A., & Guribye, F. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-17705-8_12 · Cited by 17
“Many recent variants directed at the general public have been integrated with major social media and communication platforms [10,17], and have become available in areas such as education, information retrieval, business and customer support [18].”
BackgroundEnhancing smart home interaction through multimodal command disambiguation
Calò, T., & Russis, L. D. (2024) · Personal and Ubiquitous Computing · doi:10.1007/s00779-024-01827-3 · Cited by 21
Exploring Verbal Uncanny Valley Effects with Vague Language in Computer Speech
Clark, L., Ofemile, A. Y., & Cowan, B. R. (2020) · Prosody, phonology and phonetics · doi:10.1007/978-981-15-6627-1_17 · Cited by 24
Diabetes and conversational agents: the AIDA project case study
Alloatti, F., Bosca, A., Di, L., & Pieraccini, F. (2021) · Discover Artificial Intelligence · doi:10.1007/s44163-021-00005-1 · Cited by 23
Mckie, I. A. S., Narayan, B., & Kocaballi, A. B. (2022) · Journal of the Australian Library and Information Association · doi:10.1080/24750158.2022.2104738 · Cited by 22
“Just like learning how to use a library or a search engine, users now are also experiencing a learning curve in relation to chatbots and voice assistants, where the expectations of how to use the devices fall short of how users are integrating them into their daily routines (Luger & Sellen, 2016).”
BackgroundAudience design and egocentrism in reference production during human-computer dialogue
Peña, P. R., Doyle, P. R., Edwards, J., Garaialde, D., Rough, D. J., Bleakley, A., Clark, L., Tobar-Henríquez, A., Branigan, H. P., Gessinger, I., et al. (2023) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103058 · Cited by 20
Dubiel, M., Leiva, L. A., Bongard-Blanchy, K., & Sergeeva, A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3685274 · Cited by 17
“Building on previous research that elicited user’s expectations regarding proactive CA support (e.g., [21, 30, 83, 84, 114, 132, 137]), our work focuses on an interactive, voice-only decision-making scenario to explore the impact of proactive interventions of a CA. Specifically, we investigate (1)…”
Ferguson, S. A., Aoyagui, P. A., Rizvi, R., Kim, Y., & Kuzminykh, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3687056 · Cited by 21
“Additionally, humans are reported to be uncomfortable when artificial systems produce uniquely human-like characteristics [110], e.g., intelligent agents displaying human-like emotional responses can trigger discomfort [142] or frustration [100, 103].”
How Do People Adapt to Use of an IoT Air Purifier?: From Low Expectation to Minimal Use
Cho, M., Lee, S., & Lee, K.-P. (2019) · PolyU Institutional Research Archive (Hong Kong Polytechnic University) · Cited by 23
Practical Challenges of Virtual Assistants and Voice Interfaces in Industrial Applications
Gärtler, M., & Schmidt, B. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.494 · Cited by 17
“Starting with dialog systems in natural language [32], extending to spoken language and integration of technical services in [5, 26, 17, 15] to more design-driven and holistic views in [16, 14, 20, 8, 18, 3, 19].”
BackgroundGünay, A., Yargın, G. T., Süner-Pla-Cerdà, S., & Kulaksız, M. (2022) · Behaviour and Information Technology · doi:10.1080/0144929x.2022.2054357 · Cited by 20
“In VA interaction, voice recognition and natural language processing abilities are the most commonly used modalities in addition to — or sometimes instead of — graphical user interfaces (GUIs) (Luger and Sellen 2016; Knote et al. 2018).”
BackgroundMethodInfluentialMurad, C., Tasnim, H., & Munteanu, C. (2022) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3543829.3543842 · Cited by 21
“Even with research and technological advances over the past decade, users continue to experience usability issues with VUIs, often leading to a lack of adoption or abandonment [11, 17, 20].”
BackgroundInfluentialAn Analysis of (Non-)Use Practices and Decisions of Internet of Things
Garg, R. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29390-1_1 · Cited by 19
“Only a few studies discuss the challenges [54] or smart voice assistants [35], and they do not specifically discuss users’ (non-)use practices and decisions.”
Völkel, S. T., Meindl, S., & Hussmann, H. (2021) · CUI 2021 - 3rd Conference on Conversational User Interfaces · doi:10.1145/3469595.3469605 · Cited by 16
“However, in particular the use of humour should be carefully aligned to the specific situation to avoid raising unrealistic expectations about the voice assistant’s capabilities [31].”
BackgroundFrom the Curtain to Kansas: Conducting Wizard-of-Oz Studies in the Wild
Mitchell, E., & Mamykina, L. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3443446 · Cited by 16
“Unfortunately, many current conversational agents fall short of this imagined potential, and there is still a need for new systems that bridge this gap [9, 18, 24].”
BackgroundToward Metaphor-Fluid Conversation Design for Voice User Interfaces
Desai, S., Chin, J., Wang, D., Cowan, B. R., & Twidale, M. B. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816223 · Cited by 16
Let’s Talk About CUIs: Putting Conversational User Interface Design Into Practice
Murad, C., Munteanu, C., Cowan, B. R., Clark, L., Porcheron, M., Candello, H. C. D. S. P., Schlögl, S., Aylett, M. P., Sin, J., Moore, R. J., et al. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3441336 · Cited by 18
“amount of information that can be remembered, system feedback, learnability, and recognition errors [11, 16, 26, 27].”
BackgroundConversational Interfaces in IoT Ecosystems: Where We Are, What Is Still Missing
Gallo, S., Paternò, F., & Malizia, A. (2023) · International Conference on Mobile and Ubiquitous Multimedia (MUM) · doi:10.1145/3626705.3627775 · Cited by 21
“Breakdowns occur when a conversational agent fails to understand user inputs, leading to frustration, loss of credibility, and dropping the conversation [21, 50, 51].”
BackgroundHoque, M. N., Mahfuz, A. A., Kindi, M. S., & Hassan, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642007 · Cited by 17
“We analyzed their responses using a grounded theory approach [22] and a set of quantitative measures.”
BackgroundMethodInfluentialAdapting a Conversational Text Generator for Online Chatbot Messaging
Klopfenstein, L. C., Delpriori, S., & Ricci, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-17705-8_8 · Cited by 16
“While most chatbots and virtual assistant promise a natural and human-like interaction, they often fail at clearly revealing their effective capabilities and in correctly processing the full context of a conversation [16].”
BackgroundStrathmann, C., Szczuka, J. M., & Krämer, N. C. (2020) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3383652.3423906 · Cited by 21
“Th e aspect of language can be discussed in relation to the fi ndings of Luger and Sellen [18].”
BackgroundLosing Its Touch: Understanding User Perception of Multimodal Interaction and Smart Assistance
Esau, M., Krauß, V., Lawo, D., & Stevens, G. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533455 · Cited by 18
“This specifcity clashes with infated hopes built up by the use of natural language as an interface typical for human-human conversation [22].”
BackgroundMethodInfluentialKim, H., Jung, I., & Lim, Y.-K. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533528 · Cited by 22
Human-AI Co-Learning for Data-Driven AIPreprint
Huang, Y., Cheng, Y.-T., Chen, L., & Hsu, J. Y.-J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12544 · Cited by 18
“Building trust through continuous feedback and adaptation : People usually cannot trust AI systems because of their uncertainty, biases, and failure [13].”
BackgroundChan, S., Li, J., Yao, B., Mahmood, A., Huang, C.-M., Jimison, H., Mynatt, E. D., & Wang, D. (2023) · arXiv · doi:10.48550/arxiv.2310.05853 · Cited by 16
Do Smart Speakers Respond to Their Errors Properly? A Study on Human-Computer Dialogue Strategy
Ge, X., Li, D., Guan, D., Xu, S., Sun, Y., & Zhou, M. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-23541-3_32 · Cited by 15
The Use of Smart Home Speakers by the Elderly: Exploratory Analyses and Potential for Big Data
Purao, S., Hao, H., & Meng, C. (2021) · Big Data Research · doi:10.1016/j.bdr.2021.100224 · Cited by 15
Conversational Error Analysis in Human-Agent Interaction
Aneja, D., McDuff, D., & Czerwinski, M. (2020) · Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents · doi:10.1145/3383652.3423901 · Cited by 15
Garg, R., Cui, H., & Kapadia, Y. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479495 · Cited by 15
“, they find that the actual functionalities offered by these devices differ from their expectations [57], and their use changes due to privacy concerns [31] and due to the fact that the smart speakers might have trouble understanding their accent or non-English words such as place names [66, 71].”
BackgroundDesigning with Multi-Agent Generative AI: Insights from Industry Early Adopters
Naik, S., Toombs, A. L., Snellinger, A., Saponas, S., & Hall, A. K. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735823 · Cited by 22
“Research in human-agent interaction underscores the importance of this transparency, trust, and explainability in agent behaviors [30].”
Imagining future digital assistants at work: A study of task management needs
Khaokaew, Y., Holcombe-James, I., Rahaman, M. S., Liono, J., Trippas, J. R., Spina, D., Bailey, P., Belkin, N. J., Bennett, P. N., Ren, Y., et al. (2022) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2022.102905 · Cited by 19
Sharma, H., & Sharma, D. (2025) · Marketing Intelligence & Planning · doi:10.1108/mip-06-2023-0268 · Cited by 15
Family-Centered Exploration of the Benefits and Burdens of Digital Home Assistants
Richards, O. K. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3308458 · Cited by 23
“Luger et al. studied user expectations of conversational agents and found user expectations to be far more complex than the rather simple operation of the systems, especially regarding known machine intelligence, system capability and goals [5].”
BackgroundChallenges in Supporting Exploratory Search through Voice Assistants
Ma, X., & Liu, A. (2020) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3405755.3406152 · Cited by 16
“For example, a user asks the voice assistant for ideas for restaurants on Valentine’s Day, with constraints on location, budget, availability, and ambience that haven’t been thought-through yet [9].”
BackgroundInfluentialJang, Y.-B. (2020) · Archives of Design Research · doi:10.15187/adr.2020.08.33.3.127 · Cited by 22
Do people intend to use AI Voice Assistants? An empirical study in Vietnam
Thi, T. D. P., & Duong, N. T. (2022) · Journal of Human Behavior in the Social Environment · doi:10.1080/10911359.2022.2106003 · Cited by 15
“At the application level of business, “voice commerce” is another milestone in the development of voice assistants (Chopra & Chivukula, 2017; Luger & Sellen, 2016).”
BackgroundSmart Home Personal Assistants: Fueled by Natural Language Processor and Blockchain Technology
Ansar, S. A., Jaiswal, K., Aggarwal, S., Shukla, S., Yadav, J., & Soni, N. (2022) · International Conference on Interdisciplinary Cyber Physical Systems (ICPS) · doi:10.1109/icps55917.2022.00029 · Cited by 16
“This allows for creative engagement, which makes them more interesting and fun to use [9].”
Background
Metadata
- Authors
- Luger, E., & Sellen, A.
- Affiliations
- Microsoft Research (United Kingdom) (GB)
- Venue
- Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2016-05-07
- Pages
- 5286-5297
- OpenAlex
- W2405187948
- Semantic Scholar
- 814e62bb4f0ec6facd640fbd3e3fcbd189716d20
- Cited by
- 1,176
- References
- 51
- Citations by year
- 2026: 164 · 2025: 175 · 2024: 119 · 2023: 116 · 2022: 107 · 2021: 148 · 2020: 141 · 2019: 126 · 2018: 54 · 2017: 25 · 2016: 1
- FWCI
- 37.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- chatbots, voice assistants, everyday use, user expectations, human-computer interaction, multimodality, system capabilities, interface design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.