Improving Usability Perception of Error-Prone AI Speakers: Elaborated Feedback Mitigates Negative Consequences of Errors
The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Users of voice user interface (VUI) often encounter errors, such as when a VUI attempts to recognize a user’s voice inputs or execute tasks. Conversation is prone to errors, and in the collaborative perspective, communicators manage common ground together to handle erroneous situations. Adopting a collaborative view of conversation, we propose that a VUI can address different types of errors by providing users with feedback to aid them in developing common ground to communicate more effectively. To test this proposal, we conducted a 2 (error type: recognition vs. execution error) × 2 (feedback elaboration: present vs. absent) mixed-design experiment in which users interacted with a VUI speaker and evaluated its usability in these four modes. Participants reported greater acceptance of feedback and higher usability perception for a speaker returning execution errors than for one returning recognition errors, particularly when the speaker presented feedback articulating reasons for the errors. This finding indicates that a VUI can employ feedback explaining the causes of errors to facilitate the development of common ground and to minimize the negative consequences of errors.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Specifically, in the personal assistance domain, individual studies focused on utilising either computer speakers [53, 54, 57, 67, 79], mobile devices [50] or smart speakers [26, 40].”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [40]
In the citing paper
“Such a recovery strategy appeared in only three of the reviewed studies [36, 40, 75].”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [40]
In the citing paper
“Similarly, [40] employed a repair strategy known as 'elaborated feedback' to develop common ground (e.g. 'I don't understand what you said. It is noisy around here.'), which positively influenced user acceptance and usability of VUIs.”
From System and User Strategies to Repair Conversational Breakdowns of Spoken Dialogue Systems: A Scoping Review (2024) · cites this paper as [40]
Citations
Cites
Perceived usefulness, perceived ease of use, and user acceptance of information technology
Davis, F. D. (1989) · MIS Quarterly · doi:10.2307/249008 · Cited by 68,483 · cited 1×
“Perceived usefulness was measured by four items adopted from Davis (1989).”
MethodClark, H. H., & Brennan, S. E. (1991) · Perspectives on socially shared cognition. · doi:10.1037/10096-006 · Cited by 4,555 · cited 3×
“This proposal is based on the collaborative view of conversation (Clark & Brennan, 1991), which regards conversation as a recursive process of building mutual understanding, or common ground , developed by its interlocutors.”
BackgroundMethodVenkatesh, V., & Morris, M. G. (2000) · MIS Quarterly · doi:10.2307/3250981 · Cited by 2,780 · cited 1×
“Perceived ease of use Perceived ease of use was assessed using four items adopted from Venkatesh and Morris (2000).”
MethodEmotions in the Workplace: Research, Theory, and Practice
Ashkanasy, N. M., Härtel, C. E. J., & Zerbe, W. J. (2000) · Cited by 688
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
(2009) · doi:10.1145/1518701 · Cited by 570
Portet, F., Vacher, M., Golanski, C., Roux, C., & Meillon, B. (2013) · Personal and Ubiquitous Computing · doi:10.1007/s00779-011-0470-5 · Cited by 421 · cited 1×
“…VUI and role of feedback VUI has the potential to increase the usability of a computer system in many domains, including home applications (Portet, Vacher, Golanski, Roux, & Meillon, 2013), mobile device (Capra & Pérez-Quiñones, 2005), television (Bures, 2012), and devices in a vehicle…”
BackgroundCollaborating on contributions to conversations
Clark, H. H., & Schaefer, E. F. (1987) · Language and Cognitive Processes · doi:10.1080/01690968708406350 · Cited by 371 · cited 1×
“Further detailing the category for errors in human – human communication (Clark & Schaefer, 1987), they defined eight subcategories of causes of errors: user input not entered into a system (Level 0), speech recognition failing to recognize and interpret user inputs (Level 1 – 3), contextual…”
BackgroundPhonetic differences between male and female speech
Simpson, A. P. (2009) · Language and Linguistics Compass · doi:10.1111/j.1749-818x.2009.00125.x · Cited by 300 · cited 1×
“Speech has distinctive phonetic characteristics, such as volume, pitch, and frequency, and such variations are related to gender (Simpson, 2001, 2009) and age differences (Benjamin, 1981) in human vocal organs.”
BackgroundAgents that Reduce work and information Overload
Maes, P. (1995) · Readings in Human–Computer Interaction · doi:10.1016/b978-0-08-051574-8.50084-4 · Cited by 221
The role of voice input for human-machine communication.
Cohen, P. R., & Oviatt, S. L. (1995) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.92.22.9921 · Cited by 203 · cited 1×
“These advantages include controls that are fast and easy-to-learn (Ruan, Wobbrock, Liou, Ng, & Landay, 2018) and the unleashing of eyes and hands to be used for other tasks (Cohen & Oviatt, 1995).”
MethodComparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phones
Ruan, S., Wobbrock, J. O., Liou, K., Ng, A., & Landay, J. A. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161187 · Cited by 187 · cited 1×
“These advantages include controls that are fast and easy-to-learn (Ruan, Wobbrock, Liou, Ng, & Landay, 2018) and the unleashing of eyes and hands to be used for other tasks (Cohen & Oviatt, 1995).”
MethodEffects of Voice Technology on Test Track Driving Performance: Implications for Driver Distraction
Ranney, T. A., Harbluk, J. L., & Noy, Y. I. (2005) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/0018720054679515 · Cited by 158 · cited 1×
“…VUI has the potential to increase the usability of a computer system in many domains, including home applications (Portet, Vacher, Golanski, Roux, & Meillon, 2013), mobile device (Capra & Pérez-Quiñones, 2005), television (Bures, 2012), and devices in a vehicle (Ranney, Harbluk, & Noy, 2005).”
BackgroundInteraction and feedback in a spoken language system: a theoretical framework
Brennan, S. E., & Hulteen, E. A. (1995) · Knowledge-Based Systems · doi:10.1016/0950-7051(95)98376-h · Cited by 148 · cited 1×
“In this view, the occurrence of errors in a conversation is part of an ordinary process, and the inter-locutors adaptively address errors developing common ground on why the errors occur and how to handle the error situation (Brennan & Hulteen, 1995).”
BackgroundSyntactic variation and dialect divergence
Harris, J. (1984) · Journal of Linguistics · doi:10.1017/s0022226700013918 · Cited by 133 · cited 1×
“Furthermore, people may structure sentences in different ways, generating variations in syntactics (Cheshire, 2005; Harris, 1984).”
BackgroundSemantic variation and sociolinguistics
Hasan, R. (1989) · Australian Journal of Linguistics · doi:10.1080/07268608908599422 · Cited by 125 · cited 1×
“Although a lexical definition of a word might be unique, its cultural interpretation may vary with social and cultural structures (Hasan, 1989).”
BackgroundDynamic consequences of differences in male and female vocal tract dimensions
Simpson, A. P. (2001) · The Journal of the Acoustical Society of America · doi:10.1121/1.1356020 · Cited by 112 · cited 1×
“Speech has distinctive phonetic characteristics, such as volume, pitch, and frequency, and such variations are related to gender (Simpson, 2001, 2009) and age differences (Benjamin, 1981) in human vocal organs.”
BackgroundCheshire, J. (2005) · Journal of Sociolinguistics · doi:10.1111/j.1360-6441.2005.00303.x · Cited by 108 · cited 1×
“Furthermore, people may structure sentences in different ways, generating variations in syntactics (Cheshire, 2005; Harris, 1984).”
BackgroundFrequency Variability in the Aged Voice
Benjamin, B. J. (1981) · Journal of Gerontology · doi:10.1093/geronj/36.6.722 · Cited by 102 · cited 1×
“Speech has distinctive phonetic characteristics, such as volume, pitch, and frequency, and such variations are related to gender (Simpson, 2001, 2009) and age differences (Benjamin, 1981) in human vocal organs.”
BackgroundProsodic Variation in Southern British English
Peppé, S. J. E., Maxim, J., & Wells, B. (2000) · Language and Speech · doi:10.1177/00238309000430030501 · Cited by 51
Demographic and Socioeconomic Basis of Ethnolinguistics
Siegel, J. S. (2017) · Springer eBooks · doi:10.1007/978-3-319-61778-7 · Cited by 14 · cited 2×
“Spoken language used in VUI is much more flexible and varied in its contents and form compared to key strokes and mouse con-trol (Siegel, 2018), and tasks performed by a VUI are often contextualized, all of which contribute to VUIs ’ generating unexpected results in interactions with users…”
BackgroundA pattern language for error management in voice user interfaces
Schnelle-Walka, D. (2010) · European Conference on Pattern Languages of Programs · doi:10.1145/2328909.2328920 · Cited by 13 · cited 1×
“Duff, Gates, and LuperFoy (1996), for example, suggested detailed stages of processing users ’ voice inputs, each having the potential to return errors (see also Schnelle-Walka, 2010).”
BackgroundMobile refinding of web information using a voice interface
Capra, R., & Pérez-Quiñones, M. A. (2005) · doi:10.1145/1111360.1111369 · Cited by 13 · cited 1×
“…VUI has the potential to increase the usability of a computer system in many domains, including home applications (Portet, Vacher, Golanski, Roux, & Meillon, 2013), mobile device (Capra & Pérez-Quiñones, 2005), television (Bures, 2012), and devices in a vehicle (Ranney, Harbluk, & Noy, 2005).”
BackgroundBureš, V. (2012) · Elektronika ir Elektrotechnika · doi:10.5755/j01.eee.122.6.1827 · Cited by 9 · cited 1×
“…VUI has the potential to increase the usability of a computer system in many domains, including home applications (Portet, Vacher, Golanski, Roux, & Meillon, 2013), mobile device (Capra & Pérez-Quiñones, 2005), television (Bures, 2012), and devices in a vehicle (Ranney, Harbluk, & Noy, 2005).”
BackgroundAn architecture for spoken dialogue management
Duff, D., Gates, B., & LuperFoy, S. (1996) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1996-270 · Cited by 8
Duff, D., Gates, B., & LuperFoy, S. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.607778 · Cited by 7 · cited 2×
“…subcategories of causes of errors: user input not entered into a system (Level 0), speech recognition failing to recognize and interpret user inputs (Level 1 – 3), contextual constraints preventing implementation of user commands (Level 4 – 6), and unintended results (Level 7) (Duff et al., 1996).”
BackgroundKoh, Y., & Kwahk, J. (2017) · The Japanese Journal of Ergonomics · doi:10.5100/jje.53.s408 · Cited by 6 · cited 4×
“Using these categories, they examined how participants behaved when confronting different types of errors, and found that participants changed the phonetics (e.g., tone) or syntactics (e.g., sentence) of their vocal inputs, depending on the types of error they encountered (Koh & Kwahk, 2017).”
BackgroundResultInfluential
Research works printed but not linked: 1
- Clark H. H., 1999, Proceedings of 1999 AAAI Fall Symposium, Psychological Models of Communication in Collaborative Systems, 43
Cited by
Alghamdi, E., Halvey, M., & Nicol, E. (2024) · ACM Conversational User Interfaces 2024 · doi:10.1145/3640794.3665558 · Cited by 36
“A few studies were carried out in other languages and regions, including China [29, 75], Sweden [22, 67], UK [60], Brazil [50], Denmark [17], South Korea [40] and Taiwan [42].”
BackgroundMethodInfluentialKim, J., Jeong, M., & Lee, S. C. (2019) · Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings · doi:10.1145/3349263.3351513 · Cited by 29
Self-citationWhy College Students Prefer Typing Over Speech Input: The Dual Perspective
Tsai, L. L. (2021) · IEEE Access · doi:10.1109/access.2021.3107457 · Cited by 8
“Examples include the user encountering execution errors and the input device may experience recognition errors [94].”
BackgroundVOICON: Geometric Motion-Based Visual Feedback in Voice User Interface
Oh, J., Kim, N. S., Yan, Y., & Lee, S. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3660741 · Cited by 4
Self-citationMilanović, K., & Pitt, J. V. (2021) · Designing Interactive Systems Conference · doi:10.1145/3461778.3462043 · Cited by 4
“In HCI, in particular with interactions with voice activated virtual assistants (VAVAs) error research is currently focusing on improving user experience through explainability [18].”
Background
Metadata
- Authors
- Lee, D., Sah, Y. J., & Lee, S.
- Affiliations
- Sungkyunkwan University (KR)
- Department of Interaction Science, Sungkyunkwan University, Seoul, South Korea
- ORCID
- Sah, Young June
- Venue
- International Journal of Human–Computer Interaction
- Publisher
- Informa UK Limited
- Published
- 2019-01-07
- Volume
- 35
- Issue
- 17
- Pages
- 1645-1652
- ISSN
- 1044-7318 (ISSN-L)
- 1532-7590
- OpenAlex
- W2909278029
- Cited by
- 9
- References
- 27
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 2 · 2022: 1 · 2021: 3 · 2019: 1
- FWCI
- 0.58
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Human-Computer Interaction (publishedVersion)
- Funders
- National Research Foundation of Korea (NRF-2017R1C 1B 1003650)
- Ministry of Science and ICT, Korea (2015-0-00914)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- voice user interface, error feedback, perceived usability, common ground, collaborative dialogue, speech recognition errors, error explanation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.