Assessing the Utility of the System Usability Scale for Evaluating Voice-based User Interfaces
The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
Abstract
Voice-based User Interfaces (VUIs) challenge our existing conceptions of usability since the standardized evaluation tools we use were typically developed for interfaces with visual feedback, whereas VUI's have predominantly eyes-free interactions. We experimented with the use of a well-validated tool, the System Usability Scale (SUS), to evaluate two existing, commercially available VUIs. We administered the SUS to 12 participants after they completed a set of scenario tasks on Amazon's Alexa and Apple's Siri. The results were consistent with previous studies comparing subjective rating scores and SUS, suggesting that the SUS is a valid evaluation tool for VUIs. Additionally, despite large, significant differences in adjective scale ratings and SUS scores, both systems performed similarly on the learnability items. We conclude with recommendations on the use of the SUS for evaluating VUIs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“SUS is commonly utilized to measure the usability of GUIs, but its validity for VUIs has been successfully tested [21].”
From Using Experience-Based Participatory Approach to Design Interactive Voice User Interfaces for Delivering Physical Activity Programs with Older Adults (2023) · cites this paper as [21]
In the citing paper
“For VUIs, SUS has been successfully tested for face validity [28], and face and construct validity for a specifc use scenario [10].”
From “A Painless Way to Learn:” Designing an Interactive Storytelling Voice User Interface to Engage Older Adults in Informal Health Information Learning (2023) · cites this paper as [28]
Citations
Cites
Constructing validity: Basic issues in objective scale development.
Clark, L. A., & Watson, D. (1995) · Psychological Assessment · doi:10.1037/1040-3590.7.3.309 · Cited by 6,567
Norman, D. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,374
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 4×
“…to argue for the SUS being the appropriate tool for cross-system comparisons - it has been validated across a wide range of interfaces and it has been shown to have strong inter-rater reliability, item sensitivity and semantically meaningful differences between high scores and low scores [1,2].”
BackgroundResultInfluentialThe Factor Structure of the System Usability Scale
Lewis, J. R., & Sauro, J. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02806-9_12 · Cited by 1,278
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Scale Construction: Developing Reliable and Valid Measurement Instruments
Hinkin, T. R., Tracey, J. B., & Enz, C. A. (1997) · Journal of Hospitality & Tourism Research · doi:10.1177/109634809702100108 · Cited by 819
Voice Interfaces in Everyday Life
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174214 · Cited by 717 · cited 2×
“[11], in listing the various issues associated with everyday interactions with VUIs, suggest also that far more can be done to increase the mutual intelligibility of VUIs and the users operating the device.”
BackgroundMethodDetermining what individual SUS scores mean
(2009) · Journal of Usability Studies archive · doi:10.5555/2835587.2835589 · Cited by 412
The Rise of Bots: A Survey of Conversational Interfaces, Patterns, and Paradigms
Klopfenstein, L. C., Delpriori, S., Malatini, S., & Bogliolo, A. (2017) · Proceedings of the 2017 Conference on Designing Interactive Systems · doi:10.1145/3064663.3064672 · Cited by 252
Ghosh, D., Foong, P. S., Zhao, S., Chen, D., & Fjeld, M. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173977 · Cited by 18
Self-citation
Research works printed but not linked: 1
- Nicole M. Radziwill and Morgan C. Benton. 2017. Evaluating Quality of Chatbots and Intelligent Conversational Agents. arXiv preprint arXiv:1704.04579. Nicole M. Radziwill and Morgan C. Benton. 2017. Evaluating Quality of Chatbots and Intelligent Conversational Agents. arXiv preprint arXiv:1704.04579. · arXiv
Cited by
Gronier, G., & Baudet, A. (2021) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2021.1898828 · Cited by 237
Usability Evaluation of Artificial Intelligence-Based Voice Assistants: The Case of Amazon Alexa
Zwakman, D. S., Pal, D., & Arpnikanondt, C. (2021) · SN Computer Science · doi:10.1007/s42979-020-00424-4 · Cited by 123
“Usability of Alexa and Siri is investigated by authors in [38] using the SUS questionnaire.”
MethodMyers, C. M., Furqan, A., & Zhu, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300277 · Cited by 58
“For comparison, Siri received a mean score of 54.167 ± 15.715 and Alexa a 84.792 ± 9.26 in a recent study [8].”
BackgroundInfluentialPyae, A., & Scifleet, P. (2018) · Proceedings of the 30th Australian Conference on Computer-Human Interaction · doi:10.1145/3292147.3292236 · Cited by 38
“We used the System Usability Scales (SUS) [14] and the User Engagement Scale (UES) [15] with 5-point Likert scales.”
MethodMildner, T., Cooney, O., Meck, A.-M., Bartl, M., Savino, G.-L., Doyle, P. R., Garaialde, D., Clark, L., Sloan, J., Wenig, N., et al. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642542 · Cited by 53
“Ghosh et al. [33] highlights a lack of accuracy for the system usability score [13, 14] — a prominent measure of subjective perceptions of usability of systems – when used to evaluate CUIs.”
Metadata
- Authors
- Ghosh, D., Foong, P. S., Zhang, S., & Zhao, S.
- Affiliations
- National University of Singapore (SG)
- Venue
- Proceedings of the Sixth International Symposium of Chinese CHI
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 11-15
- OpenAlex
- W2805804898
- Cited by
- 36
- References
- 11
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 5 · 2023: 5 · 2022: 4 · 2021: 10 · 2020: 5 · 2019: 3 · 2018: 1
- FWCI
- 3.83
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium of Chinese CHI (Chinese CHI) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- System Usability Scale, usability evaluation, Amazon Alexa, Siri, eyes-free interaction, learnability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.