Assessing the Utility of the System Usability Scale for Evaluating Voice-based User Interfaces
The DOI matches its Crossref record; 10 of the 11 references it cites were found in a registry or an index.
Abstract
Voice-based User Interfaces (VUIs) challenge our existing conceptions of usability since the standardized evaluation tools we use were typically developed for interfaces with visual feedback, whereas VUI's have predominantly eyes-free interactions. We experimented with the use of a well-validated tool, the System Usability Scale (SUS), to evaluate two existing, commercially available VUIs. We administered the SUS to 12 participants after they completed a set of scenario tasks on Amazon's Alexa and Apple's Siri. The results were consistent with previous studies comparing subjective rating scores and SUS, suggesting that the SUS is a valid evaluation tool for VUIs. Additionally, despite large, significant differences in adjective scale ratings and SUS scores, both systems performed similarly on the learnability items. We conclude with recommendations on the use of the SUS for evaluating VUIs.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“SUS is commonly utilized to measure the usability of GUIs, but its validity for VUIs has been successfully tested [21].”
From Using Experience-Based Participatory Approach to Design Interactive Voice User Interfaces for Delivering Physical Activity Programs with Older Adults (2023) · cites this paper as [21]
In the citing paper
“For VUIs, SUS has been successfully tested for face validity [28], and face and construct validity for a specifc use scenario [10].”
From “A Painless Way to Learn:” Designing an Interactive Storytelling Voice User Interface to Engage Older Adults in Informal Health Information Learning (2023) · cites this paper as [28]
Citations
Cites
Constructing validity: Basic issues in objective scale development.
Clark, L. A., & Watson, D. (1995) · Psychological Assessment · doi:10.1037/1040-3590.7.3.309 · Cited by 6,567
Norman, D. (2016) · Vahlen eBooks · doi:10.15358/9783800648108 · Cited by 6,374
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 4×
“…to argue for the SUS being the appropriate tool for cross-system comparisons - it has been validated across a wide range of interfaces and it has been shown to have strong inter-rater reliability, item sensitivity and semantically meaningful differences between high scores and low scores [1,2].”
BackgroundResultInfluentialThe Factor Structure of the System Usability Scale
Lewis, J. R., & Sauro, J. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02806-9_12 · Cited by 1,278
Luger, E., & Sellen, A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858288 · Cited by 1,176
Scale Construction: Developing Reliable and Valid Measurement Instruments
Hinkin, T. R., Tracey, J. B., & Enz, C. A. (1997) · Journal of Hospitality & Tourism Research · doi:10.1177/109634809702100108 · Cited by 819
Voice Interfaces in Everyday Life
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174214 · Cited by 717 · cited 2×
“[11], in listing the various issues associated with everyday interactions with VUIs, suggest also that far more can be done to increase the mutual intelligibility of VUIs and the users operating the device.”
BackgroundMethodDetermining what individual SUS scores mean
(2009) · Journal of Usability Studies archive · doi:10.5555/2835587.2835589 · Cited by 412
The Rise of Bots: A Survey of Conversational Interfaces, Patterns, and Paradigms
Klopfenstein, L. C., Delpriori, S., Malatini, S., & Bogliolo, A. (2017) · Proceedings of the 2017 Conference on Designing Interactive Systems · doi:10.1145/3064663.3064672 · Cited by 252
Ghosh, D., Foong, P. S., Zhao, S., Chen, D., & Fjeld, M. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173977 · Cited by 18
Self-citation
Research works printed but not linked: 1
- Nicole M. Radziwill and Morgan C. Benton. 2017. Evaluating Quality of Chatbots and Intelligent Conversational Agents. arXiv preprint arXiv:1704.04579. Nicole M. Radziwill and Morgan C. Benton. 2017. Evaluating Quality of Chatbots and Intelligent Conversational Agents. arXiv preprint arXiv:1704.04579. · arXiv
Cited by
Gronier, G., & Baudet, A. (2021) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2021.1898828 · Cited by 237
Usability Evaluation of Artificial Intelligence-Based Voice Assistants: The Case of Amazon Alexa
Zwakman, D. S., Pal, D., & Arpnikanondt, C. (2021) · SN Computer Science · doi:10.1007/s42979-020-00424-4 · Cited by 123
“Usability of Alexa and Siri is investigated by authors in [38] using the SUS questionnaire.”
MethodMyers, C. M., Furqan, A., & Zhu, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300277 · Cited by 58
“For comparison, Siri received a mean score of 54.167 ± 15.715 and Alexa a 84.792 ± 9.26 in a recent study [8].”
BackgroundInfluentialPyae, A., & Scifleet, P. (2018) · Proceedings of the 30th Australian Conference on Computer-Human Interaction · doi:10.1145/3292147.3292236 · Cited by 38
“We used the System Usability Scales (SUS) [14] and the User Engagement Scale (UES) [15] with 5-point Likert scales.”
MethodMildner, T., Cooney, O., Meck, A.-M., Bartl, M., Savino, G.-L., Doyle, P. R., Garaialde, D., Clark, L., Sloan, J., Wenig, N., et al. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642542 · Cited by 53
“Ghosh et al. [33] highlights a lack of accuracy for the system usability score [13, 14] — a prominent measure of subjective perceptions of usability of systems – when used to evaluate CUIs.”
What Can I Say? Effects of Discoverability in VUIs on Task Performance and User Experience
Kirschthaler, P., Porcheron, M., & Fischer, J. E. (2020) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3405755.3406119 · Cited by 40
Agente conversacional virtual: la inteligencia artificial para el aprendizaje autónomo
Rodríguez, J. A., Santana, M. G., Perera, V. A., & Pulido, J. R. (2021) · Pixel-Bit Revista de Medios y Educación · doi:10.12795/pixelbit.86171 · Cited by 73
Bräuer, P., & Mazarakis, A. (2022) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107362 · Cited by 42
Daphne: A Virtual Assistant for Designing Earth Observation Distributed Spacecraft Missions
Martin, A. V., & Selva, D. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2948921 · Cited by 33
“The SUS instrument has been validated and has been widely used to assess software usability including for intelligent systems [91]–[93].”
MethodDesai, S., Lundy, M., & Chin, J. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3597141 · Cited by 25
“For VUIs, SUS has been successfully tested for face validity [28], and face and construct validity for a specifc use scenario [10].”
MethodA Review of Subjective Scales Measuring the User Experience of Voice Assistants
Faruk, L. I. D., Babakerkhell, M. D., Mongkolnam, P., Chongsuphajaisiddhi, V., Funilkul, S., & Pal, D. (2024) · IEEE Access · doi:10.1109/access.2024.3358423 · Cited by 32
“For instance, 24 articles were removed because they were on scale development, but not either on VA and voice was not a major component in the interaction [24], [60].”
BackgroundFaruk, L. I. D., Pal, D., Funilkul, S., Perumal, T., & Mongkolnam, P. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2359206 · Cited by 34
“For example, (Br € uggemeier et al., 2020; Ghosh et al., 2018; Zwakman et al., 2020) have used SUS for evaluation of voice UX.”
Can digital personal assistants persuade people to exercise?
Paay, J., Kjeldskov, J., Papachristos, E., Hansen, K. M., Jørgensen, T. B., & Overgaard, K. L. (2020) · Behaviour and Information Technology · doi:10.1080/0144929x.2020.1814412 · Cited by 27
Usability Questionnaires to Evaluate Voice User Interfaces
Iñiguez-Carrillo, A. L., Gaytan-Lugo, L. S., García-Ruiz, M. Á., & Arellano, R. M. (2021) · IEEE Latin America Transactions · doi:10.1109/tla.2021.9468439 · Cited by 21
Desai, S., Hu, X., Lundy, M., & Chin, J. (2023) · International Conference on Human-Agent Interaction · doi:10.1145/3623809.3623827 · Cited by 13
Measuring Perceived Usability in Chinese Questionnaires: mTAM, SUS, and UMUX
Wang, Y., & Wang, H. (2021) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2021.1979291 · Cited by 15
“, 2015), video service systems (Huang, 2020), mobile applications (Kortumand & Sorber, 2015), online education platforms (Pal & Vanijja, 2020), Web applications (Ghosh et al., 2018; Shorman et al., 2021), and more.”
BackgroundWang, J., Yang, S., & Xu, Z. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1119355 · Cited by 12
Oduor, M., & Perälä, T. (2021) · Frontiers in Computer Science · doi:10.3389/fcomp.2021.706162 · Cited by 11
“Previous studies have found correlations of 0.822 Bangor, Kortum, and Miller (2009), 0.94 Ghosh et al. (2018), and 0.848 Grier et al. (2013).”
InfluentialMusto, C., Narducci, F., Polignano, M., de Gemmis, M., Lops, P., & Semeraro, G. M. (2021) · ACM Transactions on Information Systems · doi:10.1145/3447679 · Cited by 14
“…[Jin et al. 2019; Mah-mood and Ricci 2009; Tsumita and Takagi 2019] and subjective metrics such as understandability, user-friendliness, usability score on the System Usability Scale (SUS) , user control, trans-parency [Ghosh et al. 2018; Jin et al. 2019; Li et al. 2017; Pecune et al. 2019].”
BackgroundUna experiencia innovadora a través del podcast en la educación superior
Mulero-Henríquez, I., Bolaños, A. M. T. Á., & Cobos, M. P. D. L. (2024) · Formación universitaria · doi:10.4067/s0718-50062024000100023 · Cited by 32
Are CUIs Just GUIs with Speech Bubbles?
Simpson, J. (2020) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3405755.3406143 · Cited by 9
“However, a challenge of applying such an approach to conversational interfaces is the difficulty of phrasing task descriptions so that users cannot simply repeat the instructions back to the interface (e.g. [4,8]).”
BackgroundKaushik, A., Jacob, B., & Velavan, P. (2022) · Knowledge · doi:10.3390/knowledge2010007 · Cited by 8
Emerging mobile apps: challenges and open problems
Alrumayh, A. S., Lehman, S. M., & Tan, C. C. (2021) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-020-00055-x · Cited by 12
“Some of the more common questionnaires for overall system design include the Mean Opinion Scale (MOS) (Polkosky and Lewis 2003), System Usability Scale (SUS) (Ghosh et al. 2018), (Usability.gov.”
BackgroundVORI: A framework for testing voice user interface interactability
Alrumayh, A. S., & Tan, C. C. (2022) · High-Confidence Computing · doi:10.1016/j.hcc.2022.100069 · Cited by 6
Usability Analysis on Digital Library Information System using System Usability Scale (SUS)
Pratiwi, P. Y., Pradnyana, I. M. A., & Damayanti, N. K. W. (2023) · International Conference on Electrical and Information Technology (IEIT) · doi:10.1109/ieit59852.2023.10335582 · Cited by 5
“Lastly, its employment of neutral technology enables the assessment of a diverse array of user interfaces, encompassing websites, mobile phones, and interactive voice response systems [8][9].”
BackgroundBarzilai, G., & Rampino, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-49760-6_2 · Cited by 5
Lee, M., & Lee, S. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451812 · Cited by 5
“Developed by Brooke, SUS allows the usability practitioner to assess the usability of a given product or service quickly and easily [1, 4, 16] and is a valid evaluation tool for VUIs [7].”
BackgroundUrban Play and the Playable City: A Critical Perspective
Chisik, Y., Schouten, B., Thibault, M., & Nijholt, A. (2022) · Frontiers research topics · doi:10.3389/978-2-88974-422-0 · Cited by 5
Valente, P., Fornelos, T., Ferreira, R., Silva, D., Tavares, D., Correia, N. M. R., Magalhães, J., & Nóbrega, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42283-6_15 · Cited by 4
Pyae, A., Hlaing, S. Z., Aung, N. T. T., Kham, N. M. M., Khant, M. M., & Kyaw, M. K. (2020) · International Conference on Advanced Information Technologies (ICAIT) · doi:10.1109/icait51105.2020.9261771 · Cited by 4
“According to [27], SUS is a valid and reliable tool for evaluating VUIs.”
MethodKaushik, A. (2021) · Dublin City University Open Access Institutional Repository (Dublin City University) · Cited by 4
Architecture to Design Booking Appointment Applications for the Smart Personal Assistant Alexa
Meza, A., López, G., Quesada, L., & Guerrero, L. A. (2019) · International Conference on Ubiquitous Computing and Ambient Intelligence (UCAmI) · doi:10.3390/proceedings2019031017 · Cited by 2
“The instrument uses the SUS system (System Usability Scale) as base adapted for the evaluation of user interfaces by voice, as this system has been proved to be appropriate for evaluating VUIs, offering both convergent and concurrent validity [24].”
MethodExploring Audio Icons for Content-Based Navigation in Voice User Interfaces
Jensen, J. K., & Ashbrook, D. (2023) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3571884.3604302 · Cited by 2
“Conditions 2 and 3 achieved higher SUS scores, placing them in the top 15% of SUS evaluated interaction system designs [17].”
BackgroundUsability Analysis of Smart Speakers from a Learnability Perspective for Novel Users
Doi, T., & Nishikawa, Y. (2024) · Applied System Innovation · doi:10.3390/asi7030036 · Cited by 3
“In particular, studies on question items based on the System Usability Scale (SUS), commonly used in usability evaluation [31,32], have been conducted for subjective evaluation.”
Qu, N. Z., Li, J. J., & Chignell, M. (2025) · AI & Society · doi:10.1007/s00146-025-02688-x
Qu, N. Z., Li, J. J., & Chignell, M. (2026) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2026.100291
Lopatovska, I., & Murad, C. (2026) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3816046.3816270
“While large companies do not share their large-scale user experience (UX) questionnaires with the voice interface (VUI) and CUI research communities, there has been some exploration of the use of the SUS [3] for VUIs [7] in academia.”
Metadata
- Authors
- Ghosh, D., Foong, P. S., Zhang, S., & Zhao, S.
- Affiliations
- National University of Singapore (SG)
- Venue
- Proceedings of the Sixth International Symposium of Chinese CHI
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 11-15
- OpenAlex
- W2805804898
- Cited by
- 36
- References
- 11
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 5 · 2023: 5 · 2022: 4 · 2021: 10 · 2020: 5 · 2019: 3 · 2018: 1
- FWCI
- 3.83
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Symposium of Chinese CHI (Chinese CHI) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- System Usability Scale, usability evaluation, Amazon Alexa, Siri, eyes-free interaction, learnability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.