Measuring Voice UX Quantitatively: A Rapid Review
Open accessThe DOI matches its Crossref record; 52 of the 56 references it cites were found in a registry or an index.
Abstract
Computer voice is experiencing a renaissance through the growing popularity of voice-based interfaces, agents, and environments. Yet, how to measure the user experience (UX) of voice-based systems remains an open and urgent question, especially given that their form factors and interaction styles tend to be non-visual, intangible, and often considered disembodied or “body-less.” As a first step, we surveyed the ACM and IEEE literatures to determine which quantitative measures and measurements have been deemed important for voice UX. Our findings show that there is little consensus, even with similar situations and systems, as well as an overreliance on lab work and unvalidated scales. In response, we offer two high-level descriptive frameworks for guiding future research, developing standardized instruments, and informing ongoing review work. Our work highlights the current strengths and weaknesses of voice UX research and charts a path towards measuring voice UX in a more comprehensive way.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Although popularly used in VUI literature [57], none of these scales were developed for VUIs, and their validity for VUIs remains limited.”
From “A Painless Way to Learn:” Designing an Interactive Storytelling Voice User Interface to Engage Older Adults in Informal Health Information Learning (2023) · cites this paper as [57]
Experimental — accuracy not yet measuredIn the citing paper
“which are widely employed in VUI re-search [71, 80]”
From Examining Humanness as a Metaphor to Design Voice User Interfaces (2024) · cites this paper as [71]
In the citing paper
“While this method is common practice in VUI research [71], it notably lacks the inclusion of direct interac-tion with the technology, offering insights that are observational rather than experiential.”
From Examining Humanness as a Metaphor to Design Voice User Interfaces (2024) · cites this paper as [71]
In the citing paper
“This questionnaire from robotics research has been successfully modifed to accommodate VUIs in previous studies (e.g., [24,57]).”
From “A Painless Way to Learn:” Designing an Interactive Storytelling Voice User Interface to Engage Older Adults in Informal Health Information Learning (2023) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414 · cited 1×
“Then, they conducted two inductive thematic analyses [5, 16] to create categorical frameworks for each of the IV and DV data.”
MethodLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Guest, G., MacQueen, K., & Namey, E. (2012) · doi:10.4135/9781483384436 · Cited by 7,924
Bartneck, C., Kulić, D., Croft, E., & Zoghbi, S. (2009) · International Journal of Social Robotics · doi:10.1007/s12369-008-0001-3 · Cited by 3,193 · cited 1×
“human-robot interaction was developed [2], use the IV and DV frameworks as a guide based on consensus.”
Method(2006) · doi:10.1080/01449290500330331 · Cited by 3,026 · cited 1×
“Indeed, lack of definitional and operational consensus has been a running problem [1,18,26,28,29].”
BackgroundNass, C., Steuer, J., & Tauber, E. R. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191703 · Cited by 2,148 · cited 2×
“tend to react to humanlike features of computer voice as they would react to people, without realizing it [31, 40, 42].”
Evidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232 · cited 1×
“Rapid reviews are used to generate a quick yet systematic descriptive and narrative summary of a focused area of study [12, 14, 23].”
MethodCochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews
Garritty, C., Gartlehner, G., Nussbaumer-Streit, B., King, V. J., Hamel, C., Kamel, C., Affengruber, L., & Stevens, A. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.10.007 · Cited by 1,141 · cited 1×
“We conducted a rapid review of the literature following the Cochrane gold standard protocol [15].”
Understanding, scoping and defining user experience: a survey approach
Law, E. L.-C., Roto, V., Hassenzahl, M., Vermeeren, A. P. O. S., & Kort, J. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518813 · Cited by 1,127
Artificial intelligence and communication: A Human–Machine Communication research agenda
Guzman, A. L., & Lewis, S. C. (2020) · New Media & Society · doi:10.1177/1461444819858691 · Cited by 951 · cited 1×
“Moreover, there has been a growing recognition that social and psychological factors, including afect and emotion, trust, credibility, rapport, and the relational context, need to be considered [17, 36].”
Background
Research works printed but not linked: 2
- Maureen Dobbins. 2017. Rapid Review Guidebook: Steps for Conducting a Rapid Review. National Collaborating Centre for Methods and Tools, Hamilton, ON. Retrieved from https://www.nccmt.ca/capacity-development/rapid-review-guidebook
- Yaniv Leviathan. 2018. Google Duplex: An AI System for Accomplishing Real-World Tasks Over the Phone. Google AI Blog. Retrieved January 8, 2021 from http://ai.googleblog.com/2018/05/duplex-ai-system-for-natural-conversation.html
Cited by
Pataranutaporn, P., Liu, R., Finn, E., & Maes, P. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00720-7 · Cited by 192
Nonverbal Cues in Human–Robot Interaction: A Communication Studies Perspective
Urakami, J., & Seaborn, K. (2022) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3570169 · Cited by 185
Self-citationSchreibelmayr, S., & Mara, M. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.787499 · Cited by 72
“They prompt the idea that a phone call from a bot, for example, could soon be hardly distinguishable from a real person (Oord et al., 2016; Seaborn and Urakami, 2021)—unless a different design decision is made by the creators of the voice.”
BackgroundExperimentation for Chatbot Usability Evaluation: A Secondary Study
Ren, R., Zapata, M., Castro, J. W., Dieste, Ó., & Acuña, S. T. (2022) · IEEE Access · doi:10.1109/access.2022.3145323 · Cited by 56
“…usability, including investigations on a specific types of chatbots [17], [18], [19], a qualitative study of critical factors that affect users’ perception of chatbots [20], a discussion on chatbot quality attributes [21], and a quantitative user experience research on voice-based chatbots [22].”
BackgroundUser Experience and Usability of Voice User Interfaces: A Systematic Literature Review
Deshmukh, A. M., & Chalmeta, R. (2024) · Information · doi:10.3390/info15090579 · Cited by 55
“…shows the Source Analysis through Bradford’s Law. understanding of the intellectual structure and impact dynamics within speci fi c research domains [35] This paper addresses the challenge of quantifying user experience (UX) in voice-based systems, identifying gaps, and proposing frameworks to…”
Influential
Metadata
- Authors
- Seaborn, K., & Urakami, J.
- Affiliations
- Tokyo Institute of Technology (JP)
- Venue
- Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-08
- Pages
- 1-8
- OpenAlex
- W3135939776
- Cited by
- 55
- References
- 56
- Citations by year
- 2026: 8 · 2025: 10 · 2024: 14 · 2023: 11 · 2022: 10 · 2021: 2
- FWCI
- 6.91
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- speech interfaces, voice assistants, quantitative measurement, embodied interaction, non-visual interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Measuring Voice UX Quantitatively: A Rapid Review. https://app.cochat.ai/papers/10.1145/3411763.3451712
- The paper
- Seaborn, K., & Urakami, J. (2021). Measuring Voice UX Quantitatively: A Rapid Review. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411763.3451712
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.