User Experience of Alexa when controlling music: comparison of face and construct validity of four questionnaires
Open accessThe DOI matches its Crossref record; 32 of the 33 references it cites were found in a registry or an index.
Abstract
We evaluate the user experience (UX) of Amazon's Alexa when users play and control music. For measuring UX we use established UX metrics (SASSI, SUISQ-R, SUS, AttrakDiff). We investigated face validity by asking users to rate how well they think a questionnaire measures what it is supposed to measure and we assessed construct validity by correlating UX scores of questionnaires with each other. We find a mismatch between face and construct validity of the evaluated questionnaires. Specifically, users feel that SASSI represents their experience better than other questionnaires, however this is not supported by correlations between questionnaires, which suggest that all investigated questionnaires measure UX to a similar extent. Importantly, the fact that face validity and construct validity diverge is not surprising as this has been observed before. Our work adds to existing literature by providing face and construct validity scores of UX questionnaires for interactions with the common speech assistant Alexa.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For VUIs, SUS has been successfully tested for face validity [28], and face and construct validity for a specifc use scenario [10].”
From “A Painless Way to Learn:” Designing an Interactive Storytelling Voice User Interface to Engage Older Adults in Informal Health Information Learning (2023) · cites this paper as [10]
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971 · cited 1×
“Power analyses were conducted using [8, 9].”
Statistical power analyses using G∗Power 3.1: tests for correlation and regression analyses
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.1149 · Cited by 37,615
Convergent and discriminant validation by the multitrait-multimethod matrix.
Campbell, D. T., & Fiske, D. W. (1959) · Psychological Bulletin · doi:10.1037/h0046016 · Cited by 17,116 · cited 1×
“One way of assessing differences in what metrics measure is pairwise comparison of correlations between questionnaires [6].”
BackgroundConstruct validity in psychological tests.
Cronbach, L. J., & Meehl, P. E. (1955) · Psychological Bulletin · doi:10.1037/h0040957 · Cited by 11,195
The use of ranks to avoid the assumption of normality implicit in the analysis of variance
Friedman, M. (1937) · Journal of the American Statistical Association · doi:10.1080/01621459.1937.10503522 · Cited by 7,005 · cited 1×
“For evaluating face validity, we used a non-parametric Friedman test [12] in order to determine whether there were significant differences between the rankings of the questionnaires.”
MethodTests for comparing elements of a correlation matrix
Steiger, J. H. (1980) · Psychological Bulletin · doi:10.1037/0033-2909.87.2.245 · Cited by 4,621 · cited 1×
“In this case [51]’s procedure [51] was used.”
MethodInternational Organization for Standardization (ISO)
Nahler, G. (2009) · Dictionary of Pharmaceutical Medicine · doi:10.1007/978-3-211-89836-9_723 · Cited by 3,677
(2006) · doi:10.1080/01449290500330331 · Cited by 3,026
psych: Procedures for Psychological, Psychometric, and Personality Research
Revelle, W. (2007) · CRAN: Contributed Packages · doi:10.32614/cran.package.psych · Cited by 2,965
Research works printed but not linked: 11
- Attwater D., 2019, Conversations with Computers. Talk at Fraunhofer IIS on 27th of
- Bevan Nigel, 2009, Proceedings of the Workshop UXEM, 9
- Hassenzahl Marc, Funology
- Lai-Chong Law Effie, 2009, Proceedings of the 27th international conference on Human factors in computing systems - CHI '09
- Lewis James R., 2016, Standardized Questionnaires for Voice Interaction Design. 1, 1
- James, 2017, Lewis and Jeff Sauro
- Future Market Research, 2023, Voice Assistant Market Research Report -- Global Forecast
- Pearl Cathy, Designing Voice User Interfaces: Principles of Conversational Experiences (1 ed.)
- Team Studio, RStudio: Integrated Development Environment for R. RStudio
- Sauro Jeff, 2016, Lewis
- Research Splendid, Technical Report. Splendid Research
Cited by
A Systematic Review of Voice Assistant Usability: An ISO 9241–11 Approach
Dutsinma, F. L. I., Pal, D., Funilkul, S., & Chan, J. (2022) · SN Computer Science · doi:10.1007/s42979-022-01172-3 · Cited by 104
Measuring Voice UX Quantitatively: A Rapid Review
Seaborn, K., & Urakami, J. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451712 · Cited by 55
“, lexical complexity, adaptation) [51]; Interaction time [27]; Driving performance [38]; Disclosure (user responses) [53]; Accidents [41] Subj Continued use [33]; Satisfaction [33]; System efectiveness [24]; Tone clarity [8]; Satisfaction [3]; System Usability Scale (SUS) [6, 44]; Speech User Interface Service”
BackgroundInfluentialA Review of Subjective Scales Measuring the User Experience of Voice Assistants
Faruk, L. I. D., Babakerkhell, M. D., Mongkolnam, P., Chongsuphajaisiddhi, V., Funilkul, S., & Pal, D. (2024) · IEEE Access · doi:10.1109/access.2024.3358423 · Cited by 32
“Studies have evaluated varied use cases - personal assistants [37], media centers [98], health counseling [91], automotive interfaces [122] etc.”
BackgroundMethodFaruk, L. I. D., Pal, D., Funilkul, S., Perumal, T., & Mongkolnam, P. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2359206 · Cited by 34
Desai, S., Lundy, M., & Chin, J. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3597141 · Cited by 25
Metadata
- Authors
- Brüggemeier, B., Breiter, M., Kurz, M., & Schiwy, J.
- Affiliations
- Fraunhofer Institute for Integrated Circuits (DE)
- Essen, Germany
- ORCID
- Breiter, Michael
- Venue
- Proceedings of the 2nd Conference on Conversational User Interfaces
- Publisher
- ACM
- Published
- 2020-07-22
- Pages
- 1-9
- OpenAlex
- W3042770254
- Cited by
- 12
- References
- 33
- Citations by year
- 2026: 1 · 2024: 2 · 2023: 3 · 2022: 2 · 2021: 3 · 2020: 1
- FWCI
- 1.64
- Type
- conference paper
- Language
- en
- Open access
- Locations
- ACM Conference on Conversational User Interfaces (CUI) (publishedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- Funders
- German Federal Ministry for Economic Affairs and Energy (01MK20011A)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- user experience, Amazon Alexa, voice assistants, SASSI, System Usability Scale, AttrakDiff, face validity, construct validity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: User Experience of Alexa when controlling music: comparison of face and construct validity of four questionnaires. https://app.cochat.ai/papers/10.1145/3405755.3406122
- The paper
- Brüggemeier, B., Breiter, M., Kurz, M., & Schiwy, J. (2020). User Experience of Alexa when controlling music: comparison of face and construct validity of four questionnaires. Proceedings of the 2nd Conference on Conversational User Interfaces. https://doi.org/10.1145/3405755.3406122
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.