Assessing User Satisfaction in the Era of User Experience: Comparison of the SUS, UMUX, and UMUX-LITE as a Function of Product Experience
Abstract
Nowadays, practitioners extensively apply quick and reliable scales of user satisfaction as part of their user experience analyses to obtain well-founded measures of user satisfaction within time and budget constraints. However, in the human–computer interaction literature the relationship between the outcomes of standardized satisfaction scales and the amount of product usage has been only marginally explored. The few studies that have investigated this relationship have typically shown that users who have interacted more with a product have higher satisfaction. The purpose of this article was to systematically analyze the variation in outcomes of three standardized user satisfaction scales (SUS, UMUX, UMUX-LITE) when completed by users who had spent different amounts of time with a website. In two studies, the amount of interaction was manipulated to assess its effect on user satisfaction. Measurements of the three scales were strongly correlated and their outcomes were significantly affected by the amount of interaction time. Notably, the SUS acted as a unidimensional scale when administered to people who had less product experience but was bidimensional when administered to users with more experience. Previous findings of similar magnitudes for the SUS and UMUX-LITE (after adjustment) were replicated but did not show the previously reported similarities of magnitude for the SUS and the UMUX. Results strongly encourage further research to analyze the relationships of the three scales with levels of product exposure. Recommendations for practitioners and researchers in the use of the questionnaires are also provided.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More recent research showed that the two-factor structure of the SUS possibly depends on the amount of experience that users have with a given product.
“More recent research showed that two-factor structure possibly depends on the amount of experience that users have with a given product [18].”
From Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis (2020) · cites this paper as [18]
The SUS has acceptable levels of convergent validity with other measures of perceived usability.
“acceptable levels of convergent validity with other measures of perceived usability (e.g. ) [18]”
From Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis (2020) · cites this paper as [18]
The correlations between SUS and CSQ-scores were comparable to other studies that found overlap with UMUX.
“The correlations between the SUS and CSQ-scores indicated convergent validity, comparable to other studies that also found considerable evidence for the overlap with other related questionnaires such as the Usability Metric for User Experience (UMUX) [18, 47].”
From Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis (2020) · cites this paper as [18]
Cites
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033 · see citing works
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989 · see citing works
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · see citing works
Carayannis, E. G. (2013) · Springer eBooks · doi:10.1007/978-1-4614-3858-8_100431 · Cited by 4,818 · see citing works
Usability Evaluation In Industry
(1996) · CRC Press eBooks · doi:10.1201/9781498710411 · Cited by 4,400 · see citing works
(2006) · doi:10.1080/01449290500330331 · Cited by 3,026 · see citing works
Mayer, T. F., Lazarsfeld, P. F., & Henry, N. W. (1969) · American Sociological Review · doi:10.2307/2092222 · Cited by 2,412 · see citing works
Handbook of Modern Item Response Theory
van der Linden, W. J., & Hambleton, R. K. (1997) · Springer eBooks · doi:10.1007/978-1-4757-2691-6 · Cited by 2,323 · see citing works
Exploratory latent structure analysis using both identifiable and unidentifiable models
Goodman, L. A. (1974) · Biometrika · doi:10.1093/biomet/61.2.215 · Cited by 1,616 · see citing works
The Factor Structure of the System Usability Scale
Lewis, J. R., & Sauro, J. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02806-9_12 · Cited by 1,278 · see citing works
The Thing and I: Understanding the Relationship Between User and Product
Hassenzahl, M. (2003) · Human-Computer Interaction Series · doi:10.1007/1-4020-2967-5_4 · Cited by 1,097 · see citing works
(2012) · Cited by 1,032 · see citing works
Henry, N. W. (2014) · Wiley StatsRef: Statistics Reference Online · doi:10.1002/9781118445112.stat02466 · Cited by 746 · see citing works
The Usability Metric for User Experience
Finstad, K. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2010.04.004 · Cited by 604 · see citing works
Human-Computer Interaction (3rd Edition)
Dix, A. J., Finlay, J., Abowd, G. D., & Beale, R. (2003) · Prentice-Hall, Inc eBooks · Cited by 497 · see citing works
Aesthetics and apparent usability
Tractinsky, N. (1997) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/258549.258626 · Cited by 484 · see citing works
Maxwell, S. E., & Cliff, N. (1988) · Journal of the American Statistical Association · doi:10.2307/2290163 · Cited by 407 · see citing works
When designing usability questionnaires, does it hurt to be positive?
Sauro, J., & Lewis, J. R. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979266 · Cited by 363 · see citing works
What is this evasive beast we call user satisfaction?
Lindgaard, G., & Dudek, C. (2003) · Interacting with Computers · doi:10.1016/s0953-5438(02)00063-2 · Cited by 326 · see citing works
Usability ratings for everyday products measured with the System Usability Scale
Kortum, P. T., & Bangor, A. (2013) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2012.681221 · Cited by 323 · cited 1× · see citing works
“In recent years, the growing availability of SUS data from a large number of studies (Bangor et al., 2008; Kortum & Bangor, 2012) has led to the production of norms for the interpretation of mean SUS scores, for example, the Curved Grading Scale (CGS; Sauro & Lewis, 2012).”
BackgroundUsability: Lessons Learned … and Yet to Be Learned
Lewis, J. R. (2014) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2014.930311 · Cited by 317 · see citing works
User satisfaction from commercial web sites: The effect of design and use
Zviran, M., Glezer, C., & Avni, I. (2005) · Information & Management · doi:10.1016/j.im.2005.04.002 · Cited by 309 · see citing works
Handbook of Polytomous Item Response Theory Models
(2011) · Routledge eBooks · doi:10.4324/9780203861264 · Cited by 288 · see citing works
On the dimensionality of the System Usability Scale: a test of alternative measurement models
Borsci, S., Federici, S., & Lauriola, M. (2009) · Cognitive Processing · doi:10.1007/s10339-009-0268-9 · Cited by 275 · see citing works
Self-citationUMUX-LITE: when there's no time for the SUS
Lewis, J. R., Utesch, B. S., & Maher, D. E. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2481287 · Cited by 270 · see citing works
Dix, A. J., & Shabir, N. (2011) · Human factors and ergonomics · doi:10.1201/b10855-5 · Cited by 269 · see citing works
Lallemand, C., Gronier, G., & Koenig, V. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2014.10.048 · Cited by 248 · see citing works
The effect of experience on system usability scale ratings
McLellan, S. G., Muddimer, A., & Peres, S. C. (2012) · doi:10.5555/2835476.2835478 · Cited by 190 · see citing works
The Evaluation of Accessibility, Usability, and User Experience
Petrie, H., & Bevan, N. (2009) · Human factors and ergonomics · doi:10.1201/9781420064995-c20 · Cited by 185 · see citing works
The Evaluation of Accessibility, Usability, and User Experience
Stephanidis, C. (2009) · CRC Press eBooks · doi:10.1201/9781420064995-31 · Cited by 171 · see citing works
Hassenzahl, M., Wiklund-Engblom, A., Bengs, A., Hägglund, S., & Diefenbach, S. (2015) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2015.1064664 · Cited by 167 · see citing works
A Class of Multidimensional IRT Models for Testing Unidimensionality and Clustering Items
Bartolucci, F. (2007) · Psychometrika · doi:10.1007/s11336-005-1376-9 · Cited by 96 · see citing works
Self-citationLewis, J. R. (2006) · doi:10.1002/0470048204.ch49 · Cited by 92 · see citing works
A Class of Multidimensional Latent Class IRT Models for Ordinal Polytomous Item Responses
Bacci, S., Bartolucci, F., & Gnaldi, M. (2014) · Communication in Statistics- Theory and Methods · doi:10.1080/03610926.2013.827718 · Cited by 52 · see citing works
Self-citationMultiLCIRT: An R package for multidimensional latent class item response models
Bartolucci, F., Bacci, S., & Gnaldi, M. (2013) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2013.05.018 · Cited by 51 · see citing works
Self-citationUsers' perceptions of usability and aesthetics as criteria of pre- and post-use preferences
Lee, S., & Koubek, R. J. (2012) · European J of Industrial Engineering · doi:10.1504/ejie.2012.044812 · Cited by 44 · see citing works
Critical Review of 'The Usability Metric for User Experience'
Lewis, J. R. (2013) · Interacting with Computers · doi:10.1093/iwc/iwt013 · Cited by 37 · see citing works
Response to commentaries on 'The Usability Metric for User Experience'
Finstad, K. (2013) · Interacting with Computers · doi:10.1093/iwc/iwt005 · Cited by 33 · see citing works
The Relationship Between Levels of User Experience with a Product and Perceived System Usability
Kortum, P. T., & Johnson, M. (2013) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931213571044 · Cited by 31 · see citing works
Beyond the User Preferences: Aligning the Prototype Design to the Users’ Expectations
Borsci, S., Kuljis, J., Barnett, J. C., & Pecchia, L. (2014) · Human Factors and Ergonomics in Manufacturing & Service Industries · doi:10.1002/hfm.20611 · Cited by 26 · see citing works
Self-citationComputer Systems Experiences of Users with and Without Disabilities
Borsci, S., Kurosu, M., Federici, S., et al. (2013) · CRC Press eBooks · doi:10.1201/b15619 · Cited by 25 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
The System Usability Scale: Past, Present, and Future
Lewis, J. R. (2018) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2018.1455307 · Cited by 1,958 · see citing works
Borsci, S., Malizia, A., Schmettow, M., van der Velde, F., Tariverdiyeva, G., Balaji, D., & Chamberlain, A. (2022) · Personal and Ubiquitous Computing · doi:10.1007/s00779-021-01582-9 · Cited by 266 · see citing works
Self-citationMulti-language toolkit for the system usability scale
Gao, M., Kortum, P., & Oswald, F. L. (2020) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1801173 · Cited by 140 · see citing works
Mol, M., van Schaik, A., Dozeman, E., Ruwaard, J., Vis, C., Ebert, D. D., Etzelmueller, A., Mathiasen, K., Moles, B., Mora, T., et al. (2020) · BMC Psychiatry · doi:10.1186/s12888-020-02627-8 · Cited by 92 · see citing works
Olde Keizer, R. A. C. M., van Velsen, L., Moncharmont, M., Riche, B., Ammour, N., Del Signore, S., Zia, G., Hermens, H., & N’Dja, A. (2019) · Health and Technology · doi:10.1007/s12553-019-00320-9 · Cited by 57 · see citing works
Lewis, J. R., Brown, J., & Mayes, D. K. (2015) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2015.1064665 · Cited by 44 · see citing works
Fico, G., Hernanzez, L., Cancela, J., Dagliati, A., Sacchi, L., Martinez-Millana, A., Posada, J., Manero, L., Verdú, J., Facchinetti, A., et al. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0887-8 · Cited by 22 · see citing works
- Authors
- Borsci, S., Federici, S., Bacci, S., Gnaldi, M., & Bartolucci, F.
- Affiliations
- National Institute for Health and Care Research (GB)
- University of Perugia (IT)
- Corresponding author
- Borsci, Simone
- Venue
- International Journal of Human-Computer Interaction
- Publisher
- Informa UK Limited
- Published
- 2015-06-24
- Volume
- 31
- Issue
- 8
- Pages
- 484-495
- ISSN
- 1044-7318 (ISSN-L)
- 1532-7590
- OpenAlex
- W1801891980
- Cited by
- 180
- References
- 41
- Citations by year
- 2026: 11 · 2025: 18 · 2024: 24 · 2023: 23 · 2022: 22 · 2021: 26 · 2020: 17 · 2019: 9 · 2018: 11 · 2017: 9 · 2016: 5 · 2015: 5
- FWCI
- 19.01
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Human-Computer Interaction (publishedVersion)
- Florence Research (University of Florence) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion)
- University of Twente Research Information (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Social Sciences › Decision Sciences › Information Systems and Management › Technology Adoption and User Behaviour
- Keywords
- user satisfaction, user experience, System Usability Scale, product experience, interaction time, scale reliability, website usability evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
