Psychometric Evaluation of the EMO and the SUS in the Context of a Large-Sample Unmoderated Usability Study
Abstract
This article describes the psychometric properties of the Emotional Metric Outcomes (EMO) questionnaire and the System Usability Scale (SUS) using data collected as part of a large-sample unmoderated usability study (n = 471). The EMO is a concise multifactor standardized questionnaire that provides an assessment of transaction-driven personal and relationship emotional outcomes, both positive and negative. The SUS is a well-known standardized usability questionnaire designed to assess perceived usability. In previous research, psychometric evaluation using data from a series of online surveys showed that the EMO and its component scales had high reliability and concurrent validity with loyalty and overall experience metrics but did not find the expected four-factor structure. Previous structural analyses of the SUS have had mixed results. Analysis of the EMO data from the usability study revealed the expected four-factor structure. The factor structure of the SUS appeared to be driven by item tone. The estimated reliability of the SUS (.90) was consistent with previous estimates. The EMO and its subscales were also quite reliable, with the estimates of reliability for the various EMO scales ranging from .86 to .96. Regression analysis using SUS, EMO, and Effort as predictors revealed different key drivers for the outcome metrics of Satisfaction and Likelihood-to-Recommend. The key recommendations are to include the EMO as part of the battery of poststudy standardized questionnaires, along with the SUS (or similar questionnaire), but to be cautious in reporting SUS subscales such as Usable and Learnable.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The SUS is reported to be a highly robust and versatile tool with excellent psychometric properties, with high internal reliability and concurrent validity with other measures of perceived and objective usability.
“The SUSis reported to be a highly robust and versatile tool with excellent psychometric properties. The literature shows that the internal reliability is high, with Cronbach's alphas at or > 0.9 (Bangor et al., 2008; Lewis et al., 2015) and concurrent validity with other measures of perceived and objective usability (Lewis, 2018).”
From Acceptability and Co-Development of an Online Cognitive Bias Modification Intervention for Emerging Adults With Hazardous Alcohol Use and Social Anxiety: A Mixed Methods Study (2020) · cites this paper as Lewis et al., 2015
Studies have also replicated the one-factor model of the SUS.
“and the one-factor model as well [24, 25].”
From Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis (2020) · cites this paper as [25]
A large variety of product experience was also reported in studies that found a one-factor model.
“However, a large variety of product experience was also reported in studies that found a one-factor model (e.g. ) [24, 25].”
From Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis (2020) · cites this paper as [25]
Cites
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989 · see citing works
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · see citing works
Usability Evaluation In Industry
(1996) · CRC Press eBooks · doi:10.1201/9781498710411 · Cited by 4,400 · see citing works
The Factor Structure of the System Usability Scale
Lewis, J. R., & Sauro, J. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02806-9_12 · Cited by 1,278 · see citing works
Self-citation(2012) · Cited by 1,032 · see citing works
Self-citationWhen designing usability questionnaires, does it hurt to be positive?
Sauro, J., & Lewis, J. R. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979266 · Cited by 363 · see citing works
Self-citationUsability: Lessons Learned … and Yet to Be Learned
Lewis, J. R. (2014) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2014.930311 · Cited by 317 · see citing works
Self-citationUser satisfaction from commercial web sites: The effect of design and use
Zviran, M., Glezer, C., & Avni, I. (2005) · Information & Management · doi:10.1016/j.im.2005.04.002 · Cited by 309 · see citing works
On the dimensionality of the System Usability Scale: a test of alternative measurement models
Borsci, S., Federici, S., & Lauriola, M. (2009) · Cognitive Processing · doi:10.1007/s10339-009-0268-9 · Cited by 275 · see citing works
UMUX-LITE: when there's no time for the SUS
Lewis, J. R., Utesch, B. S., & Maher, D. E. (2013) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2481287 · Cited by 270 · see citing works
Self-citationCorrelations among prototypical usability metrics
Sauro, J., & Lewis, J. R. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518947 · Cited by 221 · see citing works
Self-citationBorsci, S., Federici, S., Bacci, S., Gnaldi, M., & Bartolucci, F. (2015) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2015.1064648 · Cited by 180 · see citing works
What happened to remote usability testing?
Andreasen, M. S., Nielsen, H. V., Schrøder, S. O., & Stage, J. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240838 · Cited by 177 · see citing works
An Empirical Comparison of Lab and Remote Usability Testing of Web Sites
Tullis, T., Fleischman, S., McNulty, M., Cianchette, C., & Bergel, M. (2002) · Cited by 120 · see citing works
Lewis, J. R. (2012) · doi:10.1002/9781118131350.ch46 · Cited by 83 · see citing works
Self-citationKortum, P. T., & Peres, S. C. (2014) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2014.904177 · Cited by 81 · see citing works
Beyond the Usability Lab: Conducting Large-scale Online User Experience Studies
Albert, W., Tullis, T. S., & Tedesco, D. P. (2009) · CERN Document Server (European Organization for Nuclear Research) · Cited by 67 · see citing works
Remote Usability Evaluation: Overview and Case Studies
McFadden, E. T., Hager, D. R., Elie, C. J., & Blackwell, J. M. (2002) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2002.9669131 · Cited by 36 · see citing works
Automated summative usability studies
West, R. W., & Lehman, K. R. (2006) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1124772.1124867 · Cited by 34 · see citing works
Development and Psychometric Evaluation of the Emotional Metric Outcomes (EMO) Questionnaire
Lewis, J. R., & Mayes, D. K. (2014) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2014.930312 · Cited by 24 · see citing works
Self-citation
Flagged first, then by citations
1 of the 21 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
The System Usability Scale: Past, Present, and Future
Lewis, J. R. (2018) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2018.1455307 · Cited by 1,958 · see citing works
Self-citationItem benchmarks for the system usability scale
(2018) · Journal of Usability Studies archive · doi:10.5555/3294033.3294037 · Cited by 355 · see citing works
Self-citationMeasuring Perceived Usability: The CSUQ, SUS, and UMUX
Lewis, J. R. (2018) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2017.1418805 · Cited by 302 · see citing works
Self-citationUSABILITY AND USER EXPERIENCE: DESIGN AND EVALUATION
Lewis, J. R., & Sauro, J. (2021) · Wiley eBooks · doi:10.1002/9781119636113.ch38 · Cited by 156 · see citing works
Self-citationRevisiting the factor structure of the system usability scale
Lewis, J. R., & Sauro, J. (2017) · Journal of Usability Studies archive · doi:10.5555/3190867.3190870 · Cited by 113 · see citing works
Self-citationMol, M., van Schaik, A., Dozeman, E., Ruwaard, J., Vis, C., Ebert, D. D., Etzelmueller, A., Mathiasen, K., Moles, B., Mora, T., et al. (2020) · BMC Psychiatry · doi:10.1186/s12888-020-02627-8 · Cited by 92 · see citing works
Standardized Usability Questionnaires: Features and Quality Focus
Assila, A., de Oliveira, K. M., & Ezzedine, H. (2016) · HAL · Cited by 97 · see citing works
Standardized usability questionnaires
Sauro, J., & Lewis, J. R. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-802308-2.00008-4 · Cited by 95 · see citing works
Self-citationAssessing the usability of complex psychosocial interventions: The Intervention Usability Scale
Lyon, A. R., Pullmann, M. D., Jacobson, J., Osterhage, K. P., Achkar, M. A., Renn, B. N., Munson, S. A., & Areán, P. A. (2021) · Implementation Research and Practice · doi:10.1177/2633489520987828 · Cited by 87 · see citing works
Measuring Perceived Usability: SUS, UMUX, and CSUQ Ratings for Four Everyday Products
Lewis, J. R. (2018) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2018.1533152 · Cited by 66 · see citing works
Self-citationAlhadreti, O. (2020) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2020.1861766 · Cited by 47 · see citing works
Rahmadiana, M., Karyotaki, E., Schulte, M., Ebert, D. D., Passchier, J., Cuijpers, P., Berger, T., van Ballegooijen, W., Wimbarti, S., & Riper, H. (2021) · JMIR Mental Health · doi:10.2196/20036 · Cited by 44 · see citing works
Castilla, D., Jaén, I., Suso‐Ribera, C., García‐Soriano, G., Zaragozá, I., Bretón-López, J. M., Mira, A., Díaz‐García, A., & García‐Palacios, A. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2209840 · Cited by 35 · see citing works
Performance differences between instructions on paper vs digital glasses for a simple assembly task
Rodriguez, F. S., Saleem, K., Spilski, J., & Lachmann, T. (2021) · Applied Ergonomics · doi:10.1016/j.apergo.2021.103423 · Cited by 31 · see citing works
Prior, K., Salemink, E., Wiers, R. W., Teachman, B. A., Piggott, M., Newton, N. C., Teesson, M., Baillie, A. J., Campbell, S., & Stapinski, L. A. (2020) · Alcoholism: Clinical and Experimental Research · doi:10.1111/acer.14452 · Cited by 18 · see citing works
Benaida, M., Namoun, A., & Taleb, A. (2018) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2018.090848 · Cited by 17 · see citing works
Usability Perceptions of the Madrasati Platform by Teachers in Saudi Arabian Schools
Shishah, W. (2021) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2021.0120839 · Cited by 13 · see citing works
Eisenstadt, M., Liverpool, S., Metaxa, A.-M., Ciuvat, R. M., & Carlsson, C. (2021) · JMIR Formative Research · doi:10.2196/31064 · Cited by 12 · see citing works
van Doorn, M., Monsanto, A., Boeschoten, C. M., van Amelsvoort, T., Popma, A., Öry, F. G., Álvarez‐Jiménez, M., Gleeson, J., Jaspers, M., & Nieman, D. H. (2023) · Frontiers in Digital Health · doi:10.3389/fdgth.2022.1020753 · Cited by 9 · see citing works
Chandler, J., Christon, L., Benfield, K., Pairet, S., Hoffman, M., Treiber, F. A., Mueller, M., & James, W. E. (2023) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2023.101062 · Cited by 7 · see citing works
User-Centred Design in Content Management System Development: The Case of EMasters
Nakić, J., Burčul, A., & Marangunić, N. (2019) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v13i08.10727 · Cited by 7 · see citing works
Bai, E., Song, S. L., Fraser, H., & Ranney, M. L. (2022) · Applied Clinical Informatics · doi:10.1055/s-0041-1740923 · Cited by 7 · see citing works
Khayyatkhoshnevis, P., Tillberg, S., Latimer, E. A., Aubry, T., Fisher, A., & Mago, V. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-05311-5_16 · Cited by 6 · see citing works
Matula, K. A., Minar, P., Daraiseh, N. M., Lin, L., Recker, M., & Lipstein, E. A. (2022) · Health Expectations · doi:10.1111/hex.13618 · Cited by 6 · see citing works
Modelling assistants based on information reuse: a user evaluation for language engineering
Mora-Segura, Á., Lara, J. D., & Wimmer, M. (2023) · Software & Systems Modeling · doi:10.1007/s10270-023-01094-5 · Cited by 6 · see citing works
2Vita-B Physical: An Intelligent Home Rehabilitation System Based on Microsoft Azure Kinect
Antico, M., Balletti, N., Ciccotelli, A., Ciccotelli, M., Laudato, G., Lazich, A., Notarantonio, M., Oliveto, R., Ricciardi, S., Scalabrino, S., et al. (2021) · Frontiers in Human Dynamics · doi:10.3389/fhumd.2021.678529 · Cited by 5 · see citing works
Measuring the Perceived Clutter of Websites
Lewis, J. R., & Sauro, J. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2359205 · Cited by 5 · see citing works
Self-citationVideo conferencing as a face-to-face online meeting app: user preference based on usability testing
Wibowo, A. W. A., Rahmawati, B. D., & Mastrisiswadi, H. (2021) · Jurnal Sistem dan Manajemen Industri · doi:10.30656/jsmi.v5i2.3432 · Cited by 5 · see citing works
Sahar, N. H., Badlishah-Sham, S. F., & Ramli, A. S. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18179405 · Cited by 4 · see citing works
Amarti, K., Schulte, M. H. J., Kleiboer, A. M., van Genugten, C., Oudega, M. L., Sonnenberg, C., Gonçalves, G., Rocha, A., & Riper, H. M. (2022) · JMIR Research Protocols · doi:10.2196/41445 · Cited by 3 · see citing works
Ledermann, K., Khaled, O. A., Caon, M., Berger, T., Chabwine, J. N., Wicht, J., & Martin-Söelch, C. (2021) · JMIR Research Protocols · doi:10.2196/26930 · Cited by 2 · see citing works
Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability ScalePreprint
Lyon, A. R., Pullmann, M. D., Jacobson, J., Osterhage, K. P., Achkar, M. A., Renn, B. N., Munson, S. A., & Areán, P. A. (2020) · doi:10.21203/rs.3.rs-22305/v2 · Cited by 2 · see citing works
Amarti, K., Čihařová, M., Provoost, S., Schulte, M. H. J., Kleiboer, A. M., el Hassouni, A., Gonçalves, G., & Riper, H. M. (2025) · Internet Interventions · doi:10.1016/j.invent.2025.100900 · Cited by 1 · see citing works
Exploring Usability of a Clinical Decision Support System for Cancer Care: A User‐Centered Study
Chyzhyk, D., Arregi, M., Errazquin, M., Ariceta, A., Sevilla, M., Álvarez, R., & Alberich-Inchausti, M. (2025) · Cancer Reports · doi:10.1002/cnr2.70173 · Cited by 1 · see citing works
Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability ScalePreprint
Lyon, A. R., Pullmann, M. D., Jacobson, J., Osterhage, K. P., Achkar, M. A., Renn, B. N., Munson, S. A., & Areán, P. A. (2020) · doi:10.21203/rs.3.rs-22305/v1
Campos, L. F. D. A., Soares, I. G., Dias, C. D. P., Fernandes, F. R., Bontempo, K. P., & Maia, A. L. D. M. (2022) · DAT Journal · doi:10.29147/datjournal.v7i3.656
罗, 林. (2024) · Advances in Psychology · doi:10.12677/ap.2024.148524
Optimized User Experience for Labeling Systems for Predictive Maintenance Applications
Hallmann, M., Stern, M., Henning, J. C., Franke, U., Ostertag, T., da Costa, J. P. C. L., & Voigt‐Antons, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-92689-1_13
Wedig, I. J., Petushek, E. J., Durocher, J. J., McDaniel, J., & Elmer, S. J. (2025) · Frontiers in Physiology · doi:10.3389/fphys.2025.1631562
Amarti, K., Čihařová, M., Provoost, S., Schulte, M. H. J., Kleiboer, A. M., el Hassouni, A., Gonçalves, G., & Riper, H. M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5357442
Mol, M., van Schaik, A., Dozeman, E., Ruwaard, J., Vis, C., Ebert, D. D., Etzelmueller, A., Mathiasen, K., Moles, B., Mora, T., et al. (2020) · Figshare · doi:10.6084/m9.figshare.12291794
Hansa, J., Lefarth, R., & Hansen, H. (2026) · International Journal of Language & Communication Disorders · doi:10.1111/1460-6984.70289
Constraint-Guided SUS Inference from Natural Language Feedback
Cianconi, L., Terceiro, L., & Silva, J. O. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-30552-7_33
- Authors
- Lewis, J. R., Brown, J., & Mayes, D. K.
- Affiliations
- IBM (United States) (US)
- State Farm (United States) (US)
- Corresponding author
- Lewis, James R.
- Venue
- International Journal of Human-Computer Interaction
- Publisher
- Informa UK Limited
- Published
- 2015-06-25
- Volume
- 31
- Issue
- 8
- Pages
- 545-553
- ISSN
- 1044-7318 (ISSN-L)
- 1532-7590
- OpenAlex
- W2120270590
- Cited by
- 44
- References
- 21
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 2 · 2023: 5 · 2022: 5 · 2021: 9 · 2020: 7 · 2019: 1 · 2018: 5 · 2017: 1 · 2016: 2
- FWCI
- 5.07
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- International Journal of Human-Computer Interaction (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- System Usability Scale, psychometric properties, factor structure, reliability statistics, concurrent validity, perceived usability, user satisfaction, likelihood to recommend, user loyalty, effort
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
