Is It Time to Go Positive? Assessing the Positively Worded System Usability Scale (SUS)
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Objective: The goal of the research presented in this paper was to determine if the positively worded System Usability Scale (SUS) can be used in place of the positively and negatively worded standard SUS instrument for the subjective assessment of usability, and whether the results found here replicate those of Sauro and Lewis. Background: Sauro and Lewis’ previous study found no evidence that responses to SUS items differed across the standard SUS and the modified, positively worded version of the SUS when participants assessed websites. This study replicates and extends this work by examining a large number of different systems with larger sample sizes to add to the generalizability of previous findings. Methods: So that participants could retrospectively assess 20 products, the standard SUS was administered to 268 participants and the positive SUS to 698 participants. SUS scores were computed and the data analyzed using psychometric methods to explore how the two versions of the SUS differed. Results: The standard and positive versions of the SUS yielded similar SUS scores. In addition, both versions of the scale demonstrated evidence in support of reliability and validity. Conclusion: Either version of the SUS can be used with confidence to measure subjective usability. Furthermore, the scores generated from both versions of the SUS can be directly compared. Applications: In situations where cognitive load, participants’ spoken language, or item consistency with other surveys being given may be a factor, the positive SUS is a viable alternative to the standard SUS.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · cited 4×
“An additional benefit of the SUS is that its psychometric properties are reasonably well established and understood (Bangor, Kortum, & Miller, 2008; Kortum & Acemyan, 2019; Kortum & Bangor, 2013; Kortum & Oswald, 2018; Lewis & Sauro, 2009; McLellan, Muddimer, & Peres, 2012; Sauro, 2011).”
BackgroundMethodResultInfluentialSelf-citationThe Sources of Four Commonly Reported Cutoff Criteria: What Did They Really Say?
Lance, C. E., Butts, M. M., & Michels, L. C. (2006) · Organizational Research Methods · doi:10.1177/1094428105284919 · Cited by 2,053 · cited 1×
“…reliabilities (.80 or even higher) are generally preferred in settings where decisions are being made on the basis of individual scores, as opposed to looking at overall correlations between measures (Lance, Butts, & Michels, 2006), as is the case when we are comparing systems with one another.”
BackgroundThe System Usability Scale: Past, Present, and Future
Lewis, J. R. (2018) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2018.1455307 · Cited by 1,958 · cited 1×
“Although designed as a “quick and dirty” scale, the advantages of speed and ease of administration, and scoring have led to a widespread adoption of the SUS, making it the most widely used usability measurement today (Lewis, 2018).”
BackgroundPashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321 · cited 1×
“First, it is well known that there is a replication crisis ongoing in the field of psychology (Open Science Collaboration, 2015; Pashler & Wagenmakers, 2012), which has demonstrated the need to gather results that further bolster, weigh against, or otherwise provide important qualifications to the…”
BackgroundThe Factor Structure of the System Usability Scale
Lewis, J. R., & Sauro, J. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02806-9_12 · Cited by 1,278
Cronbach's alpha reliability: Interval estimation, hypothesis testing, and sample size planning
Bonett, D. G., & Wright, T. A. (2015) · Journal of Organizational Behavior · doi:10.1002/job.1960 · Cited by 1,237
Positive and negative global self-esteem: A substantively meaningful distinction or artifactors?
Marsh, H. W. (1996) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.70.4.810 · Cited by 811
The Usability Metric for User Experience
Finstad, K. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2010.04.004 · Cited by 604 · cited 1×
“There are a number of validated subjective usability measurements available (e.g., USE, Lund, 2001; UMUX, Finstad, 2010; ASQ, Lewis, 1991; PSSUQ, Lewis, 1992), but the SUS, or System Usability Scale (Brooke, 1996), is one of the most widely used and accepted measures today.”
BackgroundIneffectiveness of Reverse Wording of Questionnaire Items: Let’s Learn from Cows in the Rain
Sonderen, E. V., Sanderman, R., & Coyne, J. C. (2013) · PLoS ONE · doi:10.1371/journal.pone.0068967 · Cited by 544 · cited 2×
“However, general research on the issue since then has demonstrated that measures with positively and negatively worded questions result in a negligible impact on reducing acqui-escence bias (van Sonderen, Sanderman, & Coyne, 2013).”
Background
Cited by
Vlachogianni, P., & Tselios, N. (2022) · Journal of Research on Technology in Education · doi:10.1080/15391523.2020.1867938 · Cited by 386
Multi-language toolkit for the system usability scale
Gao, M., Kortum, P., & Oswald, F. L. (2020) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1801173 · Cited by 140
Self-citationXu, Z., Wang, G., Zhai, S., & Liu, P. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2176986 · Cited by 12
A One-Page Text Entry Method Optimized for Rectangle Smartwatches
Jang, R., Jung, C., Mohaisen, D., Lee, K., & Nyang, D. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2021.3057226 · Cited by 5
“respondents [33], and provides the reliability and validity as the standard SUS (mixture of positive and negative questions) [34].”
Background
Metadata
- Authors
- Kortum, P., Acemyan, C. Z., & Oswald, F. L.
- Venue
- Human Factors: The Journal of the Human Factors and Ergonomics Society
- Publisher
- SAGE Publications
- Published
- 2020-01-08
- Volume
- 63
- Issue
- 6
- Pages
- 987-998
- ISSN
- 0018-7208 (ISSN-L)
- 1547-8181
- OpenAlex
- W2999052780
- Cited by
- 51
- References
- 13
- FWCI
- 4.08
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Is It Time to Go Positive? Assessing the Positively Worded System Usability Scale (SUS). https://app.cochat.ai/papers/10.1177/0018720819881556
- The paper
- Kortum, P., Acemyan, C. Z., & Oswald, F. L. (2021). Is It Time to Go Positive? Assessing the Positively Worded System Usability Scale (SUS). Human Factors: The Journal of the Human Factors and Ergonomics Society. https://doi.org/10.1177/0018720819881556
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.