When designing usability questionnaires, does it hurt to be positive?
The DOI matches its Crossref record; 28 of the 30 references it cites were found in a registry or an index.
Abstract
When designing questionnaires there is a tradition of including items with both positive and negative wording to minimize acquiescence and extreme response biases. Two disadvantages of this approach are respondents accidentally agreeing with negative items (mistakes) and researchers forgetting to reverse the scales (miscoding).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This scale was chosen because it is a very easy scale to administer to participants; it can be used on small sample sizes with reliable results and is valid, which means it can easily differentiate between usable and unusable applications [53].”
From Are Saudi Arabian banks' mobile applications accessible for blind or partially sighted users?: a customers' perspective and evaluation (2025) · cites this paper as [53]
In the citing paper
“Moreover, there is evidence that the mixed tone caused respondents to make mistakes and researchers to miscode the questionnaire [48].”
From Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis (2020) · cites this paper as [48]
In the citing paper
“In 2011, Sauro and Lewis tested a positive version of the SUS and found no significant difference between the mean overall SUS scores of the positive and mixed versions [48].”
From Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis (2020) · cites this paper as [48]
In the citing paper
“In addition, they found no evidence for strong acquiescence bias or extreme response bias.”
From Dimensionality of the system usability scale among professionals using internet-based interventions for depression: a confirmatory factor analysis (2020) · cites this paper as [48]
In the citing paper
“indicating grade A and excellent system usability”
From Digital Self-Management in Support of Patients Living With Chronic Pain: Feasibility Pilot Study (2020) · cites this paper as [51]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Review of Psychometric theory (2nd ed.).
SITGREAVES, R. (1979) · Contemporary Psychology: A Journal of Reviews · doi:10.1037/018882 · Cited by 11,425 · cited 2×
“Designers of attitudinal questionnaires (of which questionnaires that measure satisfaction with usability are one type) are trained to consider questionnaire response styles such as extreme response bias and acquiescence bias [17].”
BackgroundSUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181
IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use
Lewis, J. R. (1995) · International Journal of Human-Computer Interaction · doi:10.1080/10447319509526110 · Cited by 2,429 · cited 1×
“The ASQ, PSSUQ and CSUQ [12], [13], [14] are exceptions, with all positive items.”
BackgroundSelf-citationDevelopment of an instrument measuring user satisfaction of the human-computer interface
Chin, J. P., Diehl, V. A., & Norman, L. K. (1988) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '88 · doi:10.1145/57167.57203 · Cited by 1,609 · cited 2×
“The strategy of alternating item wording has been applied in the construction of most of the popular usability questionnaires, including the System Usability Scale (SUS) [6], SUMI [11], and QUIS [7].”
MethodThurstone, L. L. (1928) · American Journal of Sociology · doi:10.1086/214483 · Cited by 1,464 · cited 1×
“These results were consistent with one of the earliest reports of attitudes in scale construction [27].”
ResultThe Factor Structure of the System Usability Scale
Lewis, J. R., & Sauro, J. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-02806-9_12 · Cited by 1,278 · cited 4×
“Factor Analysis of the Questionnaires Finally, we compared the factor structures of the two versions of the SUS with the SUS factor structure reported in [15], based on the large sample of SUS questionnaires collected by [3] (and replicated by [5]).”
BackgroundMethodInfluentialSelf-citationSierles, F. (1982) · Clinical Behavioral Science · doi:10.1007/978-94-011-7973-7_4 · Cited by 985
Presser, S., Rossi, P. H., Wright, J. D., & Anderson, A. B. (1985) · Social Forces · doi:10.2307/2578982 · Cited by 921
Weijters, B., Cabooter, E., & Schillewaert, N. (2010) · International Journal of Research in Marketing · doi:10.1016/j.ijresmar.2010.02.004 · Cited by 801
Research works printed but not linked: 6
- Karn K. Little A. Nelson G. Sauro J. Kirakowski J. Albert W. & Norman K. (2008) Subjective Ratings of Usability: Reliable or Ridiculous? Panel Presentation at the Usability Professionals Association (UPA 2008) Conference Baltimore MD. Karn K. Little A. Nelson G. Sauro J. Kirakowski J. Albert W. & Norman K. (2008) Subjective Ratings of Usability: Reliable or Ridiculous? Panel Presentation at the Usability Professionals Association (UPA 2008) Conference Baltimore MD.
- 10.5555/1965361 · doi:10.5555/1965361
- Nunnally , J. C. ( 1978 ). Psychometric theory ( 2 nd ed.). New York, NY : McGraw-Hill . Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York, NY: McGraw-Hill.
- Molich , R. , Kirakowski , J. , Sauro , J. , & Tullis , T. , ( 2009 ) Comparative Usability Task Measurement Workshop (CUE-8) at the UPA 2009 Conference in Portland , OR. Molich, R., Kirakowski, J., Sauro, J., & Tullis, T., (2009) Comparative Usability Task Measurement Workshop (CUE-8) at the UPA 2009 Conference in Portland, OR.
- 10.1177/014920639702300503 · doi:10.1177/014920639702300503 · Paper page
- SPSS. ( 2003 ). SPSS for Windows , Rel. 12 . 0 . 1 . (2003). Chicago, IL: SPSS Inc. SPSS. (2003). SPSS for Windows, Rel. 12.0.1. (2003). Chicago, IL: SPSS Inc.
Cited by
The System Usability Scale: Past, Present, and Future
Lewis, J. R. (2018) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2018.1455307 · Cited by 1,958
“Sauro and Lewis (2011) administered a retrospective survey using the standard and positive versions of the SUS ( n = 213) across seven websites.”
BackgroundInfluentialSelf-citation(2013) · Journal of Usability Studies archive · Cited by 1,357
Item benchmarks for the system usability scale
(2018) · Journal of Usability Studies archive · doi:10.5555/3294033.3294037 · Cited by 355
Self-citationThe Development and Validation of the Game User Experience Satisfaction Scale (GUESS)
Phan, M. H., Keebler, J. R., & Chaparro, B. S. (2016) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720816669646 · Cited by 342
“Past studies tended to employ reversed-scoring method in an attempt to minimize acquies-cence bias and extreme response bias (Sauro & Lewis, 2011).”
BackgroundMethodEuropean Portuguese Validation of the System Usability Scale (SUS)
Martins, A. I., Rosa, A. F., Queirós, A., Silva, A., & Rocha, N. P. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.09.273 · Cited by 326
“Sauro and Lewis [22] found that there were some problems both on the part of users in responding to questions and of researchers in scoring the questionnaires.”
Background
Metadata
- Authors
- Sauro, J., & Lewis, J. R.
- Affiliations
- Oracle Corporation, Denver, Colorado, USA
- IBM (United States) (US)
- ORCID
- Lewis, James R.
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2011-05-07
- Pages
- 2215-2224
- OpenAlex
- W2014817805
- Cited by
- 363
- References
- 30
- Citations by year
- 2026: 14 · 2025: 22 · 2024: 38 · 2023: 47 · 2022: 34 · 2021: 39 · 2020: 30 · 2019: 33 · 2018: 28 · 2017: 16 · 2016: 22 · 2015: 22 · 2014: 5 · 2013: 10 · 2012: 3
- FWCI
- 15.75
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- usability questionnaires, acquiescence bias, response errors, questionnaire design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: When designing usability questionnaires, does it hurt to be positive? https://app.cochat.ai/papers/10.1145/1978942.1979266
- The paper
- Sauro, J., & Lewis, J. R. (2011). When designing usability questionnaires, does it hurt to be positive? Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/1978942.1979266
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.