Getting the right design and the design right
The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
We present a study comparing usability testing of a single interface versus three functionally equivalent but stylistically distinct designs. We found that when presented with a single design, users give significantly higher ratings and were more reluctant to criticize than when presented with the same design in a group of three. Our results imply that by presenting users with alternative design solutions, subjective ratings are less prone to inflation and give rise to more and stronger criticisms when appropriate. Contrary to our expectations, our results also suggest that usability testing by itself, even when multiple designs are presented, is not an effective vehicle for soliciting constructive suggestions about how to improve the design from end users. It is a means to identify problems, not provide solutions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“and non-linear [13,87]: requires constant reconsideration of potential variations of existing text throughout the passage.”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Effective revision goes beyond minor editorial changes, and may help writers rework ideas, and powerfully affect their knowledge [23,87] as they explore alternative variations to find a line of argument [23].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Professional writers often write and rewrite text hundreds of times [12,87] and recommend rewriting as a core strategy for writing well [12,101].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Effective revision goes beyond minor editorial changes, and may help writers rework ideas, and powerfully affect their knowledge [23,87] as they explore alternative variations to find a line of argument [23].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Recognizing that revision is inherently nonlinear-with writers often revisiting earlier sections of a passageand recursive, manifesting in repeated cycles throughout the writing process [23,87], we aimed to align our LLM integrations with this fluid, iterative nature of revision.”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
(1993) · Cited by 9,879
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,504
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
Paper Prototyping: The Fast and Easy Way to Design and Refine User Interfaces
Snyder, C. (2003) · Cited by 953
Rettig, M. (1994) · Communications of the ACM · doi:10.1145/175276.175288 · Cited by 440
Low vs. high-fidelity prototyping debate
Rudd, J., Stern, K., & Isensee, S. (1996) · Interactions · doi:10.1145/223500.223514 · Cited by 422
High-Fidelity or Low-Fidelity, Paper or Computer? Choosing Attributes when Testing Web Prototypes
Walker, M., Takayama, L., & Landay, J. A. (2002) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120204600513 · Cited by 235
Sefelin, R., Tscheligi, M., & Giller, V. (2003) · CHI '03 extended abstracts on Human factors in computing systems - CHI '03 · doi:10.1145/765891.765986 · Cited by 209
Comparing usability problems and redesign proposals as input to practical systems development
Hornbæk, K., & Frøkjær, E. (2005) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055027 · Cited by 148 · cited 3×
“Consequently, if one agrees with the findings of [5], then our study emphasizes the need to adopt approaches beyond standard usability testing techniques in order to generate redesign proposals.”
BackgroundInfluentialNotes on design practice: stories and prototypes as catalysts for communication
Erickson, T. D. (1995) · John Wiley & Sons, Inc. eBooks · Cited by 136
Mis-usability: on the uses and misuses of usability testing
Dicks, R. (2002) · ACM International Conference on Systems Documentation (SIGDOC) · doi:10.1145/584955.584960 · Cited by 82 · cited 1×
“” [3]”
Talking through design: requirements and resistance in cooperative prototyping
Bowers, J., & Pycock, J. (1994) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191773 · Cited by 69 · cited 1×
“As stated by Bowers and Pycock [1] “explicit issuing of requests for redesign of DNP [Designer’s Note Pad, a tool that was prototyped and evaluated in their study] is very rare in our materials.”
Usability Evaluation and Prototype Fidelity: Users and Usability Professionals
Catani, M. B., & Biers, D. W. (1998) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129804201901 · Cited by 58 · cited 1×
“This is largely due to their relatively low cost, coupled with the results of a number of researchers [2, 13, 15, 16] who have found that the usability data that they got from low and high fidelity prototypes were comparable.”
ResultDoes the Fidelity of Software Prototypes Affect the Perception of Usability?
Wiklund, M., Thurrott, C., & Dumas, J. S. (1992) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193129203600429 · Cited by 47
The Simplicity Shift: Innovative Design Tactics in a Corporate World
Jenson, S. (2002) · Cited by 44
Comparative design review: an exercise in parallel design
Nielsen, J., & Desurvire, H. (1993) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/169059.169327 · Cited by 20 · cited 1×
“However, there is some literature on “parallel design” where different teams independently work on the same problem [ 7 , 8, 9, 10], but this only touches on what we are interested in with the latter.”
BackgroundDiversified parallel design: contrasting design approaches
Nielsen, J., Fernandes, T., Wagner, A., Wolf, R., & Ehrlich, K. (1994) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/259963.260224 · Cited by 10 · cited 1×
“However, there is some literature on “parallel design” where different teams independently work on the same problem [7, 8, 9 , 10], but this only touches on what we are interested in with the latter.”
BackgroundParallel design in the classroom
Ovaska, S., & Räihä, K. (1995) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/223355.223666 · Cited by 6 · cited 1×
“However, there is some literature on “parallel design” where different teams independently work on the same problem [7, 8, 9, 10 ], but this only touches on what we are interested in with the latter.”
Background
Cited by
Usability evaluation considered harmful (some of the time)
Greenberg, S., & Buxton, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357074 · Cited by 598
Peer and self assessment in massive online classes
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2505057 · Cited by 390
“Such work is especially important in design education because successful design often requires generating and reflecting on multiple ideas [Tohidi et al. 2006; Buxton 2007], and on exploration and iteration [Fallman 2003].”
BackgroundParallel prototyping leads to better design results, more divergence, and increased self-efficacy
Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1879831.1879836 · Cited by 367
“…combat this, creating multiple alternatives in parallel may encourage people to more effectively discover unseen constraints and opportunities [Cross 2006], enumerate more diverse solutions [Buxton 2007], and obtain more authentic and diverse feedback from potential users [Tohidi et al. 2006].”
BackgroundBringing Transparency Design into Practice
Eiband, M., Schneider, H., Bilandzic, M., Fazekas-Con, J., Haug, M., & Hussmann, H. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172961 · Cited by 250
“For the same reason, if multiple prototypes have been developed (as in our project), it is sensible to recruit different user groups for each prototype to investigate the effect of the different design solutions on users’ mental models (in contrast to the general recommendation to show users more than one prototype to…”
BackgroundVoyant: generating structured feedback on visual designs using a crowd of non-experts
Xu, A., Huang, S.-W., & Bailey, B. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531604 · Cited by 157
“For example, similar to results reported in [31], participants reported that feedback received from peers is often overly optimistic and lacks clear direction for improvement: “They [peers] know what went into it and how hard I worked.”
BackgroundResultInfluential
Metadata
- Authors
- Tohidi, M., Buxton, W., Baecker, R., & Sellen, A.
- Affiliations
- University of Toronto (CA)
- Microsoft Research, Toronto, Canada
- Microsoft Research (United Kingdom) (GB)
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2006-04-22
- Pages
- 1243-1252
- OpenAlex
- W2073271971
- Cited by
- 211
- References
- 18
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 10 · 2023: 11 · 2022: 4 · 2021: 20 · 2020: 12 · 2019: 9 · 2018: 15 · 2017: 9 · 2016: 10 · 2015: 15 · 2014: 10 · 2013: 13 · 2012: 12
- FWCI
- 5.26
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- OCAD University Open Research Repository (OCAD University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Keywords
- usability testing, interface design, design alternatives, user ratings, rating inflation, user feedback, design improvement, user-centered design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.