Getting the right design and the design right
The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
We present a study comparing usability testing of a single interface versus three functionally equivalent but stylistically distinct designs. We found that when presented with a single design, users give significantly higher ratings and were more reluctant to criticize than when presented with the same design in a group of three. Our results imply that by presenting users with alternative design solutions, subjective ratings are less prone to inflation and give rise to more and stronger criticisms when appropriate. Contrary to our expectations, our results also suggest that usability testing by itself, even when multiple designs are presented, is not an effective vehicle for soliciting constructive suggestions about how to improve the design from end users. It is a means to identify problems, not provide solutions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“and non-linear [13,87]: requires constant reconsideration of potential variations of existing text throughout the passage.”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Effective revision goes beyond minor editorial changes, and may help writers rework ideas, and powerfully affect their knowledge [23,87] as they explore alternative variations to find a line of argument [23].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Professional writers often write and rewrite text hundreds of times [12,87] and recommend rewriting as a core strategy for writing well [12,101].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Effective revision goes beyond minor editorial changes, and may help writers rework ideas, and powerfully affect their knowledge [23,87] as they explore alternative variations to find a line of argument [23].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Recognizing that revision is inherently nonlinear-with writers often revisiting earlier sections of a passageand recursive, manifesting in repeated cycles throughout the writing process [23,87], we aimed to align our LLM integrations with this fluid, iterative nature of revision.”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
(1993) · Cited by 9,879
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,504
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
Paper Prototyping: The Fast and Easy Way to Design and Refine User Interfaces
Snyder, C. (2003) · Cited by 953
Rettig, M. (1994) · Communications of the ACM · doi:10.1145/175276.175288 · Cited by 440
Low vs. high-fidelity prototyping debate
Rudd, J., Stern, K., & Isensee, S. (1996) · Interactions · doi:10.1145/223500.223514 · Cited by 422
High-Fidelity or Low-Fidelity, Paper or Computer? Choosing Attributes when Testing Web Prototypes
Walker, M., Takayama, L., & Landay, J. A. (2002) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120204600513 · Cited by 235
Sefelin, R., Tscheligi, M., & Giller, V. (2003) · CHI '03 extended abstracts on Human factors in computing systems - CHI '03 · doi:10.1145/765891.765986 · Cited by 209
Comparing usability problems and redesign proposals as input to practical systems development
Hornbæk, K., & Frøkjær, E. (2005) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055027 · Cited by 148 · cited 3×
“Consequently, if one agrees with the findings of [5], then our study emphasizes the need to adopt approaches beyond standard usability testing techniques in order to generate redesign proposals.”
BackgroundInfluentialNotes on design practice: stories and prototypes as catalysts for communication
Erickson, T. D. (1995) · John Wiley & Sons, Inc. eBooks · Cited by 136
Cited by
Items 201–269 of 269
Sketching the Polyphonic Design Space of Theme Parks
Durrant, A. C., Golembewski, M., & Kirk, D. S. (2018) · Human-computer interaction series · doi:10.1007/978-3-319-68213-6_9 · Cited by 1
“A particular strength of the card technique was that it could be used to explore multiple and alternative design situations (Tohidi et al., 2006).”
ParaXplore: Demystifying the Exploration of Large Design Spaces
Shireen, N. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3308459 · Cited by 1
Clemmensen, T. (2021) · Human-computer interaction series · doi:10.1007/978-3-030-71796-4_3 · Cited by 1
Clemmensen, T. (2021) · Human-computer interaction series · doi:10.1007/978-3-030-71796-4_4 · Cited by 1
Vaugh, T., & Ryan, M. (2015) · Maynooth University ePrints and eTheses Archive (Maynooth University) · doi:10.34961/1523 · Cited by 1
Bergstrand, F., & Gabrielsson, M. N. (2008) · Gothenburg University Publications Electronic Archive (Gothenburg University) · Cited by 1
Evaluating new interactions in healthcare: challenges and approaches
Randell, R., Fitzpatrick, G., Wilson, S., Mamykina, L., & Tang, C. (2009) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1520340.1520737 · Cited by 1
“Is it necessary to explore the impact of all of these components? If not, which should take priority? What other components should we consider? How can we understand the impact of these different components? Should we be testing multiple designs in order to, for example, understand the benefits of different interface…”
BackgroundExtending Software Engineering Collaboration towards the Intersection of Software and Art
Ahmed, S. U. (2011) · Cited by 1
Designing an effective user interface for the Android tablet environment
Chang, G. (2015) · Open University of Cape Town (University of Cape Town) · Cited by 1
Exploring Electronic Storyboards as Interdisciplinary Design Tools for Pervasive Computing
Forsyth, J. B. (2015) · VTechWorks (Virginia Tech) · Cited by 1
An information-theoretic account of human–computer interaction
Trendafilov, D. (2017) · Enlighten: Theses (The University of Glasgow) · Cited by 1
Amano, T. (2019) · University of the Arts London Research Online (University of the Arts London) · Cited by 1
The trade-off between usability and security in the context of eGovernment
Alshamsi, A. (2019) · Keele Research Repository (Keele University) · Cited by 1
FlexMind: Supporting Deeper Creative Thinking with LLMsPreprint
yang, Y., Mohanty, V., Chen, Y.-Y., Hong, M. K., Martelaro, N., & Kittur, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.21685 · Cited by 1
“Developing multiple concepts simultaneously, rather than serially, also encourages comparison and helps designers de-tach from their initial concepts, facilitating more critical evaluation and integration of ideas [14, 77].”
Data expression : understanding and supporting alternatives in data analysis processes
Liu, J. (2021) · theses.fr (ABES) · doi:10.70675/708a4fddz43d4z4371z8092z4eea2b2b6600 · Cited by 1
Participatory, not Punitive: Student-Driven AI Policy Recommendations in a Design ClassroomPreprint
Seki, K., Vijay, M., & Kotturi, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.10851 · Cited by 1
“Participants gravitated toward divergent thinking—the essential stage in design that focuses on generating many ideas before converging on a subset to pursue [120].”
Rethinking the UI of GenUI: A Tale of Two DesignsPreprint
Chen, X. `., Petridis, S., Du, R., & Li, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.13843 · Cited by 1
“• Workflow: Instead of going depth-first, we employ a breadth-first workflow, found in numerous creativity support tools Tohidi et al. (2006); Lim et al. (2008); Suh et al. (2024); Chang et al. (2018); Ma et al. (2025), that starts with generating a design space for users to see and explore…”
Kunert, T. (2009) · Human-computer interaction series · doi:10.1007/978-1-84882-275-7_6
What makes a good design critic? food design vs. product design criticism
Venkatacharya, P. S., Kessler, J., Hardeman, T., Seiber, E., & Buxton, B. (2010) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1753846.1753933
Evaluating new interactions in healthcare
Randell, R., & Mamykina, L. (2009)
Design and Evaluation of a Self-monitoring Application for Chronic Headaches
Rahman, S. M. W. (2015) · Libraries and Cultural Resources (University of Calgary) · doi:10.11575/prism/25568
Chakraborty, A. (2015) · India HCI (IndiaHCI) · doi:10.1145/2835966.2836286
Vesuvala, C. X. (2009) · Summit (Simon Fraser University)
Lazy Disclosure - Mixing Cloud and Local Storage
Bouck-Standen, D., Cassens, J., & Herczeg, M. (2012) · IICS
Designing and Evaluating a Tablet Application for Guitar Effect Control
Ekroll, L. K. (2015) · BIBSYS Brage (BIBSYS (Norway))
Procyshyn, C. (2012) · CRC Press eBooks · doi:10.3109/9781439825440-42
Text and visual annotation tools for scalable design feedback
Mangutov, K. (2016) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Worker Expertise and Expert Rubrics in Crowdsourced Design Critique
Yuan, A., Luther, K., Krause, M., Dow, S. P., & Hartmann, B. (2015) · UC Berkeley
Fontana, F. (2018) · Aaltodoc (Aalto University)
Yehia, Z. K. M. (2020) · Nottingham ePrints (University of Nottingham)
Expanding Interface Design Capabilities through Semantic and Data-Driven Analyses
Swearngin, A. (2020) · ResearchWorks at the University of Washington (University of Washington)
Professional Design Feedback: Types and Expert Preferences
Sakalli, I. (2017) · Carleton University's Institutional Repository (MacOdrum Library, Carleton University) · doi:10.22215/etd/2017-11823
Lim, C. S., Foong, P. S., Koh, G. C. H., & Yeow, A. Y. K. (2021) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3411763.3443453
“Tohidi et al’s [15] oft-cited paper on getting the design right suggests that ideation generates more elaborate and more honest prototype feedback from test users.”
BackgroundInteractive Attunement Catalyzes Creative Learning
Ngoon, T. J. (2021) · eScholarship (California Digital Library)
E-Totem, Digital Locative Media to Support E-Participation in Cities
Furtado, V., Lima, L., Chagas, D. A., Pinheiro, V. C. M., Caminha, C., Furtado, E. S., & Mafra, M. L. P. (2021) · IGI Global eBooks · doi:10.4018/978-1-6684-3706-3.ch070
Clemmensen, T. (2021) · Human-computer interaction series · doi:10.1007/978-3-030-71796-4_8
(2008) · Elsevier eBooks · doi:10.1016/b978-012370866-3.50018-7
Optimizing Mentor-Student Communication with Symbolic Design for Message StatesPreprint
Jin, Y., & Yu, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.04227
“Another study [TBBS06] analyzed usability testing of a single interface with three designs that had the same functionality but different styles.”
BackgroundIncorporating Different Verbal Cues to Improve Text-Based Computer-Delivered Health MessagingPreprint
Cox, S. R. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.13633
Palea, D., Guo, A., Nair, A., Anderson, R., Charagulla, N., Su, N. M., & Lee, D. T. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686984
“…Process , User Information , and Comfort and Safety ), our interview involved asking users 1-2 questions on the overall topic, and then eliciting their thoughts through asking them to compare different options [104] for different aspects of a group formation platform (see Table 2 ).”
Gleißl, B., Haslbeck, A., Hoesl, A., & Liu, M. (2025) · Cognition Technology & Work · doi:10.1007/s10111-025-00800-1
“Usually, the favored design direction is then further explored in the second diamond (‘getting the design right’, Tohidi et al. 2006).”
Score-Based Sentence Completion for Interactive Systems Evaluation
Camara, F. G., Cockton, G., & Calvary, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05005-2_14
InfoVids: Reimagining the Viewer Experience with Alternative Visualization-Presenter RelationshipsPreprint
Chung, J. W., Zhou, T., Chen, I., Hsu, K., Rossi, R. A., Siu, A. F., Guo, S., Dernoncourt, F., Tompkin, J., & Huang, J. J. S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.03164
Puranik, A., Chen, E., Peiris, R. L., & Kong, H.-K. H. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816977
“This choice follows established prior work showing that early-stage prototypes are effective for exploration and reflection rather than evaluation, helping participants articulate needs, expectations, and values around emerging technologies [21, 24, 67].”
Prototyping von Rich Internet Applikationen
Binggeli, A., Blume, M., & Sun, Y. (2009) · HSR (HSR University of Applied Sciences Rapperswil)
Exploring the Role of Social Support when Integrating Generative AI into Small Business Workflows
Lauro, Q. R., Bigham, J. P., & Kotturi, Y. (2024) · Maryland Shared Open Access Repository (USMAI Consortium) · doi:10.13016/m2h895-x3xv
Sanobar, Y. (2023) · cIRcle (University of British Columbia) · doi:10.14288/1.0437959
Kaluzny, K., & Johansson, H. (2024) · DiVA (Stockholm University)
Nagaraj Kashyap, D. (2016) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Design av brukeropplevelser i systemutvikling: en casestudie
Ruud, I. (2014) · NVA (Nasjonalt vitenarkiv)
Human Technology, 2011 VOLUME 7, NUMBER 3 (The entire issue)
(2011) · Jyväskylä University Digital Archive (University of Jyväskylä)
Larsson, A. (2023) · DiVA (Digitala Vetenskapliga Arkivet)
Human-Centered Program Synthesis
Ferdowsifard, K. (2024) · eScholarship (California Digital Library)
Kong, H. K. (2019) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Prototypointimenetelmien käyttö käyttäjäkeskeisessä suunnitteluprosessissa
Karttunen, T. (2017) · Aaltodoc (Aalto University)
Design-tekniikoiden käyttö ja arviointi pienessä yrityksessä
Naalisvaara, L. (2015) · Aaltodoc (Aalto University)
Designing for Knowledge Capture in Fabrication Workshops
Rigaud, C. (2023) · theses.fr (ABES) · doi:10.70675/24e7cef6z5867z4a8fz951fz9b709dc4ca10
Kokemuksia käytettävyystestauksesta: Ääneenajattelun ja ohjaajan läsnäolon vaikutuksia
Riihiaho, S. (2015) · Aaltodoc (Aalto University)
Park, H., & Kim, H. K. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2664690
AnimationDiff: A Visual Comparison Tool for Generated 3D Character Animations
Sidenmark, L., Zhou, Q., Fitzmaurice, G. W., & Anderson, F. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3812971
“Visual comparison is a fundamental task in creative and knowledge-intensive tasks for sensemaking and understanding the design space [19], ultimately aiding the creator to the right design [57].”
Vierula, S. (2026) · Lauda (University of Lapland)
The Effects of Early Socio-Technical Decisions on UXDebt
Rodríguez, A., Grigera, J., Gardey, J. C., Farfallini, L., Rossi, G. H., & Garrido, A. (2026) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1145/3794915.3795788
“Teams may also pre-maturely commit to early User Interface (UI) solutions, making subsequent changes harder [52].”
How interface design affects the composition, interpretation, and utilization of feedback
Crain, P. A. (2022) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Lee, S. W., Choi, J., Yun, Y., Jin, S., Jang, Y., Hwang, G., Park, S. W., Ban, S., & Hyun, K. H. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2739092
Mylar: Exploring Self-Curation as a Reflexive Creative PracticePreprint
Chier, K., Almeda, S. G., & Hartmann, B. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.21213
“Prior research in parallel and comparative prototyping has shown that alternatives can support comparative reasoning and situated exploration [25], sharpen critique and evaluation [67], and foster greater divergence while reducing friction [20].”
Failing Successfully: Reducing Academic Perfectionism with Transformational Games
Credit, K., DeCossard, R., Teffera, L., Windust, K., Sookhoo, R., Xu, S., Deng, E., Musuraca, R. S., & Hammer, J. (2026) · Annual Symposium on Computer-Human Interaction in Play · doi:10.1145/3800965.3834325
Design and implementation of a timeline for the Web
Jansson, T. (2015) · DiVA (Digitala Vetenskapliga Arkivet)
Metadata
- Authors
- Tohidi, M., Buxton, W., Baecker, R., & Sellen, A.
- Affiliations
- University of Toronto (CA)
- Microsoft Research, Toronto, Canada
- Microsoft Research (United Kingdom) (GB)
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2006-04-22
- Pages
- 1243-1252
- OpenAlex
- W2073271971
- Cited by
- 211
- References
- 18
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 10 · 2023: 11 · 2022: 4 · 2021: 20 · 2020: 12 · 2019: 9 · 2018: 15 · 2017: 9 · 2016: 10 · 2015: 15 · 2014: 10 · 2013: 13 · 2012: 12
- FWCI
- 5.26
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- OCAD University Open Research Repository (OCAD University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Keywords
- usability testing, interface design, design alternatives, user ratings, rating inflation, user feedback, design improvement, user-centered design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.