Getting the right design and the design right
The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
We present a study comparing usability testing of a single interface versus three functionally equivalent but stylistically distinct designs. We found that when presented with a single design, users give significantly higher ratings and were more reluctant to criticize than when presented with the same design in a group of three. Our results imply that by presenting users with alternative design solutions, subjective ratings are less prone to inflation and give rise to more and stronger criticisms when appropriate. Contrary to our expectations, our results also suggest that usability testing by itself, even when multiple designs are presented, is not an effective vehicle for soliciting constructive suggestions about how to improve the design from end users. It is a means to identify problems, not provide solutions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“and non-linear [13,87]: requires constant reconsideration of potential variations of existing text throughout the passage.”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Effective revision goes beyond minor editorial changes, and may help writers rework ideas, and powerfully affect their knowledge [23,87] as they explore alternative variations to find a line of argument [23].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Professional writers often write and rewrite text hundreds of times [12,87] and recommend rewriting as a core strategy for writing well [12,101].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Effective revision goes beyond minor editorial changes, and may help writers rework ideas, and powerfully affect their knowledge [23,87] as they explore alternative variations to find a line of argument [23].”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
In the citing paper
“Recognizing that revision is inherently nonlinear-with writers often revisiting earlier sections of a passageand recursive, manifesting in repeated cycles throughout the writing process [23,87], we aimed to align our LLM integrations with this fluid, iterative nature of revision.”
From ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models (2024) · cites this paper as [87]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
(1993) · Cited by 9,879
(1993) · Elsevier eBooks · doi:10.1016/c2009-0-21512-1 · Cited by 5,504
The Usability Engineering Lifecycle
NIELSEN, J. (1993) · Usability Engineering · doi:10.1016/b978-0-08-052029-2.50007-3 · Cited by 2,657
Paper Prototyping: The Fast and Easy Way to Design and Refine User Interfaces
Snyder, C. (2003) · Cited by 953
Rettig, M. (1994) · Communications of the ACM · doi:10.1145/175276.175288 · Cited by 440
Low vs. high-fidelity prototyping debate
Rudd, J., Stern, K., & Isensee, S. (1996) · Interactions · doi:10.1145/223500.223514 · Cited by 422
High-Fidelity or Low-Fidelity, Paper or Computer? Choosing Attributes when Testing Web Prototypes
Walker, M., Takayama, L., & Landay, J. A. (2002) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/154193120204600513 · Cited by 235
Sefelin, R., Tscheligi, M., & Giller, V. (2003) · CHI '03 extended abstracts on Human factors in computing systems - CHI '03 · doi:10.1145/765891.765986 · Cited by 209
Comparing usability problems and redesign proposals as input to practical systems development
Hornbæk, K., & Frøkjær, E. (2005) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1054972.1055027 · Cited by 148 · cited 3×
“Consequently, if one agrees with the findings of [5], then our study emphasizes the need to adopt approaches beyond standard usability testing techniques in order to generate redesign proposals.”
BackgroundInfluentialNotes on design practice: stories and prototypes as catalysts for communication
Erickson, T. D. (1995) · John Wiley & Sons, Inc. eBooks · Cited by 136
Cited by
Usability evaluation considered harmful (some of the time)
Greenberg, S., & Buxton, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357074 · Cited by 598
Peer and self assessment in massive online classes
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2505057 · Cited by 390
“Such work is especially important in design education because successful design often requires generating and reflecting on multiple ideas [Tohidi et al. 2006; Buxton 2007], and on exploration and iteration [Fallman 2003].”
BackgroundParallel prototyping leads to better design results, more divergence, and increased self-efficacy
Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1879831.1879836 · Cited by 367
“…combat this, creating multiple alternatives in parallel may encourage people to more effectively discover unseen constraints and opportunities [Cross 2006], enumerate more diverse solutions [Buxton 2007], and obtain more authentic and diverse feedback from potential users [Tohidi et al. 2006].”
BackgroundBringing Transparency Design into Practice
Eiband, M., Schneider, H., Bilandzic, M., Fazekas-Con, J., Haug, M., & Hussmann, H. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172961 · Cited by 250
“For the same reason, if multiple prototypes have been developed (as in our project), it is sensible to recruit different user groups for each prototype to investigate the effect of the different design solutions on users’ mental models (in contrast to the general recommendation to show users more than one prototype to…”
BackgroundVoyant: generating structured feedback on visual designs using a crowd of non-experts
Xu, A., Huang, S.-W., & Bailey, B. (2014) · Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing · doi:10.1145/2531602.2531604 · Cited by 157
“For example, similar to results reported in [31], participants reported that feedback received from peers is often overly optimistic and lacks clear direction for improvement: “They [peers] know what went into it and how hard I worked.”
BackgroundResultInfluentialDesigning with interactive example galleries
Lee, B., Srivastava, S., Kumar, R., Brafman, R., & Klemmer, S. R. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753667 · Cited by 195
“[47] demonstrated that exposing users to multiple alternative designs aids their discussions .”
BackgroundSketchStory: Telling More Engaging Stories with Data through Freeform Sketching
Lee, B., Kazi, R. H., & Smith, G. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.191 · Cited by 195
Design as exploration: creating interface alternatives through parallel authoring and runtime tuning
Hartmann, B., Yu, L., Allison, A., Yang, Y., & Klemmer, S. R. (2008) · Proceedings of the 21st annual ACM symposium on User interface software and technology · doi:10.1145/1449715.1449732 · Cited by 151
“Finally, presenting multiple alternatives in user studies facilitates participants’ ability to understand design tradeoffs and offer critical feedback [ 33 ].”
BackgroundA fieldwork of the future with user enactments
Odom, W., Zimmerman, J., Davidoff, S., Forlizzi, J., Dey, A. K., & Lee, M. K. (2012) · Proceedings of the Designing Interactive Systems Conference · doi:10.1145/2317956.2318008 · Cited by 175
Structuring, Aggregating, and Evaluating Crowdsourced Design Critique
Luther, K., Tolentino, J.-L., Wu, W., Pavel, A., Bailey, B. P., Agrawala, M., Hartmann, B., & Dow, S. P. (2015) · Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2675133.2675283 · Cited by 161
“Critiques can help novices to understand key principles in a domain [12], to compare alternatives [7, 27], to articulate the goals and assumptions behind their work, and to recognize how others perceive their work [15].”
BackgroundMenold, J., Jablokow, K., & Simpson, T. W. (2017) · Design Studies · doi:10.1016/j.destud.2017.03.001 · Cited by 157
Using participatory activities with seniors to critique, build, and evaluate mobile phones
Massimi, M., Baecker, R. M., & Wu, M. (2007) · Proceedings of the 9th international ACM SIGACCESS conference on Computers and accessibility · doi:10.1145/1296843.1296871 · Cited by 127
“We highlighted specific features of each phone and encouraged seniors to make critical comments about their designs [23].”
BackgroundSelf-citationPrototyping dynamics: sharing multiple designs improves exploration, group rapport, and results
Dow, S., Fortuna, J., Schwartz, D., Altringer, B., Schwartz, D., & Klemmer, S. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979359 · Cited by 138
“Many critique providers are aware that public feedback can be emotionally fraught; consequently they take care to temper criticism [53] and supplement critique with praise [27].”
BackgroundDogmas in the assessment of usability evaluation methods
Hornbæk, K. (2009) · Behaviour and Information Technology · doi:10.1080/01449290801939400 · Cited by 136
“Especially the key concern becomes whether a UEM helps generate design ideas (see Tohidi et al. 2006).”
Yuan, A., Luther, K., Krause, M., Vennix, S. I., Dow, S. P., & Hartmann, B. (2016) · Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing · doi:10.1145/2818048.2819953 · Cited by 97
“It helps the designer understand design principles [14], recognize how others perceive their work [24], and explore and compare alternatives [7, 41].”
BackgroundPeer and Self Assessment in Massive Online Classes
Kulkarni, C. E., Koh, P. W., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. (2014) · Understanding innovation · doi:10.1007/978-3-319-06823-7_9 · Cited by 132
Learning design patterns with bayesian grammar induction
Talton, J. O., Yang, L., Kumar, R., Lim, M., Goodman, N. D., & Měch, R. (2012) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/2380116.2380127 · Cited by 130
“In addition, it has been well established that considering alternatives in the design process itself facilitates discussion, improves the rate of convergence, and increases the quality of productions [43, 18, 11].”
BackgroundUsability Testing: A Practitioner's Guide to Evaluating the User Experience
Hertzum, M. (2020) · Synthesis lectures on human-centered informatics · doi:10.2200/s00987ed1v01y202001hci045 · Cited by 124
DataInk: Direct and Creative Data-Oriented Drawing
Xia, H., Henry Riche, N., Chevalier, F., De Araujo, B., & Wigdor, D. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173797 · Cited by 84
“Yet, they impose a high threshold [51] on users, as they still require to master complex functionalities, making it laborious to generate alternative designs, which can inhibit creative exploration.”
BackgroundIdeation decks: a card-based design ideation tool
Golembewski, M., & Selby, M. (2010) · Proceedings of the 8th ACM Conference on Designing Interactive Systems · doi:10.1145/1858171.1858189 · Cited by 82
“[4] note, designers often engage in ‘parallel design’ activities, and develop, explore, and evaluate multiple design solutions to a given problem.”
BackgroundUser sketches: a quick, inexpensive, and effective way to elicit more reflective user feedback
Tohidi, M., Buxton, W. A. S., Baecker, R. M., & Sellen, A. J. (2006) · Nordic Conference on Human-Computer Interaction · doi:10.1145/1182475.1182487 · Cited by 114
Self-citationTransparent Semiconducting Oxide Technology for Touch Free Interactive Flexible Displays
Lee, S., Jeon, S., Chaji, R., & Nathan, A. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2405767 · Cited by 111
Jones, W. (2007) · Morgan Kaufmann Publishers Inc. eBooks · Cited by 98
Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints
Swearngin, A., Wang, C., Oleson, A., Fogarty, J., & Ko, A. J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376593 · Cited by 58
“When designers explicitly compare alternatives, it can enable them to make stronger critiques and better decisions [12, 48].”
BackgroundThe Economies and Dimensionality of Design Prototyping: Value, Time, Cost, and Fidelity
Tiong, E., Seow, O., Camburn, B., Teo, K., Silva, A., Wood, K. L., Jensen, D., & Yang, M. C. (2019) · Journal of Mechanical Design · doi:10.1115/1.4042337 · Cited by 89
“Prototyping efforts for exactly 50 unique design concepts were executed in parallel to more cost-efficiently discover unseen constraints and opportunities, enumerate more diverse solutions, and obtain authentic and diverse feedback [31,32].”
BackgroundGUIComp: A GUI Design Assistant with Real-Time, Multi-Faceted Feedback
Lee, C., Kim, S., Han, D., Yang, H., Park, Y.-W., Kwon, B. C., & Ko, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376327 · Cited by 60
“One suggested approach to effectively produce acceptable designs is to perform iterations among design stages [48]: produce many share-able design alternatives [16, 34], compare the alternatives [68], and evaluate how general users feel while employing the designs (e.g., attention [71]).”
BackgroundCritique Style Guide: Improving Crowdsourced Design Feedback with a Natural Language Model
Krause, M., Garncarz, T., Song, J., Gerber, E. M., Bailey, B. P., & Dow, S. P. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025883 · Cited by 61
“In design work, feedback helps designers iterate on their in-progress solutions [15], compare alternatives [12, 45], learn about the problem, and refine their process.”
BackgroundXu, A., & Bailey, B. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145252 · Cited by 64
Martinez-Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., & Clayphan, A. (2016) · Journal of Learning Analytics · doi:10.18608/jla.2015.23.3 · Cited by 65
“In line with the UX evaluation literature (Tohidi et al., 2006), this inexpensive paper-based prototyping allowed exploration of initial user experience to drive the design of the awareness tool.”
BackgroundA Classroom Study of Using Crowd Feedback in the Iterative Design Process
Xu, A., Rao, H., Dow, S. P., & Bailey, B. P. (2015) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2675133.2675140 · Cited by 82
“Peer requests are quick and simple but burn social capital and the feedback may be biased by friendship or competition [30].”
BackgroundRiihiaho, S. (2017) · Wiley eBooks · doi:10.1002/9781118976005.ch14 · Cited by 80
Information Visualization and Proxemics: Design Opportunities and Empirical Findings
Jakobsen, M. R., Haile, Y. S., Knudsen, S., & Hornbæk, K. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.166 · Cited by 79
“It has been shown that users generate more comments when exposed to several alternatives than to just one [51].”
BackgroundJuxtapeer: Comparative Peer Review Yields Higher Quality Feedback and Promotes Deeper Reflection
Cambre, J., Klemmer, S., & Kulkarni, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173868 · Cited by 56
“Contrasting cases help develop nuanced understanding Across domains as diverse as algebra [39], design [12,49], and management [47], learners benefit from articulating the similarities and differences between examples [1].”
BackgroundInteractive Guidance Techniques for Improving Creative Feedback
Ngoon, T. J., Fraser, C. A., Weingarten, A. S., Dontcheva, M., & Klemmer, S. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173629 · Cited by 52
“Receiving feedback early on is important for learners to test alternatives and course-correct [5,41].”
BackgroundFake It to Make It: Exploratory Prototyping in HRI
Zamfirescu-Pereira, J. D., Sirkin, D. M., Goedicke, D., Lc, R., Friedman, N., Mandel, I., Martelaro, N., & Ju, W. G. (2021) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3434074.3446909 · Cited by 71
“…goal of prototyping methods is to help designers to generate more alternatives, more divergence in alternatives, and to enable them to elicit stronger and more detailed feedback from users [55]—critical in the case of social robots, but perhaps less so for the factory floor or industrial variety.”
BackgroundUnderstanding the Role of Alternatives in Data Analysis Practices
Liu, J., Boukhelifa, N., & Eagan, J. R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934593 · Cited by 56
“” Designers are encouraged to explore alternatives as the approach is believed to lead to better design solutions [40].”
BackgroundThe LATUX workflow: designing and deploying awareness tools in technology-enabled learning settings
Martínez‐Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., & Clayphan, A. J. (2015) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2723576.2723583 · Cited by 66
“Main lessons learnt: In accordance with UX literature, [32] this inexpensive paper-based prototyping allowed the exploration of initial user experience to drive the design of the awareness tool.”
BackgroundMethodLoup‐Escande, É., Burkhardt, J., & Richir, S. (2013) · Le travail humain · doi:10.3917/th.761.0027 · Cited by 66
Yen, Y.-C. G., Dow, S. P., Gerber, E., & Bailey, B. P. (2017) · Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition · doi:10.1145/3059454.3059468 · Cited by 40
“It may therefore be most beneficial to perform the reflection activity before reviewing feedback during the early design stage when the focus is typically on conceptual issues and to perform it after reviewing feedback during the late stage when the focus is typically on improving aesthetics [44].”
BackgroundRiche, N. H., Offenwanger, A., Gmeiner, F., Brown, D., Romat, H., Pahud, M., Marquardt, N., Inkpen, K. M., & Hinckley, K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714259 · Cited by 62
“Seminal research demonstrated that it is critical to explore alternative designs early and throughout the whole process [49, 69].”
When Cognition Interferes with Innovation: Overcoming Cognitive Obstacles to Design Thinking
Butler, A. G., & Roberto, M. A. (2018) · Research-Technology Management · doi:10.1080/08956308.2018.1471276 · Cited by 61
Hive: Collective Design Through Network Rotation
Salehi, N., & Bernstein, M. S. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274420 · Cited by 60
“[73] for a useful frame: they distinguish two phases of design: “getting the right design”, early on when the idea is still malleable, and “getting the design right”, later when the focus is refinement.”
BackgroundParallel Prototyping Leads to Better Design Results, More Divergence, and Increased Self-efficacy
Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. (2011) · Springer eBooks · doi:10.1007/978-3-642-21643-5_8 · Cited by 59
Online Feedback Exchange: A Framework for Understanding the Socio-Psychological Factors
Foong, E., Dow, S. P., Bailey, B. P., & Gerber, E. M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025791 · Cited by 44
Firestorm: a brainstorming application for collaborative group work at tabletops
Clayphan, A. J., Collins, A., Ackad, C. J., Kummerfeld, B., & Kay, J. (2011) · ACM International Conference on Interactive Tabletops and Surfaces (ITS) · doi:10.1145/2076354.2076386 · Cited by 57
“A key aspect of the experimental design was to provide participants with two interface solutions, to facilitate elicitation of constructive advice about both [25].”
MethodReza, M., Laundry, N. M., Musabirov, I., Dushniku, P., Yu, Z. Y. “., Mittal, K., Grossman, T., Liut, M., Kuzminykh, A., & Williams, J. J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641899 · Cited by 52
“To get the right design [89] the lead author iterated through multiple versions of each interface element, and tested them with five pilot users during in-depth brainstorming and design-review sessions over six months.”
InfluentialDataInk: Direct and Creative Data-Oriented Drawing
Xia, H., Riche, N. H., Chevalier, F., Araujo, B. D., & Wigdor, D. (2018) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3170427.3186471 · Cited by 51
Paragon: An Online Gallery for Enhancing Design Feedback with Visual Examples
Kang, H. B., Amoako, G., Sengupta, N., & Dow, S. P. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174180 · Cited by 36
“Recognizing and weighing alternatives provides value through the design process [44].”
Using Anonymity and Communal Efforts to Improve Quality of Crowdsourced Feedback
Hui, J. S., Glenn, A., Jue, R., Gerber, E. M., & Dow, S. P. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13229 · Cited by 46
Prototyping Dynamics: Sharing Multiple Designs Improves Exploration, Group Rapport, and Results
Dow, S. P., Fortuna, J., Schwartz, D., Altringer, B., Schwartz, D. L., & Klemmer, S. (2012) · Springer eBooks · doi:10.1007/978-3-642-31991-4_4 · Cited by 46
Reza, M., Anastasopoulos, I., Bhandari, S., & Pardos, Z. A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714051 · Cited by 41
“With the right design [55], such interfaces could enable experts to steer the output of LLMs toward content that better aligns with the nuances and needs of their domains and transform the role of the subject matter expert from a producer to a curator — a competent and critical judge who instructs…”
More Than a Gimmick - Digital Tools for Boardgame Play
Rogerson, M. J., Sparrow, L. A., & Gibbs, M. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3474688 · Cited by 42
"It makes you think": Provocations Help Restore Critical Thinking to AI-Assisted Knowledge WorkPreprint
Drosos, I., Sarkar, A., Xiaotong, Xu, & Toronto, N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.17247 · Cited by 43
InkSight: Leveraging Sketch Interaction for Documenting Chart Findings in Computational Notebooks
Lin, Y., Li, H., Yang, L., Wu, A., & Qu, H. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3327170 · Cited by 40
The Effect of Parallel Prototyping on Design Performance, Learning, and Self-Efficacy
Dow, S. P., Glassco, A., Kass, J., Schwarz, M., & Klemmer, S. (2009) · Cited by 40
Gaining design insight through interaction prototyping tools
Hartmann, B. (2009) · Cited by 38
Interacting with social networks of intelligent things and people in the world of gastronomy
Console, L., Antonelli, F., Biamino, G., Carmagnola, F., Cena, F., Chiabrando, E., Cuciti, V., Demichelis, M., Fassio, F., Franceschi, F., et al. (2013) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2448116.2448120 · Cited by 36
“problems of usability test described in [Tohidi et al. 2006], especially regarding the post-experiment questionnaire, since it is well-known that usability tests were vulnerable to overly positive results.”
BackgroundUnderstanding Version Control as Material Interaction with Quickpose
Rawn, E., Li, J., Paulos, E., & Chasins, S. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581394 · Cited by 35
“Prior empirical studies ofer evidence that broad exploration [21], variation [86], iteration [11], and refection [13] are valuable parts of a design process.”
BackgroundGlobal health front-end medical device design: The use of prototypes to engage stakeholders
Coulentianos, M. J., Rodríguez-Calero, I., Daly, S. R., & Sienko, K. H. (2020) · Development Engineering · doi:10.1016/j.deveng.2020.100055 · Cited by 36
From in the Class or in the Wild? Peers Provide Better Design Feedback Than External Crowds
Wauck, H. C., Yen, Y.-C. G., Fu, W., Gerber, E. M., Dow, S. P., & Bailey, B. P. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025477 · Cited by 34
“At this stage, students may benefit most from reflecting on how they approached the design, how the design compares to other solutions, and how it fits in a broader social and cultural context [37].”
BackgroundInfluentialAre Engineers Condemned to Design? A Survey on Software Engineering and UI Design in Switzerland
Vukelja, L., Müller, L., & Opwis, K. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74800-7_50 · Cited by 34
“Most end users simply see their own work activities, but lack a broader view of the complete workflow [14, 26].”
BackgroundPrototyping in PLACE: a scalable approach to developing location-based apps and games
Bowser, A., Hansen, D. L., Raphael, J., Reid, M., Gamett, R. J., He, Y., Rotman, D., & Preece, J. J. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466202 · Cited by 34
Creating a rural community display with local engagement
Taylor, N., & Cheverst, K. (2010) · Designing Interactive Systems Conference · doi:10.1145/1858171.1858209 · Cited by 34
“Although the term is not strictly defined [16], participation of the users themselves and iterative approaches are often advised [11], as is the use of multiple prototypes to allow for better comparison by users [28].”
BackgroundMethodModel Sketching: Centering Concepts in Early-Stage Machine Learning Model Design
Lam, M. S., Ma, Z., Li, A., Freitas, I., Wang, D., Landay, J. A., & Bernstein, M. S. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581290 · Cited by 33
“The quick and plentiful attributes go hand in hand: methods with quick turnaround times enable practitioners to explore parallel design paths, an approach that supports more iterations, combats harmful design fixation, and leads to more diverse, higher-quality designs [13, 59].”
BackgroundArazy, O., Lindberg, A., Rezaei, M., & Samorani, M. (2020) · MIS Quarterly · doi:10.25300/misq/2020/15379 · Cited by 33
Beyond Schematic Capture: Meaningful Abstractions for Better Electronics Design Tools
Lin, R., Ramesh, R., Iannopollo, A., Sangiovanni-Vincentelli, A. L., Dutta, P., Alon, E., & Hartmann, B. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300513 · Cited by 31
“Additionally, by having participants compare-and-contrast between two interfaces, and asking for the rationale behind answers, we hope to reduce the effects of acquiescence bias [36].”
BackgroundHarmonyCut: Supporting Creative Chinese Paper-cutting Design with Form and Connotation Harmony
Wang, H., Qiu, T., Li, J., Lu, Z., & Ma, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714159 · Cited by 32
“However, some challenges persist: GenAI-generated content is often end-to-end complete, potentially causing users to become overly reliant on the model, merely selecting the model’s suggestions instead of fostering genuine creativity [82].”
Tracing impact in a usability improvement process
Uldall-Espersen, T., Frøkjær, E., & Hornbæk, K. (2007) · Interacting with Computers · doi:10.1016/j.intcom.2007.08.001 · Cited by 30
Bromark, K., Knutsson, O., & Weitz, Y. S. (2023) · Children and Youth Services Review · doi:10.1016/j.childyouth.2023.106856 · Cited by 27
XAlgo: a Design Probe of Explaining Algorithms’ Internal States via Question-Answering
Rebanal, J., Combitsis, J., Tang, Y., & Chen, X. '. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450676 · Cited by 17
The people-prototype problem: understanding the interaction between prototype format and user group
Sellen, K., Massimi, M. A., Lottridge, D. M., Truong, K. N., & Bittle, S. A. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518799 · Cited by 27
“Both videos and storyboards have their own properties that affect the type of feedback they elicit [ 13 ] as do other prototype formats [5].”
BackgroundKim, K. G., Davis, R. L., Coppi, A. E., Cattaneo, A., & Dillenbourg, P. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501854 · Cited by 17
“These alternatives provide designers with a more complete understanding of the design space [13, 14] and comparing alternatives can help them make stronger critiques and better design decisions [8, 21, 43].”
BackgroundCOLDECO: An End User Spreadsheet Inspection Tool for AI-Generated Code
Ferdowsi, K., Williams, J., Drosos, I., Gordon, A. D., Negreanu, C., Polikarpova, N., Sarkar, A., & Zorn, B. (2023) · 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00017 · Cited by 17
“Following recommendations from [48], we tried to mitigate inflation in subjective ratings and qualitative feedback by presenting multiple designs within C OL D ECO to the participants.”
MethodMagellan, an evolutionary system to foster user interface design creativity
Masson, D. H., Demeure, A., & Calvary, G. (2010) · ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) · doi:10.1145/1822018.1822032 · Cited by 26
“INTRODUCTION “Getting the right design” and “getting the design right” [16] are complementary challenges in Human Computer Interaction (HCI).”
BackgroundExiting the Design Studio: Leveraging Online Participants for Early-Stage Design Feedback
Ma, X., Li, Y., Forlizzi, J., & Dow, S. P. (2015) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/2675133.2675174 · Cited by 26
“Designers need to be able to interpret if users favor a design alternative just because they want to appease the designer [28].”
BackgroundAesthetics in Human-Computer Interaction: Views and Reviews
Ahmed, S. U., Mahmud, A. A., & Bergaust, K. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02574-7_63 · Cited by 26
“…of an application, mobile phone, web sites etc. Interaction with a system [26],[27], [28], [12], [29] Interactive art installations, museum guide, interactive learning system, ATM machines etc. Usability and User Experience [7] [30], [27], [31], [32], [25] Users’ feelings, emotion, usability.”
BackgroundAccessible haptic user interface design approach for users with visual impairments
Kim, H. N., Smith‐Jackson, T. L., & Kleiner, B. M. (2013) · Universal Access in the Information Society · doi:10.1007/s10209-013-0325-0 · Cited by 23
“Tohidi et al. used a two-tailed t test to compare different prototyping techniques based on the number of users’ statements.”
MethodInfluentialVibe Coding in Product Teams: Reconfiguring AI-Assisted Workflows, Prototyping, and Collaboration
Li, J., Hou, Y., Lin, L., Zhu, R., Cao, H., & Ali, A. E. (2026) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3808045.3808062 · Cited by 23
“At the same time, the acceleration of prototype creation risks privileging speed, what we see as intending the right design”, over reflective exploration of values and alternatives through “designing the right intention” (cf., [89]).”
Planz to put our digital information in its place
Jones, W., Hou, D., Sethanandha, B. D., Bi, S., & Gemmell, J. (2010) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1753846.1753866 · Cited by 22
“Tohidi et al. [17] observed that “when presented with a single design, users give significantly higher ratings and were more reluctant to criticize ” (p. 1243).”
BackgroundDesignWeaver: Dimensional Scaffolding for Text-to-Image Product Design
Tao, S., Liang, I., Peng, C., Wang, Z., Palani, S., & Dow, S. P. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714211 · Cited by 21
“Presenting multiple alternative options simultaneously allows designers to explore a broader range of possibilities and engage clients early in the design process [29, 97].”
Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents
Sharma, N., Zhang, Z., Lee, D., Krishnan, N., Ren, G.-J., Xiao, Z., & Li, Y. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791600 · Cited by 21
“To prevent users from blindly accepting the first plausible output, the Split-View Comparison feature allows side-by-side evaluation of different response versions generated from different feedback[120].”
Computer supported novice group critique
Easterday, M. W., Lewis, D. R., Fitzpatrick, C., & Gerber, E. M. (2014) · Designing Interactive Systems Conference · doi:10.1145/2598510.2600889 · Cited by 21
“Second, designers typically present works-in-progress that require a great deal of grounding [4].”
BackgroundDesigning an Eyes-Reduced Document Skimming App for Situational Impairments
Khan, T. A., Yoon, D., & McGrenere, J. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376641 · Cited by 21
“To get the right design [54] the lead author iterated through 3 versions of the gesture sets, tested with a total of 8 pilot users during in-depth brainstorming and design-review sessions over 3 months.”
Understanding and Supporting Creativity in Design
Warr, A. (2006) · The University of Bath Online Publications Store (The University of Bath) · Cited by 21
The Economies and Dimensionality of Prototyping: Value, Time, Cost and Fidelity
Tiong, E., Seow, O., Teo, K., Silva, A., Wood, K. L., Jensen, D., & Yang, M. C. (2018) · ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (IDETC/CIE) · doi:10.1115/detc2018-85747 · Cited by 20
“Prototyping efforts for 50 unique design concepts were executed in parallel to more cost-efficiently discover unseen constraints and opportunities, enumerate more diverse solutions, and obtain more authentic and diverse feedback [26, 27].”
BackgroundPollack, A. H., Simon, T. D., Snyder, J., & Pratt, W. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103201 · Cited by 20
“Designers simultaneously create multiple designs and prototypes to explore how to best meet the needs of their users, reduce the cognitive burden associated with the design, and maximize feedback [10,11].”
BackgroundOppenlaender, J., Kuosmanen, E., Lucero, A., & Hosio, S. J. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445380 · Cited by 20
“On the other hand, feedback from peers may be positively biased [36], and students were shown to appreciate and prefer feedback from external and anonymous sources [6, 27].”
BackgroundCelerometer and idling reminder: persuasive technology for school bus eco-driving
Pace, T., Ramalingam, S., & Roedl, D. (2007) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1240866.1240954 · Cited by 20
“Evaluation Inspired by the study Getting the Right Design and the Design Right: Testing Many is Better Than One [9], we conducted a pilot study of three different HUD concepts on drivers (n=5) with varying levels of driving experience.”
MethodDesigning Crowdcritique Systems for Formative Feedback
Easterday, M. W., Lewis, D. R., & Gerber, E. M. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-016-0125-9 · Cited by 19
“The diverse feedback from group critique helps highlight gaps between the intentions and perceptions of a design (Tohidi et al. 2006) and helps designers overcome impasses and indecision common in design projects (Lande and Leifer 2010).”
BackgroundThe Link between Inclusive Design and Innovation: Some Key Elements
Fuglerud, K. S., & Sloan, D. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39232-0_5 · Cited by 19
“It is mainly a means to identify problems, not to provide solutions [36].”
BackgroundPeerdea: Co-Designing a Peer Support Platform with Creative Entrepreneurs
Kotturi, Y., Yu, J., Khadpe, P., Gatz, E., Zheng, H., Fox, S., & Kulkarni, C. E. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637384 · Cited by 18
“First, to communicate malleability of the prototype and increase chances for critical feedback [83], Peerdea initially had a limited feature set with a simple user interface.”
BackgroundMethodInfluentialExploring inexperienced user performance of a mobile tablet application through usability testing.
Gatsou, C., Politis, A. C., & Zevgolis, D. (2013) · Cited by 18
Behavior Change Interventions Combating Online Misinformation: A Scoping Review
Konstantinou, L., & Karapanos, E. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713127 · Cited by 17
“…uncover unseen constraints, avoid fxation, and derive better and diverse solutions - are well documented in HCI and design literature (see [38, 119]), yet, parallel design has been reported to be infrequently practiced due to small budgets, lack of time and its propensity to lead to decision…”
Towards Designing Unbiased Replication Studies in Information Visualization
Sukumar, P. T., & Metoyer, R. (2018) · IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) · doi:10.1109/beliv.2018.8634261 · Cited by 17
“without comparisons, can also be biased in that more favorable ratings may be obtained for the condition being tested [66].”
BackgroundThe Unique Challenges for Creative Small Businesses Seeking Feedback on Social Media
Kotturi, Y., Blaising, A., Fox, S., & Kulkarni, C. E. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449089 · Cited by 17
“While such resistance may seem to be not aligned with best practice [67], some small business owners we met described how such a request was synonymous with admitting a lack of knowledge, and others felt it would make them seem self-aggrandizing.”
BackgroundCao, Y., Huang, Y., Truong, A., Shin, H. V., & Xia, H. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713401 · Cited by 16
Connecting Designing and Engineering Activities
Beyhl, T., Berg, G., & Giese, H. (2013) · Springer eBooks · doi:10.1007/978-3-319-01303-9_11 · Cited by 16
Why do Designers in the "Wild" Wait to Seek Feedback until Later in their Design Process?
Kotturi, Y., & Kingston, M. (2019) · Creativity and Cognition · doi:10.1145/3325480.3326580 · Cited by 15
“First, a temporal consideration: early in one's design process, especially in a successful design process [11], there are usually many ideas to consider in the design space; moreover this space is often malleable, as each idea can be taken in several different developmental directions.”
BackgroundMethodInfluentialAsk Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback
Lekschas, F., Ampanavos, S., Siangliulue, P., Pfister, H., & Gajos, K. Z. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445507 · Cited by 14
Yarmand, M., Chen, C., Cheng, K., Murphy, J. D., & Weibel, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642453 · Cited by 14
“The researchers presented and described both prototypes at once, because showing alternative design solutions can produce stronger and more authentic criticisms [86].”
Background
Metadata
- Authors
- Tohidi, M., Buxton, W., Baecker, R., & Sellen, A.
- Affiliations
- University of Toronto (CA)
- Microsoft Research, Toronto, Canada
- Microsoft Research (United Kingdom) (GB)
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2006-04-22
- Pages
- 1243-1252
- OpenAlex
- W2073271971
- Cited by
- 211
- References
- 18
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 10 · 2023: 11 · 2022: 4 · 2021: 20 · 2020: 12 · 2019: 9 · 2018: 15 · 2017: 9 · 2016: 10 · 2015: 15 · 2014: 10 · 2013: 13 · 2012: 12
- FWCI
- 5.26
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- OCAD University Open Research Repository (OCAD University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Keywords
- usability testing, interface design, design alternatives, user ratings, rating inflation, user feedback, design improvement, user-centered design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.