CoChat
Sign Up
Cited by 14Usability and User Interface DesignMobile Crowdsensing and CrowdsourcingInnovative Human-Technology Interaction

Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback

Open access
Lekschas, Fritz, Ampanavos, Spyridon, Siangliulue, Pao, Pfister, Hanspeter, Gajos, Krzysztof Z.·2021·Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3411764.3445507
Open this paper in CoChat — free accountOpen PDF

Abstract

Crowdsourced design feedback systems are emerging resources for getting large amounts of feedback in a short period of time. Traditionally, the feedback comes in the form of a declarative statement, which often contains positive or negative sentiment. Prior research has shown that overly negative or positive sentiment can strongly influence the perceived usefulness and acceptance of feedback and, subsequently, lead to ineffective design revisions. To enhance the effectiveness of crowdsourced design feedback, we investigate a new approach for mitigating the effects of negative or positive feedback by combining open-ended and thought-provoking questions with declarative feedback statements. We conducted two user studies to assess the effects of question-based feedback on the sentiment and quality of design revisions in the context of graphic design. We found that crowdsourced question-based feedback contains more neutral sentiment than statement-based feedback. Moreover, we provide evidence that presenting feedback as questions followed by statements leads to better design revisions than question- or statement-based feedback alone.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3411764.3445507

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Repeated inspections across numerous screens impose significant cognitive load and time costs.

    “The quality and coverage of findings rely heavily on the evaluator's skill; judgments often vary, and repeated inspections across numerous screens impose significant cognitive load and time costs [52].”

    From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [52]

    The Introduction (p. 1) and Background (p. 2) describe design critique sessions as time- and resource-intensive, which supports the general point about time costs. They do not say that repeated inspections across numerous screens cause those costs or impose significant cognitive load. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

  • The quality and coverage of heuristic evaluation findings rely heavily on the evaluator's skill.

    “The quality and coverage of findings rely heavily on the evaluator's skill; judgments often vary, and repeated inspections across numerous screens impose significant cognitive load and time costs [52].”

    From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [52]

    The Conclusion (p. 11) says designers with varying expertise provide and interpret feedback differently, but it does not address the quality or coverage of heuristic evaluation findings. Searches for heuristic evaluation, evaluator skill, and inspection costs found no passage establishing the attributed claim. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

  • Heuristic evaluations remain human-intensive and expertise-dependent.

    “Despite their advantages, heuristic evaluations remain humanintensive and expertise-dependent.”

    From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [52]

    The Participants section (p. 7) describes using eight design experts to judge flyer improvements, but does not identify that work as a heuristic evaluation. Searches for heuristic evaluations and their labor or expertise requirements found no passage addressing the claim. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

  • Judgments in heuristic evaluation often vary.

    “The quality and coverage of findings rely heavily on the evaluator's skill; judgments often vary, and repeated inspections across numerous screens impose significant cognitive load and time costs [52].”

    From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [52]

    The Conclusion and Future Work (p. 11) says designers with varying expertise make sense of and provide feedback differently, but it does not address variation in heuristic-evaluation judgments. Searches for heuristic evaluation and evaluator disagreement found no passage supporting that specific claim; the Background (p. 2) discusses the time cost of design critiques, not repeated inspections across screens. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

  • Prior work has extensively documented what usability issues experts identify, but far less is known about how experts reason through these issues across stages of perception, interpretation, and projection.

    “While prior work has extensively documented what usability issues experts identify [35, 52], far less is known about how experts reason through these issues across stages of perception, interpretation, and projection [43, 76].”

    From Criticmate: Stagewise Human–AI Co-Critique in Single-Screen UI Evaluation (2026) · cites this paper as [52]

    The citation is attached to the claim about how experts reason through usability issues. The Conclusion and Future Work (p. 11) discusses differences in how designers with varying expertise make sense of feedback, but our copy does not address expert reasoning through usability issues or the stages of perception, interpretation, and projection. Searches for those stages and for expert reasoning about usability issues found no supporting passage. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. [1][1] Roy F Baumeister, Ellen Bratslavsky, Catrin Finkenauer, and Kathleen D Vohs. 2001. Bad is stronger than good. Review of general psychology 5, 4 (2001), 323-370. -
  2. [3][3] Inneke Berghmans, Nathalie Druine, Filip Dochy, and Katrien Struyven. 2012. A facilitative versus directive approach in training clinical skills? Investigating students' clinical performance and perceptions. Perspectives on medical education 1, 3 (2012), 104-118. -
  3. [4][4] Caroline Brandt. 2008. Integrating feedback and reflection in teacher preparation. ELT journal 62, 1 (2008), 37-46. -
  4. [5][5] Paul Cairns, Pratyush Pandab, and Christopher Power. 2014. The influence of emotion on number entry errors. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14) . ACM, New York, NY, USA, 2293-2296. -
  5. [6][6] Douglas Carnine, Candy Stevens, Jean Clements, and Edward J Kameenui. 1982. Effects of Facultative Questions and Practice on Intermediate Students' Understanding of Character Motives. Journal of Reading Behavior 14, 2 (1982), 179-190. -
  6. [7][7] Kristine L Chambers and Joan N Vickers. 2006. Effects of bandwidth feedback and questioning on the performance of competitive swimmers. The Sport Psychologist 20, 2 (2006), 184-197. -
  7. [8][8] Angelo V Ciardiello. 1998. Did you ask a good question today? Alternative cognitive and metacognitive strategies. Journal of Adolescent & Adult Literacy 42, 3 (1998), 210-219. -
  8. [9][9] Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences (3 ed.). Lawrence Erlbaum Associates, Hillsdale, NJ, USA. -
  9. [10][10] Adam Connor and Aaron Irizarry. 2015. Discussing Design: Improving Communication and Collaboration Through Critique . O'Reilly, Sebastopol, CA, USA. -
  10. [12][12] Patrick A Crain and Brian P Bailey. 2017. Share Once or Share Often?: Exploring How Designers Approach Iteration in a Large Online Community. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (C&C '17) . ACM, New York, NY, USA, 80-92. -
  11. [13][13] Deanna Dannels, Amy Housley Gaffney, and Kelly Norris Martin. 2008. Beyond Content, Deeper than Delivery: What Critique Feedback Reveals about Communication Expectations in Design Education. International Journal for the Scholarship of teaching and Learning 2, 2 (2008), n2. -
  12. [14][14] Deanna P Dannels and Kelly Norris Martin. 2008. Critiquing critiques: A genre analysis of feedback across novice to expert design studios. Journal of Business and Technical Communication 22, 2 (2008), 135-159. -
  13. [15][15] Steven Dow, Julie Fortuna, Dan Schwartz, Beth Altringer, Daniel Schwartz, and Scott Klemmer. 2011. Prototyping Dynamics: Sharing Multiple Designs Improves Exploration, Group Rapport, and Results. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11) . ACM, New York, NY, USA, 2807-2816. -
  14. [16][16] Eureka Foong, Darren Gergle, and Elizabeth M Gerber. 2017. Novice and Expert Sensemaking of Crowdsourced Design Feedback. Proceedings of the ACM on Human-Computer Interaction 1, CSCW (2017), 1-18. -
  15. [18][18] Michael D Greenberg, Matthew W Easterday, and Elizabeth M Gerber. 2015. Critiki: A scaffolded approach to gathering design feedback from paid crowdworkers. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition (C&C '15) . ACM, New York, NY, USA, 235-244. -
  16. [21][21] Catherine M Hicks, Vineet Pandey, C Ailie Fraser, and Scott Klemmer. 2016. Framing feedback: Choosing review environment features that support high quality peer assessment. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16) . ACM, New York, NY, USA, 458-469. -
  17. [22][22] Clayton J Hutto and Eric Gilbert. 2014. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. In Eighth International AAAI Conference on Weblogs and Social Media (ICWSM '14) . AAAI, Menlo Park, CA, USA. -
  18. [23][23] Hyeonsu B Kang, Gabriel Amoako, Neil Sengupta, and Steven P Dow. 2018. Paragon: An Online Gallery for Enhancing Design Feedback with Visual Examples. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18) . ACM, New York, NY, USA, 1-13. -
  19. [24][24] Alison King. 1990. Enhancing peer interaction and learning in the classroom through reciprocal questioning. American Educational Research Journal 27, 4 (1990), 664-687. -
  20. [25][25] Alison King. 1992. Facilitating elaborative learning through guided studentgenerated questioning. Educational psychologist 27, 1 (1992), 111-126. -
  21. [26][26] Alison King. 1995. Inquiring minds really do want to know: Using questioning to teach critical thinking. Teaching of Psychology 22, 1 (1995), 13-17. -
  22. [28][28] Markus Krause, Tom Garncarz, JiaoJiao Song, Elizabeth M Gerber, Brian P Bailey, and Steven P Dow. 2017. Critique Style Guide: Improving Crowdsourced Design Feedback with a Natural Language Model. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17) . ACM, New York, NY, USA, 4627-4639. -
  23. [29][29] Kurt Luther, Amy Pavel, Wei Wu, Jari-lee Tolentino, Maneesh Agrawala, Björn Hartmann, and Steven P Dow. 2014. CrowdCrit: crowdsourcing and aggregating visual design critique. In Proceedings of the companion publication of the 17th ACM conference on Computer supported cooperative work & social computing (CSCW '14) . ACM, New York, NY, USA, 21-24. -
  24. [30][30] Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P Dow. 2015. Structuring, aggregating, and evaluating crowdsourced design critique. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15) . ACM, New York, NY, USA, 473-485. -
  25. [31][31] Jennifer Marlow and Laura Dabbish. 2014. From rookie to all-star: professional development in a graphic design social networking site. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (CSCW '14) . ACM, New York, NY, USA, 922-933. -
  26. [32][32] Tricia J Ngoon, C Ailie Fraser, Ariel S Weingarten, Mira Dontcheva, and Scott Klemmer. 2018. Interactive Guidance Techniques for Improving Creative Feedback. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18) . ACM, New York, NY, USA, 55. -
  27. [33][33] Thi Thao Duyen T Nguyen, Thomas Garncarz, Felicia Ng, Laura A Dabbish, and Steven P Dow. 2017. Fruitful Feedback: Positive affective language and source anonymity improve critique reception and work outcomes. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17) . ACM, New York, NY, USA, 1024-1034. -
  28. [35][35] David A Robb, Stefano Padilla, Britta Kalkreuter, and Mike J Chantler. 2015. Crowdsourced feedback with imagery rather than text: Would designers use it?. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15) . ACM, New York, NY, USA, 1355-1364. -
  29. [36][36] D Royce Sadler. 1989. Formative assessment and the design of instructional systems. Instructional Science 18, 2 (1989), 119-144. -
  30. [37][37] Joan Sargeant, Karen Mann, Douglas Sinclair, Cees Van der Vleuten, and Job Metsemakers. 2008. Understanding the influence of emotions and reflection upon multi-source feedback acceptance and use. Advances in Health Sciences Education 13, 3 (2008), 275-288. -
  31. [38][38] Joan M Sargeant, Karen V Mann, Cees P Van der Vleuten, and Job F Metsemakers. 2009. Reflection: a link between receiving and using assessment feedback. Advances in Health Sciences Education 14, 3 (2009), 399-410. -
  32. [39][39] Donald A Schön. 1984. The Reflective Practitioner: How professionals think in action . Vol. 5126. Basic Books, New York, NY, USA. -
  33. [41][41] Pao Siangliulue, Kenneth C Arnold, Krzysztof Z Gajos, and Steven P Dow. 2015. Toward collaborative ideation at scale: Leveraging ideas from others to generate more creative and diverse ideas. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15) . ACM, New York, NY, USA, 937-945. -
  34. [42][42] Pao Siangliulue, Joel Chan, Krzysztof Z Gajos, and Steven P Dow. 2015. Providing timely examples improves the quantity and quality of generated ideas. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition (C&C '15) . ACM, ACM, New York, NY, USA, 83-92. -
  35. [43][43] Toyin Tofade, Jamie Elsner, and Stuart T Haines. 2013. Best practice strategies for effective use of questions as a teaching tool. American journal of pharmaceutical education 77, 7 (2013), 155. -
  36. [44][44] Maryam Tohidi, William Buxton, Ronald Baecker, and Abigail Sellen. 2006. Getting the right design and the design right. In Proceedings of the SIGCHI conference on Human Factors in computing systems (CHI '06) . ACM, New York, NY, USA, 1243-1252. -
  37. [45][45] Belkis Uluoğlu. 2000. Design knowledge communicated in studio critiques. Design Studies 21, 1 (2000), 33-58. -
  38. [46][46] Wesley Willett, Jeffrey Heer, and Maneesh Agrawala. 2012. Strategies for crowdsourcing social data analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12) . ACM, New York, NY, USA, 227-236. -
  39. [47][47] Jacob O Wobbrock, Leah Findlater, Darren Gergle, and James J Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI conference on human factors in computing systems (CHI '11) . ACM, New York, NY, USA, 143-146. -
  40. [48][48] Y Wayne Wu and Brian P Bailey. 2017. Bitter Sweet or Sweet Bitter? How Valence Order and Source Identity Influence Feedback Acceptance. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (C&C '17) . ACM, New York, NY, USA, 137-147. -
  41. [49][49] Y Wayne Wu and Brian P Bailey. 2018. Soften the Pain, Increase the Gain: Enhancing Users' Resilience to Negative Valence Feedback. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1-20. -
  42. [50][50] Anbang Xu and Brian Bailey. 2012. What do you think? A case study of benefit, expectation, and interaction in a large online critique community. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (CHI '12) . ACM, New York, NY, USA, 295-304. -
  43. [51][51] Anbang Xu, Shih-Wen Huang, and Brian Bailey. 2014. Voyant: generating structured feedback on visual designs using a crowd of non-experts. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing (CSCW '14) . ACM, New York, NY, USA, 1433-1444. -
  44. [52][52] Anbang Xu, Huaming Rao, Steven P Dow, and Brian P Bailey. 2015. A classroom study of using crowd feedback in the iterative design process. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing (CSCW '15) . ACM, New York, NY, USA, 1637-1648. -
  45. [53][53] Yu-Chun Yen, Steven P Dow, Elizabeth Gerber, and Brian P Bailey. 2016. Social network, web forum, or task market? Comparing different crowd genres for design feedback exchange. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems (DIS '16) . ACM, New York, NY, USA, 773-784. -
  46. [54][54] Yu-Chun Grace Yen, Steven P Dow, Elizabeth Gerber, and Brian P Bailey. 2017. Listen to Others, Listen to Yourself: Combining Feedback Review and Reflection to Improve Iterative Design. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition (C&C '17) . ACM, New York, NY, USA, 158-170. -
  47. [55][55] Alvin Yuan, Kurt Luther, Markus Krause, Sophie Isabel Vennix, Steven P Dow, and Bjorn Hartmann. 2016. Almost an expert: The effects of rubrics and expertise on perceived value of crowdsourced design critiques. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16) . ACM, New York, NY, USA, 1005-1017.

Cited by

Sorted by citations

Authors
Lekschas, F., Ampanavos, S., Siangliulue, P., Pfister, H., & Gajos, K. Z.
Affiliations
  • Harvard University (US)
  • B12, United States
ORCID
Venue
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2021-05-06
Pages
1-12
DOI
10.1145/3411764.3445507
OpenAlex
W3121450925
Cited by
14
References
2
Citations by year
2026: 3 · 2025: 1 · 2024: 6 · 2023: 3 · 2022: 1
FWCI
1.87
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
design changes, graphic design, perceived usefulness, feedback acceptance, open-ended questions

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review