Critique Style Guide: Improving Crowdsourced Design Feedback with a Natural Language Model
The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Designers are increasingly leveraging online crowds; yet, online contributors may lack the expertise, context, and sensitivity to provide effective critique. Rubrics help feedback providers but require domain experts to write them and may not generalize across design domains. This paper introduces and tests a novel semi-automated method to support feedback providers by analyzing feedback language. In our first study, 52 students from two design courses created design solutions and received feedback from 176 online providers. Instructors, students, and crowd contributors rated the helpfulness of each feedback response. From this data, an algorithm extracted a set of natural language features (e.g., specificity, sentiment etc.) that correlated with the ratings. The features accurately predicted the ratings and remained stable across different raters and design solutions. Based on these features, we produced a critique style guide with feedback examples - automatically selected for each feature - to help providers revise their feedback through self-assessment. In a second study, we tested the validity of the guide through a between-subjects experiment (n=50). Providers wrote feedback on design solutions with or without the guide. Providers generated feedback with higher perceived helpfulness when using our style-based guidance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Likert scales, levels of measurement, and the “laws” of statistics
Norman, G. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9222-y · Cited by 4,850 · cited 1×
“In accordance with Norman [36], we choose Pearson correlation.”
MethodFormative assessment and the design of instructional systems
Sadler, D. R. (1989) · Instructional Science · doi:10.1007/bf00117714 · Cited by 4,285 · cited 3×
“Sadler [40] argues that effective feedback is able to establish an understanding of the underlying patterns and concepts (conceptual), can communicate the difference of the work to this standard (specific), and lays out steps that reduces this gap (actionable).”
BackgroundSoylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686 · cited 4×
“Improving Crowdsourced Feedback Crowd-based methods can yield feedback that is of low quality or quantity [51] because contributors may lack the motivation [51], context [3], knowledge [33], or sensitivity [54] needed to provide high-quality design feedback.”
BackgroundInfluentialHow bodies matter: five themes for interaction design
Klemmer, S. R., Hartmann, B., & Takayama, L. (2006) · Proceedings of the 6th conference on Designing Interactive systems · doi:10.1145/1142405.1142429 · Cited by 645
Peer and self assessment in massive online classes
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2505057 · Cited by 390 · cited 3×
“Luther et al. compares differences between design iterations [33, 53], while others contrast critiques with feedback produced by experts [33, 31].”
BackgroundMethodResultShepherding the crowd yields better work
Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145355 · Cited by 334 · cited 2×
“Self-assessment can achieve results comparable to external sources of feedback [14], but may not always yield the same insights possible from a diverse external audience.”
BackgroundSelf-citationTuned Models of Peer Assessment in MOOCs
Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D. (2013) · arXiv · doi:10.48550/arxiv.1307.2579 · Cited by 262
Getting the right design and the design right
Tohidi, M., Buxton, W., Baecker, R., & Sellen, A. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124960 · Cited by 211 · cited 1×
“In design work, feedback helps designers iterate on their in-progress solutions [15], compare alternatives [12, 45], learn about the problem, and refine their process.”
BackgroundPeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance
Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015) · Proceedings of the Second (2015) ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724670 · Cited by 192
Cited by
Impact of AI assistance on student agency
Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024) · Computers & Education · doi:10.1016/j.compedu.2023.104967 · Cited by 548
When and How to Use AI in the Design Process? Implications for Human-AI Design Collaboration
Lee, S.-Y., Law, M., & Hoffman, G. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2353451 · Cited by 97
Critique Me: Exploring How Creators Publicly Request Feedback in an Online Critique Community
Cheng, R., Zeng, Z., Liu, M., & Dow, S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415232 · Cited by 33
“These two metrics are commonly used to evaluate design feedback in prior studies [10, 37, 47, 51, 62], and also build on our interview findings in Study 1.”
BackgroundMethodResultSelf-citationIncorporating Training, Self-monitoring and AI-Assistance to Improve Peer Feedback Quality
Darvishi, A., Khosravi, H., Abdi, S., Sadiq, S., & Gašević, D. (2022) · Proceedings of the Ninth ACM Conference on Learning @ Scale · doi:10.1145/3491140.3528265 · Cited by 31
Ask Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback
Lekschas, F., Ampanavos, S., Siangliulue, P., Pfister, H., & Gajos, K. Z. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445507 · Cited by 14
“[28] systematically investigated the perceived usefulness of feedback along various dimensions such as length, specifcity, or complexity.”
Background
Metadata
- Authors
- Krause, M., Garncarz, T., Song, J., Gerber, E. M., Bailey, B. P., & Dow, S. P.
- Venue
- Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2017-05-02
- Pages
- 4627-4639
- OpenAlex
- W2611242648
- Cited by
- 61
- References
- 14
- FWCI
- 26.84
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Critique Style Guide: Improving Crowdsourced Design Feedback with a Natural Language Model. https://app.cochat.ai/papers/10.1145/3025453.3025883
- The paper
- Krause, M., Garncarz, T., Song, J., Gerber, E. M., Bailey, B. P., & Dow, S. P. (2017). Critique Style Guide: Improving Crowdsourced Design Feedback with a Natural Language Model. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3025453.3025883
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.