Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Expert feedback is valuable but hard to obtain for many designers. Online crowds can provide fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted an experiment with a 2x2 factorial design. Student designers received feedback on a visual design from both experts and novices, who produced feedback using either an expert rubric or no rubric. We found that rubrics helped novice workers provide feedback that was rated nearly as valuable as expert feedback. A follow-up analysis on writing style showed that student designers found feedback most helpful when it was emotionally positive and specific, and that a rubric increased the occurrence of these characteristics in feedback. The analysis also found that expertise correlated with longer critiques, but not the other favorable characteristics. An informal evaluation indicates that experts may instead have produced value by providing clearer justifications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · cited 1×
“Feedback in particular helps the recipient develop a better understanding of the goals or qualities of standard, how the recipient is progressing towards those goals, and what can be done to progress even more [20].”
BackgroundThe role of deliberate practice in the acquisition of expert performance
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993) · Psychological Review · doi:10.1037/0033-295x.100.3.363 · Cited by 9,564 · cited 1×
“These systems often strive to match the quality of work produced by experts, who have mastered domain knowledge and performance standards from years of deliberate practice [12].”
BackgroundFormative assessment and the design of instructional systems
Sadler, D. R. (1989) · Instructional Science · doi:10.1007/bf00117714 · Cited by 4,285 · cited 4×
“Developing almost any skill generally requires both practice and feedback [35].”
BackgroundInfluentialSoylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686 · cited 2×
“[1]) or provide rubrics to workers (e.”
BackgroundSelf-citationHow bodies matter: five themes for interaction design
Klemmer, S. R., Hartmann, B., & Takayama, L. (2006) · Proceedings of the 6th conference on Designing Interactive systems · doi:10.1145/1142405.1142429 · Cited by 645 · cited 1×
“It helps the designer understand design principles [14], recognize how others perceive their work [24], and explore and compare alternatives [7, 41].”
BackgroundSelf-citationPeer and self assessment in massive online classes
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2505057 · Cited by 390
Shepherding the crowd yields better work
Dow, S., Kulkarni, A., Klemmer, S., & Hartmann, B. (2012) · Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work · doi:10.1145/2145204.2145355 · Cited by 334 · cited 1×
“Feedback through self-assessment has also been explored for writing consumer reviews, achieving comparable results to external sources of feedback [9].”
ResultSelf-citationGetting the right design and the design right
Tohidi, M., Buxton, W., Baecker, R., & Sellen, A. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124960 · Cited by 211 · cited 1×
“It helps the designer understand design principles [14], recognize how others perceive their work [24], and explore and compare alternatives [7, 41].”
BackgroundThe debate on automated essay grading
Hearst, M. A. (2000) · IEEE Intelligent Systems and their Applications · doi:10.1109/5254.889104 · Cited by 176
Prototyping dynamics: sharing multiple designs improves exploration, group rapport, and results
Dow, S., Fortuna, J., Schwartz, D., Altringer, B., Schwartz, D., & Klemmer, S. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979359 · Cited by 138
Self-citation
Cited by
Examining Crowd Work and Gig Work Through The Historical Lens of Piecework
Alkhatib, A., Bernstein, M. S., & Levi, M. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025974 · Cited by 148
“This body of research has broadly worked toward the answer to one central question: What does the future hold for on–demand work and those who do it? Researchers have offered insights on this question along three major threads: First, what are the complexity limits of on–demand work — specifically, how complex are the…”
BackgroundMetaWriter: Exploring the Potential and Perils of AI Writing Support in Scientific Peer Review
Sun, L., Tao, S., Hu, J., & Dow, S. P. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637371 · Cited by 37
Self-citationCritique Me: Exploring How Creators Publicly Request Feedback in an Online Critique Community
Cheng, R., Zeng, Z., Liu, M., & Dow, S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415232 · Cited by 33
“These two metrics are commonly used to evaluate design feedback in prior studies [10, 37, 47, 51, 62], and also build on our interview findings in Study 1.”
BackgroundResultSelf-citationAsk Me or Tell Me? Enhancing the Effectiveness of Crowdsourced Design Feedback
Lekschas, F., Ampanavos, S., Siangliulue, P., Pfister, H., & Gajos, K. Z. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445507 · Cited by 14
“when elicited and structured effectively [55].”
BackgroundInfluential
Metadata
- Authors
- Yuan, A., Luther, K., Krause, M., Vennix, S. I., Dow, S. P., & Hartmann, B.
- Venue
- Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing
- Publisher
- ACM
- Published
- 2016-02-27
- Pages
- 1005-1017
- OpenAlex
- W2292054779
- Cited by
- 97
- References
- 10
- FWCI
- 8.44
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques. https://app.cochat.ai/papers/10.1145/2818048.2819953
- The paper
- Yuan, A., Luther, K., Krause, M., Vennix, S. I., Dow, S. P., & Hartmann, B. (2016). Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques. Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. https://doi.org/10.1145/2818048.2819953
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.