Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 95 of the 95 references it cites were found in a registry or an index; 17 of the 38 sources its claims cite were checked for support.

Decoupling of Usefulness and Novelty: Evaluating the Impact of Generative AI on Design Outputs and Novice Designers' Creative Thinking

Open access
Fu, Yue, Zhou, Tony, Han, Bin, Wang, Marx, Chen, Yixin, Gomes Da Costa Lai, Zelia, Pang, Rock Yuren, Reinecke, Katharina, Wobbrock, Jacob O., Hiniker, Alexis · 2026 · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems

Abstract

People are increasingly leveraging generative AI (GenAI) for design tasks, making it critical to understand GenAI’s impact on design outcomes and users’ creative capabilities. We conducted a within-subjects experiment where 36 participants designed advertisements both with and without GenAI. Evaluations from clients and online volunteers revealed that GenAI-supported designs were perceived as significantly more creative and unconventional. Additionally, online volunteers, but not clients, rated these designs as more visually appealing. However, neither group perceived differences in usefulness, and clients noted no improvement in brand alignment, highlighting a notable decoupling of novelty and usefulness (two established components of creativity) in GenAI-supported design outputs. Although short-term GenAI use did not broadly influence participants’ creative thinking or experience, subgroup analyses indicated increases in divergent thinking among participants new to GenAI relative to participants with GenAI experience. We discuss the implications of the decoupling effect and GenAI’s influence on humans’ creativity.

Overview

CoChat Summary

TL;DR

In a within-subjects lab experiment, 36 participants designed social media advertisements with and without generative AI support, and their outputs were rated by five client stakeholders and 155 blinded online volunteers. GenAI-supported designs were judged significantly more creative and unconventional, but not more useful, revealing a decoupling of novelty and usefulness in GenAI-supported design outputs.

Key findings

  • Client evaluators rated GenAI-supported designs as significantly more creative (β = 0.805, SE = 0.183, Z = 4.405, p < .001) and more unconventional (β = 1.300, SE = 0.193, Z = 6.721, p < .001) than human-only designs, but found no significant differences in perceived usefulness (β = 0.019, SE = 0.163, Z = 0.119, p = .905), visual appeal (β = 0.264, p = .153), or brand alignment (β = −0.188, p = .359).
  • Online volunteers likewise rated GenAI-supported designs as more creative (β = 0.900, SE = 0.149, Z = 6.040, p < .001), more unconventional (β = 0.658, SE = 0.106, Z = 6.218, p < .001), and more visually appealing (β = 0.447, SE = 0.138, Z = 3.245, p < .01), with no usefulness difference (β = 0.001, SE = 0.112, Z = 0.006, p = .995).
  • No overall condition differences appeared in mood (relaxation, pleasure, energy), self-efficacy, divergent thinking (AUT), or convergent thinking (Anagram). In subgroup analysis, participants without prior GenAI design tool experience showed higher fluency (M = 14.333, SD = 4.384 vs. M = 12.500, SD = 4.573, Z = −2.112, p < .05) and flexibility (M = 12.250, SD = 3.562 vs. M = 10.167, SD = 3.131, Z = −2.203, p < .05) after the GenAI session; those with prior experience showed no significant differences.
  • Of 105 eligible designs (52 human-only, 53 GenAI-supported), participants selected their own GenAI-supported design as "best" in 17 of 36 cases (47.2%). Client qualitative comments praised GenAI designs' bright colors and diversity but criticized them as "too busy," illegible, or socially inappropriate/uncanny.

Methods

  • Study One: within-subjects lab experiment with 36 participants (94% under 30; 81% women; 67% with at least some prior GenAI design experience), each completing two counterbalanced 25-minute sessions — one using ChatGPT (DALL-E 3) and one using their own preferred tools — producing 110 designs (105 eligible). Measures: Alternative Uses Test (divergent thinking), Anagram Test (convergent thinking), and self-reported mood and self-efficacy; analyzed with Wilcoxon signed-rank tests.
  • Study Two: five "client" evaluators (members of the target research group) rated all 105 designs (N = 525 ratings) on usefulness, creativity, unconventionality, brand alignment, and visual appeal, plus optional free-response comments; 155 online volunteers (LabintheWild, aged 9–64, M = 25.6) each rated 10 randomly presented designs (N = 1550 ratings), blinded to condition.
  • Analysis: Generalized Linear Mixed Models with fixed effect of GenAI support and random intercepts for raters and for designs nested within participants; thematic analysis of client qualitative comments.

Limitations

  • Stated by the authors: reliance on subjective rater evaluations rather than objective metrics such as click-through rates; use of only GPT-4/DALL-E 3 rather than design-specific tools, limiting generalizability; the artificial controlled lab setting; qualitative analysis drawn from optional free-response text from only five clients; and a sample of mostly novice/early-career designers from a college community and alumni networks, limiting generalizability to experienced industry designers.
  • Evident in the text: the intervention was a single 25-minute session, so findings speak only to short-term effects; the client rater pool was very small (n = 5), and the study notes that knowing a design is AI-generated might itself influence evaluator ratings, which was not tested.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3772318.3791329

Cites

95 of 95 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

10 Valid · 1 Not valid · 10 Not checked · 76 Own results and interpretations

51 more claims

Create a free account to see all 97 claims and to verify the 6 references still open.

See all claims — free account
  • Probably not a substantial claim. 25
  • It may state two separate claims as one. 25

Create a free account to see these claims.

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Consistentz = -2.112reported p < .05 · recomputed p = .035

    “Z = −2.112, p < .05”

  • Consistentz = -2.203reported p < .05 · recomputed p = .028

    “Z = −2.203, p < .05”

  • Consistentz = 4.405reported p < .001 · recomputed p < .001

    “Z = 4.405, p < .001”

  • Consistentz = 6.721reported p < .001 · recomputed p < .001

    “Z = 6.721, p < .001”

  • Consistentz = 0.119reported p = .905 · recomputed p = .905

    “Z = 0.119, p = .905”

5 of 11 · the rest aren’t loaded

6 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Citations

Cites

Show all 80 references

Research works printed but not linked: 6

  1. Teresa M Amabile. 1996. Creativity and innovation in organizations. Vol. 5. Harvard Business School Boston.
  2. Oxford Analytica. 2023. GenAI will transform workplace tasks across industries. Emerald Expert Briefingsoxan-db (2023).
  3. Nicholas M Davis, Yanna Popova, Ivan Sysoev, Chih-Pin Hsiao, Dingtian Zhang, and Brian Magerko. 2014. Building Artistic Computer Colleagues with an Enactive Model of Creativity.. In ICCC. 38–45.
  4. Daniel Kahneman. 2011. Thinking, fast and slow. macmillan.
  5. Jie Li, Hancheng Cao, Laura Lin, Youyang Hou, Ruihao Zhu, and Abdallah El Ali. 2024. User experience design professionals’ perceptions of generative artificial intelligence. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1–18.
  6. Robert J Sternberg, Jacques Lautrey, and Todd I Lubart. 2003. Where are we in the Field of Intelligence, How did we get here, and Where are we Going?American Psychological Association.

Cited by

See all 2

Metadata

Authors
Fu, Y., Zhou, T., Han, B., Wang, M., Chen, Y., Gomes Da Costa Lai, Z., Pang, R. Y., Reinecke, K., Wobbrock, J. O., & Hiniker, A.
Affiliations
  • University of Washington (US)
  • Human Centered Design and Engineering, University of Washington, Seattle, Washington, USA
Venue
Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2026-04-13
Pages
1-17
OpenAlex
W7154165194
Cited by
2
References
80
Citations by year
2026: 2
FWCI
54.12
Type
conference paper
Language
en
Open access
Locations
CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
Topics
  • Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
Keywords
generative AI, novice designers, creative thinking, AI-assisted design, design education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Decoupling of Usefulness and Novelty: Evaluating the Impact of Generative AI on Design Outputs and Novice Designers' Creative Thinking. https://app.cochat.ai/papers/10.1145/3772318.3791329
The paper
Fu, Y., Zhou, T., Han, B., Wang, M., Chen, Y., Gomes Da Costa Lai, Z., Pang, R. Y., Reinecke, K., Wobbrock, J. O., & Hiniker, A. (2026). Decoupling of Usefulness and Novelty: Evaluating the Impact of Generative AI on Design Outputs and Novice Designers' Creative Thinking. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3772318.3791329

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.