Cited by 192Creativity in Education and NeuroscienceDesign Education and PracticeInnovative Human-Technology Interaction

The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index; none of the 40 sources its claims cite is checked for support yet.

Homogenization Effects of Large Language Models on Human Creative Ideation

Open access
Anderson, Barrett R, Shah, Jash Hemant, Kreminski, Max·2024·Creativity and Cognition

Abstract

Large language models (LLMs) are now being used in a wide variety of contexts, including as creativity support tools (CSTs) intended to help their users come up with new ideas. But do LLMs actually support user creativity? We hypothesized that the use of an LLM as a CST might make the LLM’s users feel more creative, and even broaden the range of ideas suggested by each individual user, but also homogenize the ideas suggested by different users. We conducted a 36-participant comparative user study and found, in accordance with the homogenization hypothesis, that different users tended to produce less semantically distinct ideas with ChatGPT than with an alternative CST. Additionally, ChatGPT users generated a greater number of more detailed ideas, but felt less responsible for the ideas they generated. We discuss potential implications of these findings for users, designers, and developers of LLM-based CSTs.

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

A 36-participant within-subjects study compared ChatGPT and the Oblique Strategies deck for divergent ideation, finding that ChatGPT users produced more ideas but more semantically homogeneous ideas at the group level, not at the individual level, and felt less responsible for their ideas. ChatGPT also increased fluency, flexibility, and elaboration, but did not increase unique ideas.

Key findings

  • Group-level homogenization: ideas generated with ChatGPT were less divergent from the average embedding of all ideas for that task (M = .24, SD = .07) than ideas generated with Oblique Strategies (M = .28, SD = .08), t(32) = 2.154, p = 0.038, d = .47, 95% CI [.00, .07].
  • Individual-level homogenization: no observable difference between ChatGPT (M = .65, SD = .07) and Oblique Strategies (M = .66, SD = .08), t(32) = .944, p = 0.352, d = .12, 95% CI [-.04, .01].
  • Responsibility: participants assigned less responsibility to themselves and more to the tool with ChatGPT (M = 48.17%, SD = 26.22%) than with Oblique Strategies (M = 63.63%, SD = 17.36%), t(32) = 3.21, p = 0.003, d = .67, 95% CI [-24.60%, -5.51%].
  • Other creativity facets: ChatGPT produced more ideas (M = 8.39, SD = 3.39 vs. M = 7.32, SD = 3.22; about 15% increase; p = 0.044), more idea categories (M = 8.58, SD = 2.90 vs. M = 6.77, SD = 3.69; about 27% more; p = 0.001), and greater stoplisted word count (M = 8.25, SD = 4.61 vs. M = 6.46, SD = 2.55; t(32) = 2.32, p = 0.237, d = .48), while unique ideas did not differ (M = .74, SD = .69 vs. M = .97, SD = 1.00; p = 0.108).

Methods

  • Within-subjects experiment: 36 participants recruited, 33 analyzed after excluding 3 for failure to follow directions; participants completed four divergent ideation tasks based on a subset of the Torrance Tests of Creative Thinking, half with ChatGPT (GPT-3.5 versions from May and August 2023) and half with the Oblique Strategies web app, producing 1271 ideas total, with 8 minutes per prompt and randomized/balanced order.
  • Measures: semantic similarity via sentence embeddings (all-MiniLM-L6-v2) for homogenization; Creativity Support Index and bespoke responsibility ratings; fluency by idea count, flexibility by human coding into 181 idea categories, elaboration by stoplisted word count; plus talk-aloud process observations and post-session interviews.
  • Analysis: paired comparisons reported with t(32), p-values, effect sizes, and 95% confidence intervals; semantic embedding model validated against human idea categorization, with all-MiniLM-L6-v2 averaging 58.92% agreement.

Limitations

  • Stated: the lab context may not reflect real-world LLM creativity support; participants had fixed, relatively short time per prompt; ideation prompts were supplied by experimenters rather than formulated by users.
  • Stated: the study used general-purpose ChatGPT rather than an LLM-based CST designed specifically for divergent ideation, so alternative deployments may avoid homogenization; prior evidence suggests feedback-tuned LLMs may homogenize where base LLMs do not.
  • Evident from the text: the paper did not evaluate idea quality, and its semantic similarity measure is imperfect—even the best embedding model achieved only 59% agreement with human coders on average, and cosine similarity between sentence embeddings may be influenced more by noun overlap than by all nuances of sentence meaning.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3635636.3656204

Cites

67 of 67 references verified0 citation errors · 0 not checkable

Statistics

18 tests recomputed · 1 with the reported and recomputed p on opposite sides of .05

Verification

Claims the paper makes

54 Own result · 31 Not checked yet · 31 Own interpretation · 8 Own method · 4 No basis given

Verify all 40 references — free account

Show all 83 claims

  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 38
  • It may state two separate claims as one. 14
45 of 45
  • The paper relies on this without support.

    The Oblique Strategies deck is a valid alternative creativity support tool for comparison with ChatGPT.

    “We directed participants to a web app version of the deck [76] as an alternative CST to ChatGPT in our control condition.”
    “We conducted a within-subjects experiment to evaluate the effects of using two different CSTs for idea generation: ChatGPT and the Oblique Strategies (OS) deck.”
  • The paper relies on this without support.

    ChatGPT 3.5 is representative of LLM-based creativity support tools for generalizing homogenization effects.

    “Participants in this study used the versions of ChatGPT 3.5 released on May 3rd 2023 (n=6, 16.6%) and on August 3rd, 2023 (30, 83.3%).”
    “Collectively, our results suggest that LLM-driven homogenization stems from the LLM providing different users with similar ideas”
  • The paper relies on this without support.

    The sample size of 33 participants is sufficient to detect the reported effects.

    “Our sample included 33 participants, ranging in age from 22 to 44 (M=28.36, SD=6.84)”
    “Of the 36 participants originally recruited, three were excluded from all analyses for failure to follow direction”
  • The paper relies on this without support.

    An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.

    “Many of our findings follow naturally from the fact that an LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.”
  • The paper relies on this without support.

    The Oblique Strategies deck consists of a relatively small set of fixed text strings.

    “The deck consists of a relatively small set of fixed text strings”
  • The paper relies on this without support.

    Aggregated word embeddings are generally outperformed on semantic similarity tasks by more recent transformer-based sentence embeddings, which can take sentence structure into account.

    “Aggregated word embeddings are generally outperformed on semantic similarity tasks by more recent transformer-based sentence embeddings, which (unlike aggregated word embeddings) are able to take sentence structure into account.”
  • The authors state this as an assumption.

    Human-constructed idea categories from the flexibility analysis are a valid ground truth for semantic similarity judgments.

    “Our experiment took the human-constructed idea categories produced by our flexibility analysis (Section 4.3.2) as a source of ground truth for semantic similarity judgments”
  • The paper relies on this without support.

    OpenAI's sentence embedding models are proprietary, cost money to use, cannot be run locally, and are not open source.

    “OpenAI models (unlike the SentenceTransformers models) are proprietary: they cost money to use, cannot be run locally, and are not open source.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

4 of 4

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upt(32) = 2.32reported p = .237 · recomputed p = .027The two fall on opposite sides of .05

    “t (32) = 2.32, p = .237”

  • Doesn’t add upt(32) = 2.154reported p = .038 · recomputed p = .039

    “t (32) = 2.154, p = .038”

  • Doesn’t add upr(32) = 0.43reported p = .008 · recomputed p = .011

    “r (32)=.43, p = .008”

  • Doesn’t add upr(32) = 0.46reported p = .004 · recomputed p = .006

    “r (32)=.46, p = .004”

  • Not confirmedt(35) = 1.028reported p = .312 · recomputed p = .311

    “t (35) = 1.028, p = .312”

  • Not confirmedt(32) = 0.091reported p = .927 · recomputed p = .928

    “t (32) = .091, p = .927”

  • Consistentt(32) = 0.944reported p = .352 · recomputed p = .352

    “t (32) = .944, p = .352”

  • Consistentt(32) = 3.21reported p = .003 · recomputed p = .003

    “t (32) = 3.21, p = .003”

  • Consistentt(32) = 2.1reported p = .044 · recomputed p = .044

    “t (32) = 2.10, p = .044”

  • Consistentt(32) = 3.5reported p = .001 · recomputed p = .001

    “t (32) = 3.50, p = .001”

  • Consistentt(32) = 1.65reported p = .108 · recomputed p = .109

    “t (32) = 1.65, p = .108”

  • Consistentt(32) = 0.134reported p = .894 · recomputed p = .894

    “t (32) = .134, p = .894”

  • Consistentt(32) = 1.424reported p = .164 · recomputed p = .164

    “t (32) = 1.424, p = .164”

  • Consistentr(32) = 0.23reported p = .185 · recomputed p = .191

    “r (32)=.23, p = .185”

  • Consistentt(32) = 1.016reported p = .317 · recomputed p = .317

    “t (32) = 1.016, p = .317”

  • Consistentt(32) = 0.276reported p = .784 · recomputed p = .784

    “t (32) = .276, p = .784”

  • Consistentt(32) = 0.222reported p = .826 · recomputed p = .826

    “t (32) = .222, p = .826”

  • Consistentt(32) = 0.208reported p = .836 · recomputed p = .837

    “t (32) = .208, p = .836”

18 of 18

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 50 references

Research works printed but not linked: 22

  1. Margaret A Boden. 2004. The Creative Mind: Myths and Mechanisms. Routledge.
  2. Rishi Bommasani Kathleen A Creel Ananya Kumar Dan Jurafsky and Percy S Liang. 2022. Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? In Advances in Neural Information Processing Systems Vol. 35. 3663–3678.
  3. Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel Ziegler Jeffrey Wu Clemens Winter Chris Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems Vol. 33. 1877–1901.
  4. Daniel Buschek, Lukas Mecke, Florian Lehmann, and Hai Dang. 2021. Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems. In Joint Proceedings of the ACM IUI 2021 Workshops.
  5. Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020. How Novelists Use Generative Language Models: An Exploratory User Study. In Workshop on Human-AI Co-Creation with Generative Models (HAI-GEN 2020).
  6. Kathy Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Sage.
  7. John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching stories with generative pretrained language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems.
  8. Anil R Doshi and Oliver Hauser. 2023. Generative artificial intelligence enhances creativity. Available at SSRN (2023).
  9. Ziv Epstein, Aaron Hertzmann, and Investigators of Human Creativity. 2023. Art and the science of generative AI. Science 380, 6650 (2023), 1110–1111.
  10. Agnes Guenther, Boris Eisenbart, and Andy Dong. 2017. Creativity as a way to innovate successfully. In DS 87-8 Proceedings of the 21st International Conference on Engineering Design (ICED 17) Vol 8: Human Behaviour in Design, Vancouver, Canada, 21-25.08. 2017. 389–398.
  11. Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in Psychology. Vol. 52. Elsevier, 139–183.
  12. Tom Hewett, Mary Czerwinski, Michael Terry, Jay Nunamaker, Linda Candy, Bill Kules, and Elisabeth Sylvan. 2005. Creativity support tool evaluation methods and metrics. In Creativity Support Tools: A workshop sponsored by the National Science Foundation. 10–24.
  13. David Kirsh. 2011. Creative cognition in choreography. In Proceedings of the 2nd International Conference on Computational Creativity.
  14. Max Kreminski and John Joon Young Chung. 2024. Intent Elicitation in Mixed-Initiative Co-Creativity. In Joint Proceedings of the ACM IUI Workshops.
  15. Max Kreminski, Isaac Karth, Michael Mateas, and Noah Wardrip-Fruin. 2022. Evaluating mixed-initiative creative interfaces via expressive range coverage analysis. In IUI Workshops. 34–45.
  16. Max Kreminski and Michael Mateas. 2021. Reflective creators. In International Conference on Computational Creativity. 309–318.
  17. Isabelle Levent and Lila Shroff. 2023. The Model is the Message. In The Second Workshop on Intelligent and Interactive Writing Assistants.
  18. Roberta M Milgram and Norman A Milgram. 1976. Creative thinking and creative performance in Israeli students.Journal of educational psychology 68, 3 (1976), 255.
  19. OpenAI. 2023. ChatGPT: A Large-Scale Conversational Language Model. https://www.openai.com/research/chatgpt. Accessed: August 3rd, 2023.
  20. Long Ouyang Jeffrey Wu Xu Jiang Diogo Almeida Carroll Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray John Schulman Jacob Hilton Fraser Kelton Luke Miller Maddie Simens Amanda Askell Peter Welinder Paul F Christiano Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems Vol. 35. 27730–27744.
  21. Melissa Roemmele. 2021. Inspiration through observation: Demonstrating the influence of automatically generated text on creative writing. In Proceedings of the 12th International Conference on Computational Creativity (ICCC ’21).
  22. E Paul Torrance. 1966. Torrance tests of creative thinking. Educational and Psychological Measurement (1966).

Cited by

Sorted by citations

Items 101–192 of 192

Metadata

Authors
Anderson, B. R., Shah, J. H., & Kreminski, M.
Affiliations
  • Independent Researcher, United States
  • Santa Clara University (US)
Venue
Creativity and Cognition
Publisher
ACM
Published
2024-06-23
Pages
413-425
OpenAlex
W4391556516
Cited by
188
References
50
Citations by year
2026: 97 · 2025: 79 · 2024: 12
FWCI
117.47
Type
conference paper
Language
en
Open access
Locations
  • Creativity and Cognition (publishedVersion, cc-by-nc-sa)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • Scholar Commons (Santa Clara University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
  • Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
large language models, creativity support tools, creative idea generation, ChatGPT, idea generation, AI-assisted creativity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review