CoChat
Sign Up
Cited by 192Creativity in Education and NeuroscienceDesign Education and PracticeInnovative Human-Technology Interaction

Homogenization Effects of Large Language Models on Human Creative Ideation

Open access
Anderson, Barrett R, Shah, Jash Hemant, Kreminski, Max·2024·Creativity and Cognition
DOI 10.1145/3635636.3656204
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) are now being used in a wide variety of contexts, including as creativity support tools (CSTs) intended to help their users come up with new ideas. But do LLMs actually support user creativity? We hypothesized that the use of an LLM as a CST might make the LLM’s users feel more creative, and even broaden the range of ideas suggested by each individual user, but also homogenize the ideas suggested by different users. We conducted a 36-participant comparative user study and found, in accordance with the homogenization hypothesis, that different users tended to produce less semantically distinct ideas with ChatGPT than with an alternative CST. Additionally, ChatGPT users generated a greater number of more detailed ideas, but felt less responsible for the ideas they generated. We discuss potential implications of these findings for users, designers, and developers of LLM-based CSTs.

AI snapshot

Written from the paper’s full text.

TL;DR

A 36-participant within-subjects study compared ChatGPT and the Oblique Strategies deck for divergent ideation, finding that ChatGPT users produced more ideas but more semantically homogeneous ideas at the group level, not at the individual level, and felt less responsible for their ideas. ChatGPT also increased fluency, flexibility, and elaboration, but did not increase unique ideas.

Key findings

  • Group-level homogenization: ideas generated with ChatGPT were less divergent from the average embedding of all ideas for that task (M = .24, SD = .07) than ideas generated with Oblique Strategies (M = .28, SD = .08), t(32) = 2.154, p = 0.038, d = .47, 95% CI [.00, .07].
  • Individual-level homogenization: no observable difference between ChatGPT (M = .65, SD = .07) and Oblique Strategies (M = .66, SD = .08), t(32) = .944, p = 0.352, d = .12, 95% CI [-.04, .01].
  • Responsibility: participants assigned less responsibility to themselves and more to the tool with ChatGPT (M = 48.17%, SD = 26.22%) than with Oblique Strategies (M = 63.63%, SD = 17.36%), t(32) = 3.21, p = 0.003, d = .67, 95% CI [-24.60%, -5.51%].
  • Other creativity facets: ChatGPT produced more ideas (M = 8.39, SD = 3.39 vs. M = 7.32, SD = 3.22; about 15% increase; p = 0.044), more idea categories (M = 8.58, SD = 2.90 vs. M = 6.77, SD = 3.69; about 27% more; p = 0.001), and greater stoplisted word count (M = 8.25, SD = 4.61 vs. M = 6.46, SD = 2.55; t(32) = 2.32, p = 0.237, d = .48), while unique ideas did not differ (M = .74, SD = .69 vs. M = .97, SD = 1.00; p = 0.108).

Methods

  • Within-subjects experiment: 36 participants recruited, 33 analyzed after excluding 3 for failure to follow directions; participants completed four divergent ideation tasks based on a subset of the Torrance Tests of Creative Thinking, half with ChatGPT (GPT-3.5 versions from May and August 2023) and half with the Oblique Strategies web app, producing 1271 ideas total, with 8 minutes per prompt and randomized/balanced order.
  • Measures: semantic similarity via sentence embeddings (all-MiniLM-L6-v2) for homogenization; Creativity Support Index and bespoke responsibility ratings; fluency by idea count, flexibility by human coding into 181 idea categories, elaboration by stoplisted word count; plus talk-aloud process observations and post-session interviews.
  • Analysis: paired comparisons reported with t(32), p-values, effect sizes, and 95% confidence intervals; semantic embedding model validated against human idea categorization, with all-MiniLM-L6-v2 averaging 58.92% agreement.

Limitations

  • Stated: the lab context may not reflect real-world LLM creativity support; participants had fixed, relatively short time per prompt; ideation prompts were supplied by experimenters rather than formulated by users.
  • Stated: the study used general-purpose ChatGPT rather than an LLM-based CST designed specifically for divergent ideation, so alternative deployments may avoid homogenization; prior evidence suggests feedback-tuned LLMs may homogenize where base LLMs do not.
  • Evident from the text: the paper did not evaluate idea quality, and its semantic similarity measure is imperfect—even the best embedding model achieved only 59% agreement with human coders on average, and cosine similarity between sentence embeddings may be influenced more by noun overlap than by all nuances of sentence meaning.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3635636.3656204

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

18 tests recomputed · 1 with the reported and recomputed p on opposite sides of .05

Claims the paper makes

0 of 40 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § ABSTRACT · p. 1

    In a 36-participant comparative user study, ideas produced with ChatGPT were less divergent from the average embedding of all ideas generated for that task (M = .24, SD = .07) than ideas produced with the Oblique Strategies deck (M = .28, SD = .08), t(32) = 2.154, p = 0.038, d = .47, 95% CI [.00, .07].

    “different users tended to produce less semantically distinct ideas with ChatGPT than with an alternative CST”
    • Also stated in: § 4.1 Homogenization · p. 5 · plain
      “When participants used ChatGPT, the ideas they produced were less divergent from the average embedding of all ideas generated for that task, ( M = .24, SD = .07), compared to the ideas that they produced when using OS, ( M = .28, SD = .08), t (32) = 2.154, p = 0.038, d = .47, 95% CI [.00,.07].”
    • Also stated in: § 4.1 Homogenization · p. 5 · plain
      “At the group level, ideas produced with the help of ChatGPT were more homogenized.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 1 INTRODUCTION · p. 2

    In the same study, ideas produced with ChatGPT were not observably more divergent from the average embedding of all of the other ideas that the same participant generated for the same ideation prompt (M = .65, SD = .07) than ideas produced with the Oblique Strategies deck (M = .66, SD = .08), t(32) = .944, p = 0.352, d = .12, 95% CI [-.04, .01].

    “With which CST do participants produce more semantically similar ideas at the individual level? (A: no difference )”
    • Also stated in: § 4.1 Homogenization · p. 5 · plain
      “When participants used ChatGPT, the ideas they produced were not observably more divergent from the average embedding of all of the other ideas that they themselves generated for the same ideation prompt, ( M = .65, SD = .07), compared to semantic dissimilarity for ideas that they produced when using OS, ( M = .66, SD = .08), t (32) = .944, p = 0.352, d = .12, 95% CI [-.04,.01].”
    • Also stated in: § 4.1 Homogenization · p. 5 · plain
      “At the individual level, we did not observe a difference in homogenization for ideas generated with the help of ChatGPT.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 1 INTRODUCTION · p. 2

    Participants generated ideas that hit about 27% more categories when using ChatGPT (M = 8.58, SD = 2.90) compared to the number of categories covered when using the Oblique Strategies deck (M = 6.77, SD = 3.69), t(32) = 3.50, p = 0.001, d = .54, 95% CI [.75, 2.86].

    “Beyond originality, do ChatGPT and non-ChatGPT users differ in terms of other facets of creativity, such as fluency, flexibility, and elaboration? (A: ChatGPT → higher fluency, flexibility, and elaboration )”
    • Also stated in: § 4.3 Other Facets of Creativity · p. 6 · plain
      “Participants generated ideas that hit about 27% more categories when using ChatGPT ( M = 8.58, SD =2.90), compared to the number of categories covered when using OS ( M = 6.77, SD = 3.69), t (32) = 3.50, p = 0.001, d = .54, 95% CI [.75,2.86].”
    Quote in the paperSubstantial claim
  • Findingown data
    § ABSTRACT · p. 1

    Participants generated about one additional idea-approximately a 15% increase-when using ChatGPT (M = 8.39, SD = 3.39) compared to the number of ideas they generated when using the Oblique Strategies deck (M = 7.32, SD = 3.22), t(32) = 2.10, p = 0.044, d = .32, 95% CI [.03, 2.11].

    “ChatGPT users generated a greater number of more detailed ideas”
    • Also stated in: § 1 INTRODUCTION · p. 2 · plain
      “Beyond originality, do ChatGPT and non-ChatGPT users differ in terms of other facets of creativity, such as fluency, flexibility, and elaboration? (A: ChatGPT → higher fluency, flexibility, and elaboration )”
    • Also stated in: § 4.3 Other Facets of Creativity · p. 5 · plain
      “Participants generated about one additional idea-approximately a 15% increase-when using ChatGPT ( M = 8.39, SD =3.39), compared to the number of ideas they generated when using OS ( M = 7.32, SD = 3.22), t (32) = 2.10, p = 0.044, d = .32, 95% CI [.03,2.11].”
    Quote in the paperSubstantial claim
  • Findingown data
    § ABSTRACT · p. 1

    Participants assigned less responsibility to themselves (and more to the tool) for ideas generated while using ChatGPT (M = 48.17%, SD = 26.22%) than for ideas generated while using the Oblique Strategies deck (M = 63.63%, SD = 17.36%), t(32) = 3.21, p = 0.003, d = .67, 95% CI [-24.60%, -5.51%].

    “felt less responsible for the ideas they generated”
    • Also stated in: § 1 INTRODUCTION · p. 2 · plain
      “Do ChatGPT users feel more or less responsible for the ideas they produce? (A: less responsible )”
    • Also stated in: § 4.2 Sense of Responsibility · p. 5 · plain
      “Participants assigned less responsibility to themselves (and more to the tool) for ideas generated while using ChatGPT ( M =48.17%, SD =26.22%), compared to ideas generated while using OS ( M =63.63%, SD =17.36%), t (32) = 3.21, p = 0.003, d = .67, 95% CI [-24.60%, -5.51%,].”
    Quote in the paperSubstantial claim
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 38
  • May combine two claims. 14

  • The quote only partly states this claim.

    Findingown data
    § ABSTRACT · p. 1

    ChatGPT users generated more detailed ideas (higher elaboration) than users of the Oblique Strategies deck.

    “ChatGPT users generated a greater number of more detailed ideas”
    • Also stated in: § 1 INTRODUCTION · p. 2 · plain
      “Beyond originality, do ChatGPT and non-ChatGPT users differ in terms of other facets of creativity, such as fluency, flexibility, and elaboration? (A: ChatGPT → higher fluency, flexibility, and elaboration )”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4.5 Creative Process · p. 7

    Other prompting strategies included asking for relevant scientific or factual information, asking for responses from a specific perspective, or adding creative constraints.

    “Other prompting strategies included asking for relevant scientific or factual information (e.g. 'What are the most popular dolls?', 'How common is face blindness?', etc.), asking for responses from a specific perspective (e.g. project manager, scientist, athlete, etc.), or adding creative constraints (e.g. make this toy for a dog, respond in the form of a story, etc.).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 4.4 Retrospective Reflections · p. 6

    Participants made self-deprecating remarks regarding their familiarity and technical ability with ChatGPT, but made no such remarks regarding OS.

    “Participants also made self-deprecating remarks regarding their familiarity and technical ability with ChatGPT (5.56%, n = 2), but made no such remarks regarding OS.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.5 Creative Process · p. 7

    Of the participants who copied directly from ChatGPT output, the majority copied selectively; only 1 copied the entirety of the output without any attempt at curation or editing.

    “The majority of these participants copied selectively: only 1 copied the entirety of the ChatGPT output without any attempt at curation or editing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.5 Creative Process · p. 7

    Most participants (72.22%, n=26) iterated on their prompt.

    “Most participants (72.22%, n = 26) also iterated on their prompt, either by adding more context to their initial prompt, probing with more specific questions or variations, or by asking ChatGPT to regenerate answers or provide additional responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.4 Retrospective Reflections · p. 6

    Participants reported that the LLM responses became too specific too quickly.

    “and that the LLM responses became too specific too quickly (5.56%, n = 2).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.4 Retrospective Reflections · p. 6

    The second most common theme in participant interviews was a positive sentiment regarding the speed and accuracy of the LLM-provided responses.

    “The second most common theme was a positive sentiment regarding the speed and accuracy of the LLM-provided responses (25.00%, n = 9).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 4.4 Retrospective Reflections · p. 6

    Participants reported that the LLM responses were too repetitive.

    “that the LLM responses were too repetitive (8.33%, n = 3)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § 7 CONCLUSION · p. 10

    ChatGPT users exhibit greater fluency, flexibility, and elaboration than users of an alternative CST.

    “Coupled with evidence that ChatGPT users exhibit greater fluency, flexibility, and elaboration than users of an alternative CST”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.2 Inferential Distance · p. 8

    Even disengaged users typically did not simply copy-paste ChatGPT output unexamined into their list of ideas; only one participant in the entire study directly copy-pasted the entirety of a ChatGPT response without any further curation or modification.

    “we only observed one participant in the entire study directly copy-paste the entirety of a ChatGPT response without any further curation or modification”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingown data
    § 5.1 Creative Fixation · p. 8

    Some participants viewed ChatGPT as a 'very fast and accurate' authoritative source that 'gave [...] the right answers'.

    “some participants in our study viewed ChatGPT as a 'very fast and accurate' authoritative source that 'gave [...] the right answers'”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 4.3 Other Facets of Creativity · p. 6

    Coding of ideas into categories was conducted while blind to the CST condition.

    “Coding was conducted while blind to the CST condition (i.e., coders did not know if an idea was generated with or without LLM support).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § 3.3 Procedure · p. 4

    The order of CSTs and prompts was randomized per participant and balanced across the entire experiment.

    “The order of CSTs and prompts was randomized per participant and balanced across the entire experiment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 2.1 Prior Studies of Homogenization · p. 2

    GPT-generated short stories are less diverse than human-written stories in structure but exhibit greater gender and sexual diversity in character description.

    “Begus [7] compares human-written to GPT-generated short stories, observing that GPTgenerated stories are less diverse than human-written stories in structure but exhibit greater gender and sexual diversity in character description.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] [7] Nina Begus. 2023. Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytelling. arXiv preprint arXiv:2310.12902 (2023). -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 4.4 Retrospective Reflections · p. 7

    The Collaboration subscale is irrelevant and often omitted in exclusively single-user contexts like that of the study.

    “We did not collect responses for the Collaboration subscale, which is irrelevant and often omitted in exclusively single-user contexts like that of our study (e.g., [5, 87]).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [5] [5] Ilhan Aslan, Katharina Weitz, Ruben Schlagowski, Simon Flutura, Susana Garcia Valesco, Marius Pfeil, and Elisabeth André. 2019. Creativity support and multimodal pen-based interaction. In 2019 International Conference on Multimodal Interaction . 135-144. - source

      Not checked in the free run.

    • cites [87] [87] Qian Wan and Zhicong Lu. 2023. GANCollage: A GAN-Driven Digital Mood Board to Facilitate Ideation in Creativity Support. In Proceedings of the 2023 ACM Designing Interactive Systems Conference . 136-146. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.1 Prior Studies of Homogenization · p. 2

    Human-written short stories substantially outperform LLM-generated stories on all dimensions of creativity measured by the TTCT, including originality.

    “Chakrabarty et al. [15] compare LLM-generated short stories to stories by expert human writers and find that the human-written stories substantially outperform the LLM-generated stories on all dimensions of creativity measured by the TTCT, including the originality dimension.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] [15] Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2023. Art or Artifice? Large Language Models and the False Promise of Creativity. arXiv preprint arXiv:2309.14556 (2023). -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.3 Evaluating Creativity and CSTs · p. 3

    The Torrance Tests of Creative Thinking (TTCT) evaluate creativity according to four facets of creative output: fluency, flexibility, originality, and elaboration.

    “The Torrance Tests of Creative Thinking (TTCT) [85] evaluate the creativity of test-takers according to four facets of creative output: fluency , or sheer quantity of artifacts created; flexibility , or quantity of distinct categories of artifacts created; originality , or dissimilarity of created artifacts to others' creations; and elaboration , or level of detail in created artifacts.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [85] [85] E Paul Torrance. 1966. Torrance tests of creative thinking. Educational and Psychological Measurement (1966). -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 2.1 Prior Studies of Homogenization · p. 2

    Padmakumar and He found a homogenization effect from LLM assistance at both the lexical and content levels, but only for the instruction-tuned LLM InstructGPT.

    “Padmakumar and He evaluate the baseline GPT-3 model [12] versus the instructionfinetuned variant InstructGPT [68] in the context of a short-form argumentative essay writing task and find a homogenization effect from LLM assistance at both the lexical and content levels, but only for the instruction-tuned LLM [69].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69] [69] Vishakh Padmakumar and He He. 2023. Does Writing with Language Models Reduce Content Diversity? arXiv preprint arXiv:2309.05196 (2023). -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 3.1 Participants · p. 3

    Three of the 36 originally recruited participants were excluded from all analyses for failure to follow direction.

    “Of the 36 participants originally recruited, three were excluded from all analyses for failure to follow direction (e.g., not using ChatGPT when prompted to do so).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 2.1 Prior Studies of Homogenization · p. 2

    Observation of LLM-generated text shapes the ideas expressed by the LLM's user even when the user does not incorporate LLM-generated text directly into their writing.

    “Roemmele [74] proposes and provides evidence for an 'inspiration through observation' model of how human writers are influenced by LLMs: observation of LLM-generated text shapes the ideas expressed by the LLM's user even when the user does not incorporate LLM-generated text directly into their writing.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [74] [74] Melissa Roemmele. 2021. Inspiration through observation: Demonstrating the influence of automatically generated text on creative writing. In Proceedings of the 12th International Conference on Computational Creativity (ICCC '21) . -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.1 Prior Studies of Homogenization · p. 2

    Predictive text encourages predictable writing in the context of single-word suggestions given by smartphone keyboards.

    “Arnold et al. found that 'predictive text encourages predictable writing' [4] in the context of singleword suggestions given by smartphone keyboards.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [4] [4] Kenneth C Arnold, Krysta Chauncey, and Krzysztof Z Gajos. 2020. Predictive text encourages predictable writing. In Proceedings of the 25th International Conference on Intelligent User Interfaces . 128-138. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3.1 Participants · p. 3

    The study protocol and recruitment materials were approved by the Santa Clara University IRB.

    “Our study protocol and recruitment materials were approved by the Santa Clara University IRB.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § 2.2 Reasons to Expect Homogenization · p. 3

    Groups tend to be less creative than individuals when majoritarian convergence processes are used, and minority dissent within group creative processes tends to lead to greater creativity.

    “groups tend to be less creative than individuals when majoritarian convergence processes are used, and minority dissent within group creative processes tends to lead to greater creativity [30]”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [30] [30] Umer Farooq, John M Carroll, and Craig H Canoe. 2008. Designing for creativity in computer-supported cooperative work. International Journal of e-Collaboration (IJeC) 4, 4 (2008), 51-75. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.3 Evaluating Creativity and CSTs · p. 3

    The Creativity Support Index (CSI) is a widely used and psychometrically validated survey instrument for assessing a CST's ability to assist a user engaged in creative work.

    “The Creativity Support Index (CSI) [17] is a widely used and psychometrically validated survey instrument that attempts to standardize some aspects of this experience reporting process across different CSTs.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] [17] Erin Cherry and Celine Latulipe. 2014. Quantifying the creativity support of digital tools through the Creativity Support Index. ACM Transactions on ComputerHuman Interaction (TOCHI) 21, 4 (2014). - source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation
    § 2.1 Prior Studies of Homogenization · p. 2

    LLM-supplied next-phrase suggestions may alter the form and content of a human user's writing even when the user dislikes these suggestions.

    “Bhat et al. [8] find that LLM-supplied next-phrase suggestions may alter the form and content of a human user's writing even when the user dislikes these suggestions.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [8] [8] Advait Bhat, Saaket Agashe, Parth Oberoi, Niharika Mohile, Ravi Jangir, and Anirudha Joshi. 2023. Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing. In Proceedings of the 28th International Conference on Intelligent User Interfaces . 436-452. - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 2.1 Prior Studies of Homogenization · p. 2

    LLM support tends to increase the diversity of a user's vocabulary but may reduce their feelings of ownership for the text they produce.

    “Lee et al. [54] find that LLM support tends to increase the diversity of a user's vocabulary but may reduce their feelings of ownership for the text they produce.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [54] [54] Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a humanAI collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems . - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 2.1 Prior Studies of Homogenization · p. 2

    An opinionated LLM-based CST influences the opinions expressed by its users in argumentative writing.

    “Jakesch et al. [41] find that an opinionated LLM-based CST influences the opinions expressed by its users in argumentative writing.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [41] [41] Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-writing with opinionated language models affects users' views. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . - source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 5 DISCUSSION · p. 8

    It may still be difficult for LLM users to adapt their prompting strategies to elicit more diverse responses from the LLM.

    “though it may still be difficult for them to adapt their prompting strategies to elicit more diverse responses from the LLM.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 5.5 Implications for LLM-Based CST Design · p. 9

    For an LLM to directly transform a brief expression of intent into a substantially larger artifact, it will necessarily have to make a large number of creative decisions on the user's behalf, and these decisions will be made by reference to the relative likelihood of different alternatives.

    “For an LLM to directly transform a brief expression of intent into a substantially larger artifact (e.g., a passage of text), it will necessarily have to make a large number of creative decisions (e.g., individual word choices) on the user's behalf, and these decisions will be made by reference to the relative likelihood of different alternatives.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning
    § 5.3 User Adaptation to Homogenization · p. 8

    Participants can see all of their own ideas, so they can actively modulate the diversity of the set of ideas that they have generated.

    “This may be because participants can see all of their own ideas, so they can actively modulate the diversity of the set of ideas that they have generated”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 5 DISCUSSION · p. 8

    LLM users might be able to resist the homogenization effect if they are given information about what kinds of stereotyped outputs the LLM tends to produce in a particular creative context.

    “Third, we infer that LLM users might be able to resist the homogenization effect if they are given information about what kinds of stereotyped outputs the LLM tends to produce in a particular creative context”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 6 LIMITATIONS · p. 10

    The study made use of a general-purpose LLM-based chatbot (ChatGPT) rather than an LLM-based CST specifically designed to support divergent ideation.

    “Because our study made use of a general-purpose LLM-based chatbot (ChatGPT) rather than an LLM-based CST specifically designed to support divergent ideation”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § 6 LIMITATIONS · p. 10

    Most organic divergent ideation scenarios are not subject to these constraints.

    “Most organic divergent ideation scenarios are not subject to these constraints.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 6 LIMITATIONS · p. 10

    The study was conducted in a lab context and consequently may not reflect some aspects of how people use LLMs for creativity support 'in the wild'.

    “Our study was conducted in a lab context and consequently may not reflect some aspects of how people use LLMs for creativity support 'in the wild'.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 6 LIMITATIONS · p. 10

    The ideation prompts in the study were supplied by the researchers, rather than being formulated independently by CST users.

    “additionally, the ideation prompts in our study were supplied by us, rather than being formulated independently by CST users.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 6 LIMITATIONS · p. 10

    Participants were only allowed a fixed and relatively short amount of time to respond to each ideation prompt.

    “In order to keep study sessions to a manageable length, participants were only allowed a fixed and relatively short amount of time to respond to each ideation prompt”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning
    § 5.5 Implications for LLM-Based CST Design · p. 9

    The LLM itself functions most directly as a means of refracting user-supplied inputs through the lens of statistical patterns learned from its training data.

    “the LLM itself functions most directly as a means of refracting user-supplied inputs through the lens of statistical patterns learned from its training data.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.

  • May combine two claims.

    Interpretationreasoning
    § 5.3 User Adaptation to Homogenization · p. 8

    Participants cannot easily gauge the similarity or dissimilarity of their own responses to other people's responses, so they cannot actively intervene to modulate diversity at the group level.

    “However, participants cannot easily gauge the similarity or dissimilarity of their own responses to other people's responses, so they cannot actively intervene to modulate diversity at the group level.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationcitation
    § 6 LIMITATIONS · p. 10

    Interventions at both the AI and UI layers may help to mitigate homogenization effects in future LLM-based CSTs.

    “These findings suggest that interventions at both the AI and UI layers may help to mitigate homogenization effects in future LLM-based CSTs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § A VALIDATING SENTENCE EMBEDDINGS FOR HOMOGENIZATION ANALYSIS · p. 13

    The proprietary nature of OpenAI's sentence embedding models limits the replicability of research results derived from them and therefore their suitability for research pipelines.

    “This limits the replicability of research results derived from OpenAI models and therefore the suitability of these models for research pipelines.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: OpenAI's sentence embedding models are proprietary, cost money to use, cannot be run locally, and are not open source.

  • May combine two claims.

    Interpretationown data
    § 5.1 Creative Fixation · p. 8

    Users viewing ChatGPT as authoritative may have led them to accept LLM-suggested ideas as valid, but it did not seem to constrain the process of ideation by displacing ideas that were incompatible with those suggested by ChatGPT.

    “Altogether, we conclude that users viewing ChatGPT as authoritative may have led them to accept LLM-suggested ideas as valid, but it did not seem to constrain the process of ideation by displacing ideas that were somehow incompatible with those suggested by ChatGPT.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 6 LIMITATIONS · p. 10

    Padmakumar and He have found evidence that feedback-tuned LLMs may induce creative homogenization where base (non-feedback-tuned) LLMs do not.

    “Padmakumar and He have found evidence that feedbacktuned LLMs may induce creative homogenization where base (nonfeedback-tuned) LLMs do not [69]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69] [69] Vishakh Padmakumar and He He. 2023. Does Writing with Language Models Reduce Content Diversity? arXiv preprint arXiv:2309.05196 (2023). -

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation
    § 6 LIMITATIONS · p. 10

    Alternative deployments of LLMs may be able to avoid homogenizing user ideas.

    “it should be noted that alternative deployments of LLMs may be able to avoid homogenizing user ideas.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning
    § A VALIDATING SENTENCE EMBEDDINGS FOR HOMOGENIZATION ANALYSIS · p. 12

    Transformer-based sentence embeddings have not yet been validated against human judgments of semantic similarity in the context of creativity research specifically.

    “However, the rapid pace of progress in machine learning research means that transformerbased sentence embeddings have not yet been validated against human judgments of semantic similarity in the context of creativity research specifically.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § A VALIDATING SENTENCE EMBEDDINGS FOR HOMOGENIZATION ANALYSIS · p. 13

    There exists some evidence that cosine similarity between sentence embeddings is more strongly influenced by overlap in the set of nouns than by other similarities.

    “There also exists some evidence that cosine similarity between sentence embeddings is more strongly influenced by overlap in the set of nouns than by other similarities [66]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [66] [66] Dmitry Nikolaev and Sebastian Padó. 2023. Representation biases in sentence transformers. arXiv preprint arXiv:2301.13039 (2023). -

      Not checked in the free run.

45 of 45

  • The paper relies on this without support.

    The Oblique Strategies deck is a valid alternative creativity support tool for comparison with ChatGPT.

    “We directed participants to a web app version of the deck [76] as an alternative CST to ChatGPT in our control condition.”
    “We conducted a within-subjects experiment to evaluate the effects of using two different CSTs for idea generation: ChatGPT and the Oblique Strategies (OS) deck.”

  • The paper relies on this without support.

    ChatGPT 3.5 is representative of LLM-based creativity support tools for generalizing homogenization effects.

    “Participants in this study used the versions of ChatGPT 3.5 released on May 3rd 2023 (n=6, 16.6%) and on August 3rd, 2023 (30, 83.3%).”
    “Collectively, our results suggest that LLM-driven homogenization stems from the LLM providing different users with similar ideas”

  • The paper relies on this without support.

    The sample size of 33 participants is sufficient to detect the reported effects.

    “Our sample included 33 participants, ranging in age from 22 to 44 (M=28.36, SD=6.84)”
    “Of the 36 participants originally recruited, three were excluded from all analyses for failure to follow direction”

  • The paper relies on this without support.

    An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.

    “Many of our findings follow naturally from the fact that an LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.”

  • The paper relies on this without support.

    The Oblique Strategies deck consists of a relatively small set of fixed text strings.

    “The deck consists of a relatively small set of fixed text strings”

  • The paper relies on this without support.

    Aggregated word embeddings are generally outperformed on semantic similarity tasks by more recent transformer-based sentence embeddings, which can take sentence structure into account.

    “Aggregated word embeddings are generally outperformed on semantic similarity tasks by more recent transformer-based sentence embeddings, which (unlike aggregated word embeddings) are able to take sentence structure into account.”

  • The authors state this as an assumption.

    Human-constructed idea categories from the flexibility analysis are a valid ground truth for semantic similarity judgments.

    “Our experiment took the human-constructed idea categories produced by our flexibility analysis (Section 4.3.2) as a source of ground truth for semantic similarity judgments”

  • The paper relies on this without support.

    OpenAI's sentence embedding models are proprietary, cost money to use, cannot be run locally, and are not open source.

    “OpenAI models (unlike the SentenceTransformers models) are proprietary: they cost money to use, cannot be run locally, and are not open source.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • State-of-the-art generative AI systems match human creativity in certain evaluations and can enhance the creative output of knowledge workers.

    “State-ofthe-art generative AI systems have reached proficiency levels that not only match human creativity in certain evaluations [20,33], but can also enhance the creative output of knowledge workers [1,2,30].”

    From Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking (2025) · cites this paper as [2]

    This source is cited for the sentence’s second clause, about enhancing creative output, not for matching human creativity. Section 4.3 (pp. 5–6) reports that participants generated more ideas and covered more idea categories with ChatGPT, but it did not evaluate idea quality. The study tested divergent-ideation tasks with a mixed group of students and professionals (Section 3.1, p. 3), so its findings do not establish enhanced creative output for knowledge workers generally. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Emerging evidence shows measurable shifts in human writing styles, creative ideation, and divergent thinking following the widespread adoption of systems like ChatGPT.

    “Emerging evidence shows measurable shifts in human writing styles, creative ideation, and divergent thinking following the widespread adoption of systems like ChatGPT [46, 6, 7, 24, 44, 79].”

    From Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) (2025) · cites this paper as [6]

    Section 4.1 (p. 5) reports that ideas generated with ChatGPT were more homogenized at the group level than ideas generated with a non-AI tool, supporting a measurable shift in creative ideation on divergent-thinking tasks. The study does not establish changes in human writing styles or measure shifts over time following widespread adoption; it compares participants’ outputs under two tool conditions. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Some studies have found evidence of a homogenization effect from LLM assistance.

    “Other studies have found evidence of a homogenization effect (i.e., less semantically distinct outputs) from LLM assistance across a variety of GenAI text and image generation tools [8, 12, 31, 99, 139].”

    From The Impact of Generative AI on Creativity in Software Development: A Research Agenda (2025) · cites this paper as [8]

  • Most existing studies on AI's impact on creativity have focused on textual outputs using text-to-text models, such as writing essays, fictional stories, or imaginative and experimental scenarios.

    “Moreover, most existing studies on AI's impact on creativity have focused on textual outputs using text-to-text models, like writing essays [75], ￿ ctional stories [28], or imaginative and experimental scenarios [9, 10].”

    From Creativity in the Age of AI: Evaluating the Impact of Generative AI on Design Outputs and Designers' Creative Thinking (2024) · cites this paper as [9]

  • Across both experiments, LLM assistance was helpful during the exposure phase.

    “Across both experiments, we observed that LLM assistance was helpful during the exposure phase, aligning with previous research that highlights LLMs' ability to enhance creative task performance when users have access to AI-generated ideas or strategies [2,14].”

    From Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking (2025) · cites this paper as [2]

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

  • Inconsistent p-value across alpha = .05: t (32) = 2.32, p = .237

    Major

    The paper reports p = .237 here; recomputing from the test statistic and its degrees of freedom gives p = .027 (two-tailed). The two fall on opposite sides of alpha = .05. The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .237 · recomputed p = .027

  • Inconsistent p-value: r (32)=.46, p = .004

    Minor

    The paper reports p = .004 here; recomputing from the test statistic and its degrees of freedom gives p = .006 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .004 · recomputed p = .006

  • Inconsistent p-value: r (32)=.43, p = .008

    Minor

    The paper reports p = .008 here; recomputing from the test statistic and its degrees of freedom gives p = .011 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .008 · recomputed p = .011

  • Inconsistent p-value: t (32) = 2.154, p = .038

    Minor

    The paper reports p = .038 here; recomputing from the test statistic and its degrees of freedom gives p = .039 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .038 · recomputed p = .039

4 of 4

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upt(32) = 2.32reported p = .237 · recomputed p = .027The two fall on opposite sides of .05

    “t (32) = 2.32, p = .237”

  • Doesn’t add upt(32) = 2.154reported p = .038 · recomputed p = .039

    “t (32) = 2.154, p = .038”

  • Doesn’t add upr(32) = 0.43reported p = .008 · recomputed p = .011

    “r (32)=.43, p = .008”

  • Doesn’t add upr(32) = 0.46reported p = .004 · recomputed p = .006

    “r (32)=.46, p = .004”

  • Not confirmedt(35) = 1.028reported p = .312 · recomputed p = .311

    “t (35) = 1.028, p = .312”

  • Not confirmedt(32) = 0.091reported p = .927 · recomputed p = .928

    “t (32) = .091, p = .927”

  • Consistentt(32) = 0.944reported p = .352 · recomputed p = .352

    “t (32) = .944, p = .352”

  • Consistentt(32) = 3.21reported p = .003 · recomputed p = .003

    “t (32) = 3.21, p = .003”

  • Consistentt(32) = 2.1reported p = .044 · recomputed p = .044

    “t (32) = 2.10, p = .044”

  • Consistentt(32) = 3.5reported p = .001 · recomputed p = .001

    “t (32) = 3.50, p = .001”

  • Consistentt(32) = 1.65reported p = .108 · recomputed p = .109

    “t (32) = 1.65, p = .108”

  • Consistentt(32) = 0.134reported p = .894 · recomputed p = .894

    “t (32) = .134, p = .894”

  • Consistentt(32) = 1.424reported p = .164 · recomputed p = .164

    “t (32) = 1.424, p = .164”

  • Consistentr(32) = 0.23reported p = .185 · recomputed p = .191

    “r (32)=.23, p = .185”

  • Consistentt(32) = 1.016reported p = .317 · recomputed p = .317

    “t (32) = 1.016, p = .317”

  • Consistentt(32) = 0.276reported p = .784 · recomputed p = .784

    “t (32) = .276, p = .784”

  • Consistentt(32) = 0.222reported p = .826 · recomputed p = .826

    “t (32) = .222, p = .826”

  • Consistentt(32) = 0.208reported p = .836 · recomputed p = .837

    “t (32) = .208, p = .836”

18 of 18

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Percentage of women does not match the count

    Minor

    The reported 12 women among 33 participants is 36.36%, not 33.36%.

    Reported 33.36 · recomputed 36.363636

  • CSI test appears to include excluded participants

    Minor

    A paired t-test with 35 degrees of freedom corresponds to 36 participants, but the paper says three of its 36 recruits were excluded from all analyses. The other paired comparisons report 32 degrees of freedom.

  • Low-engagement percentage conflicts between table and text

    Minor

    Table 1 assigns 30.56% to eight participants reporting low engagement, while the accompanying text assigns 22.22% to the same count.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 25

  1. Nina Begus. 2023. Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytelling. arXiv preprint arXiv:2310.12902 (2023). · arXiv
  2. Margaret A Boden. 2004. The Creative Mind: Myths and Mechanisms. Routledge.
  3. Rishi Bommasani Kathleen A Creel Ananya Kumar Dan Jurafsky and Percy S Liang. 2022. Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? In Advances in Neural Information Processing Systems Vol. 35. 3663–3678.
  4. Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel Ziegler Jeffrey Wu Clemens Winter Chris Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems Vol. 33. 1877–1901.
  5. Daniel Buschek, Lukas Mecke, Florian Lehmann, and Hai Dang. 2021. Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems. In Joint Proceedings of the ACM IUI 2021 Workshops.
  6. Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020. How Novelists Use Generative Language Models: An Exploratory User Study. In Workshop on Human-AI Co-Creation with Generative Models (HAI-GEN 2020).
  7. Kathy Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Sage.
  8. John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching stories with generative pretrained language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems.
  9. Anil R Doshi and Oliver Hauser. 2023. Generative artificial intelligence enhances creativity. Available at SSRN (2023).
  10. Ziv Epstein, Aaron Hertzmann, and Investigators of Human Creativity. 2023. Art and the science of generative AI. Science 380, 6650 (2023), 1110–1111.
  11. Agnes Guenther, Boris Eisenbart, and Andy Dong. 2017. Creativity as a way to innovate successfully. In DS 87-8 Proceedings of the 21st International Conference on Engineering Design (ICED 17) Vol 8: Human Behaviour in Design, Vancouver, Canada, 21-25.08. 2017. 389–398.
  12. Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in Psychology. Vol. 52. Elsevier, 139–183.
  13. Tom Hewett, Mary Czerwinski, Michael Terry, Jay Nunamaker, Linda Candy, Bill Kules, and Elisabeth Sylvan. 2005. Creativity support tool evaluation methods and metrics. In Creativity Support Tools: A workshop sponsored by the National Science Foundation. 10–24.
  14. David Kirsh. 2011. Creative cognition in choreography. In Proceedings of the 2nd International Conference on Computational Creativity.
  15. Max Kreminski and John Joon Young Chung. 2024. Intent Elicitation in Mixed-Initiative Co-Creativity. In Joint Proceedings of the ACM IUI Workshops.
  16. Max Kreminski, Isaac Karth, Michael Mateas, and Noah Wardrip-Fruin. 2022. Evaluating mixed-initiative creative interfaces via expressive range coverage analysis. In IUI Workshops. 34–45.
  17. Max Kreminski and Michael Mateas. 2021. Reflective creators. In International Conference on Computational Creativity. 309–318.
  18. Isabelle Levent and Lila Shroff. 2023. The Model is the Message. In The Second Workshop on Intelligent and Interactive Writing Assistants.
  19. Roberta M Milgram and Norman A Milgram. 1976. Creative thinking and creative performance in Israeli students.Journal of educational psychology 68, 3 (1976), 255.
  20. OpenAI. 2023. ChatGPT: A Large-Scale Conversational Language Model. https://www.openai.com/research/chatgpt. Accessed: August 3rd, 2023.
  21. Long Ouyang Jeffrey Wu Xu Jiang Diogo Almeida Carroll Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray John Schulman Jacob Hilton Fraser Kelton Luke Miller Maddie Simens Amanda Askell Peter Welinder Paul F Christiano Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems Vol. 35. 27730–27744.
  22. Melissa Roemmele. 2021. Inspiration through observation: Demonstrating the influence of automatically generated text on creative writing. In Proceedings of the 12th International Conference on Computational Creativity (ICCC ’21).
  23. Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2023. Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. arXiv preprint arXiv:2310.12953 (2023). · arXiv
  24. E Paul Torrance. 1966. Torrance tests of creative thinking. Educational and Psychological Measurement (1966).
  25. Qian Wan Siying Hu Yu Zhang Piaohong Wang Bo Wen and Zhicong Lu. 2023. “It Felt Like Having a Second Mind”’: Investigating Human-AI Co-creativity in Prewriting with Large Language Models. arXiv preprint arXiv:2307.10811 (2023). · arXiv

Cited by

Sorted by citations

Related

Authors
Anderson, B. R., Shah, J. H., & Kreminski, M.
Affiliations
  • Independent Researcher, United States
  • Santa Clara University (US)
ORCID
Venue
Creativity and Cognition
Publisher
ACM
Published
2024-06-23
Pages
413-425
DOI
10.1145/3635636.3656204
OpenAlex
W4391556516
Cited by
188
References
50
Citations by year
2026: 97 · 2025: 79 · 2024: 12
FWCI
117.47
Type
conference paper
Language
en
Open access
Licence
Locations
  • Creativity and Cognition (publishedVersion, cc-by-nc-sa)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • Scholar Commons (Santa Clara University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
  • Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
large language models, creativity support tools, creative idea generation, ChatGPT, idea generation, AI-assisted creativity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review