The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index; none of the 40 sources its claims cite is checked for support yet.
Homogenization Effects of Large Language Models on Human Creative Ideation
Open accessAbstract
Large language models (LLMs) are now being used in a wide variety of contexts, including as creativity support tools (CSTs) intended to help their users come up with new ideas. But do LLMs actually support user creativity? We hypothesized that the use of an LLM as a CST might make the LLM’s users feel more creative, and even broaden the range of ideas suggested by each individual user, but also homogenize the ideas suggested by different users. We conducted a 36-participant comparative user study and found, in accordance with the homogenization hypothesis, that different users tended to produce less semantically distinct ideas with ChatGPT than with an alternative CST. Additionally, ChatGPT users generated a greater number of more detailed ideas, but felt less responsible for the ideas they generated. We discuss potential implications of these findings for users, designers, and developers of LLM-based CSTs.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
A 36-participant within-subjects study compared ChatGPT and the Oblique Strategies deck for divergent ideation, finding that ChatGPT users produced more ideas but more semantically homogeneous ideas at the group level, not at the individual level, and felt less responsible for their ideas. ChatGPT also increased fluency, flexibility, and elaboration, but did not increase unique ideas.
Key findings
- Group-level homogenization: ideas generated with ChatGPT were less divergent from the average embedding of all ideas for that task (M = .24, SD = .07) than ideas generated with Oblique Strategies (M = .28, SD = .08), t(32) = 2.154, p = 0.038, d = .47, 95% CI [.00, .07].
- Individual-level homogenization: no observable difference between ChatGPT (M = .65, SD = .07) and Oblique Strategies (M = .66, SD = .08), t(32) = .944, p = 0.352, d = .12, 95% CI [-.04, .01].
- Responsibility: participants assigned less responsibility to themselves and more to the tool with ChatGPT (M = 48.17%, SD = 26.22%) than with Oblique Strategies (M = 63.63%, SD = 17.36%), t(32) = 3.21, p = 0.003, d = .67, 95% CI [-24.60%, -5.51%].
- Other creativity facets: ChatGPT produced more ideas (M = 8.39, SD = 3.39 vs. M = 7.32, SD = 3.22; about 15% increase; p = 0.044), more idea categories (M = 8.58, SD = 2.90 vs. M = 6.77, SD = 3.69; about 27% more; p = 0.001), and greater stoplisted word count (M = 8.25, SD = 4.61 vs. M = 6.46, SD = 2.55; t(32) = 2.32, p = 0.237, d = .48), while unique ideas did not differ (M = .74, SD = .69 vs. M = .97, SD = 1.00; p = 0.108).
Methods
- Within-subjects experiment: 36 participants recruited, 33 analyzed after excluding 3 for failure to follow directions; participants completed four divergent ideation tasks based on a subset of the Torrance Tests of Creative Thinking, half with ChatGPT (GPT-3.5 versions from May and August 2023) and half with the Oblique Strategies web app, producing 1271 ideas total, with 8 minutes per prompt and randomized/balanced order.
- Measures: semantic similarity via sentence embeddings (all-MiniLM-L6-v2) for homogenization; Creativity Support Index and bespoke responsibility ratings; fluency by idea count, flexibility by human coding into 181 idea categories, elaboration by stoplisted word count; plus talk-aloud process observations and post-session interviews.
- Analysis: paired comparisons reported with t(32), p-values, effect sizes, and 95% confidence intervals; semantic embedding model validated against human idea categorization, with all-MiniLM-L6-v2 averaging 58.92% agreement.
Limitations
- Stated: the lab context may not reflect real-world LLM creativity support; participants had fixed, relatively short time per prompt; ideation prompts were supplied by experimenters rather than formulated by users.
- Stated: the study used general-purpose ChatGPT rather than an LLM-based CST designed specifically for divergent ideation, so alternative deployments may avoid homogenization; prior evidence suggests feedback-tuned LLMs may homogenize where base LLMs do not.
- Evident from the text: the paper did not evaluate idea quality, and its semantic similarity measure is imperfect—even the best embedding model achieved only 59% agreement with human coders on average, and cosine similarity between sentence embeddings may be influenced more by noun overlap than by all nuances of sentence meaning.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
54 Own result · 31 Not checked yet · 31 Own interpretation · 8 Own method · 4 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“different users tended to produce less semantically distinct ideas with ChatGPT than with an alternative CST”
- Also stated in: § 4.1 Homogenization · p. 5 · plain
“When participants used ChatGPT, the ideas they produced were less divergent from the average embedding of all ideas generated for that task, ( M = .24, SD = .07), compared to the ideas that they produced when using OS, ( M = .28, SD = .08), t (32) = 2.154, p = 0.038, d = .47, 95% CI [.00,.07].”
- Also stated in: § 4.1 Homogenization · p. 5 · plain
“At the group level, ideas produced with the help of ChatGPT were more homogenized.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.1 Homogenization · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With which CST do participants produce more semantically similar ideas at the individual level? (A: no difference )”
- Also stated in: § 4.1 Homogenization · p. 5 · plain
“When participants used ChatGPT, the ideas they produced were not observably more divergent from the average embedding of all of the other ideas that they themselves generated for the same ideation prompt, ( M = .65, SD = .07), compared to semantic dissimilarity for ideas that they produced when using OS, ( M = .66, SD = .08), t (32) = .944, p = 0.352, d = .12, 95% CI [-.04,.01].”
- Also stated in: § 4.1 Homogenization · p. 5 · plain
“At the individual level, we did not observe a difference in homogenization for ideas generated with the help of ChatGPT.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.1 Homogenization · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Beyond originality, do ChatGPT and non-ChatGPT users differ in terms of other facets of creativity, such as fluency, flexibility, and elaboration? (A: ChatGPT → higher fluency, flexibility, and elaboration )”
- Also stated in: § 4.3 Other Facets of Creativity · p. 6 · plain
“Participants generated ideas that hit about 27% more categories when using ChatGPT ( M = 8.58, SD =2.90), compared to the number of categories covered when using OS ( M = 6.77, SD = 3.69), t (32) = 3.50, p = 0.001, d = .54, 95% CI [.75,2.86].”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.3 Other Facets of Creativity · p. 6 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT users generated a greater number of more detailed ideas”
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
“Beyond originality, do ChatGPT and non-ChatGPT users differ in terms of other facets of creativity, such as fluency, flexibility, and elaboration? (A: ChatGPT → higher fluency, flexibility, and elaboration )”
- Also stated in: § 4.3 Other Facets of Creativity · p. 5 · plain
“Participants generated about one additional idea-approximately a 15% increase-when using ChatGPT ( M = 8.39, SD =3.39), compared to the number of ideas they generated when using OS ( M = 7.32, SD = 3.22), t (32) = 2.10, p = 0.044, d = .32, 95% CI [.03,2.11].”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“felt less responsible for the ideas they generated”
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
“Do ChatGPT users feel more or less responsible for the ideas they produce? (A: less responsible )”
- Also stated in: § 4.2 Sense of Responsibility · p. 5 · plain
“Participants assigned less responsibility to themselves (and more to the tool) for ideas generated while using ChatGPT ( M =48.17%, SD =26.22%), compared to ideas generated while using OS ( M =63.63%, SD =17.36%), t (32) = 3.21, p = 0.003, d = .67, 95% CI [-24.60%, -5.51%,].”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We did not observe any relationship between the number of LLM prompts entered and average homogeneity of ideas generated, r (32)=.23, p = .185.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying a 5% uniqueness threshold to idea categories, we again did not observe a difference in the number of unique responses (with a 5% threshold) produced by participants using ChatGPT ( M = .87, SD = .88) or OS ( M = .90, SD = 1.07), t (32) = .134, p = 0.894, d = .02, 95% CI [-.34,.39].”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and between the number of prompts entered and weighted uniqueness scores, r (32)=.46, p = .004.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This was similar to the interaction pattern we observed with OS”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants also reported finding ChatGPT to be less engaging (22.22%, n =8)”
plainn = 8- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The quote only partly states this claim. 1
- Probably not a substantial claim. 38
- It may state two separate claims as one. 14
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT users generated a greater number of more detailed ideas”
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
“Beyond originality, do ChatGPT and non-ChatGPT users differ in terms of other facets of creativity, such as fluency, flexibility, and elaboration? (A: ChatGPT → higher fluency, flexibility, and elaboration )”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Other prompting strategies included asking for relevant scientific or factual information (e.g. 'What are the most popular dolls?', 'How common is face blindness?', etc.), asking for responses from a specific perspective (e.g. project manager, scientist, athlete, etc.), or adding creative constraints (e.g. make this toy for a dog, respond in the form of a story, etc.).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants also made self-deprecating remarks regarding their familiarity and technical ability with ChatGPT (5.56%, n = 2), but made no such remarks regarding OS.”
plainn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The majority of these participants copied selectively: only 1 copied the entirety of the ChatGPT output without any attempt at curation or editing.”
plainn = 15- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most participants (72.22%, n = 26) also iterated on their prompt, either by adding more context to their initial prompt, probing with more specific questions or variations, or by asking ChatGPT to regenerate answers or provide additional responses.”
plainn = 26- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and that the LLM responses became too specific too quickly (5.56%, n = 2).”
plainn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The second most common theme was a positive sentiment regarding the speed and accuracy of the LLM-provided responses (25.00%, n = 9).”
plainn = 9- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“that the LLM responses were too repetitive (8.33%, n = 3)”
plainn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Coupled with evidence that ChatGPT users exhibit greater fluency, flexibility, and elaboration than users of an alternative CST”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we only observed one participant in the entire study directly copy-paste the entirety of a ChatGPT response without any further curation or modification”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“some participants in our study viewed ChatGPT as a 'very fast and accurate' authoritative source that 'gave [...] the right answers'”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Coding was conducted while blind to the CST condition (i.e., coders did not know if an idea was generated with or without LLM support).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The order of CSTs and prompts was randomized per participant and balanced across the entire experiment.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Begus [7] compares human-written to GPT-generated short stories, observing that GPTgenerated stories are less diverse than human-written stories in structure but exhibit greater gender and sexual diversity in character description.”
What the sources say
- Not checked yetcites [7] [7] Nina Begus. 2023. Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytelling. arXiv preprint arXiv:2310.12902 (2023). -
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“We did not collect responses for the Collaboration subscale, which is irrelevant and often omitted in exclusively single-user contexts like that of our study (e.g., [5, 87]).”
What the sources say
- Not checked yetcites [5] [5] Ilhan Aslan, Katharina Weitz, Ruben Schlagowski, Simon Flutura, Susana Garcia Valesco, Marius Pfeil, and Elisabeth André. 2019. Creativity support and multimodal pen-based interaction. In 2019 International Conference on Multimodal Interaction . 135-144. - source
- Not checked yetcites [87] [87] Qian Wan and Zhicong Lu. 2023. GANCollage: A GAN-Driven Digital Mood Board to Facilitate Ideation in Creativity Support. In Proceedings of the 2023 ACM Designing Interactive Systems Conference . 136-146. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Chakrabarty et al. [15] compare LLM-generated short stories to stories by expert human writers and find that the human-written stories substantially outperform the LLM-generated stories on all dimensions of creativity measured by the TTCT, including the originality dimension.”
What the sources say
- Not checked yetcites [15] [15] Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2023. Art or Artifice? Large Language Models and the False Promise of Creativity. arXiv preprint arXiv:2309.14556 (2023). -
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The Torrance Tests of Creative Thinking (TTCT) [85] evaluate the creativity of test-takers according to four facets of creative output: fluency , or sheer quantity of artifacts created; flexibility , or quantity of distinct categories of artifacts created; originality , or dissimilarity of created artifacts to others' creations; and elaboration , or level of detail in created artifacts.”
What the sources say
- Not checked yetcites [85] [85] E Paul Torrance. 1966. Torrance tests of creative thinking. Educational and Psychological Measurement (1966). -
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Padmakumar and He evaluate the baseline GPT-3 model [12] versus the instructionfinetuned variant InstructGPT [68] in the context of a short-form argumentative essay writing task and find a homogenization effect from LLM assistance at both the lexical and content levels, but only for the instruction-tuned LLM [69].”
What the sources say
- Not checked yetcites [69] [69] Vishakh Padmakumar and He He. 2023. Does Writing with Language Models Reduce Content Diversity? arXiv preprint arXiv:2309.05196 (2023). -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the 36 participants originally recruited, three were excluded from all analyses for failure to follow direction (e.g., not using ChatGPT when prompted to do so).”
plainn = 36- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Roemmele [74] proposes and provides evidence for an 'inspiration through observation' model of how human writers are influenced by LLMs: observation of LLM-generated text shapes the ideas expressed by the LLM's user even when the user does not incorporate LLM-generated text directly into their writing.”
What the sources say
- Not checked yetcites [74] [74] Melissa Roemmele. 2021. Inspiration through observation: Demonstrating the influence of automatically generated text on creative writing. In Proceedings of the 12th International Conference on Computational Creativity (ICCC '21) . -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Arnold et al. found that 'predictive text encourages predictable writing' [4] in the context of singleword suggestions given by smartphone keyboards.”
What the sources say
- Not checked yetcites [4] [4] Kenneth C Arnold, Krysta Chauncey, and Krzysztof Z Gajos. 2020. Predictive text encourages predictable writing. In Proceedings of the 25th International Conference on Intelligent User Interfaces . 128-138. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study protocol and recruitment materials were approved by the Santa Clara University IRB.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“groups tend to be less creative than individuals when majoritarian convergence processes are used, and minority dissent within group creative processes tends to lead to greater creativity [30]”
What the sources say
- Not checked yetcites [30] [30] Umer Farooq, John M Carroll, and Craig H Canoe. 2008. Designing for creativity in computer-supported cooperative work. International Journal of e-Collaboration (IJeC) 4, 4 (2008), 51-75. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“The Creativity Support Index (CSI) [17] is a widely used and psychometrically validated survey instrument that attempts to standardize some aspects of this experience reporting process across different CSTs.”
What the sources say
- Not checked yetcites [17] [17] Erin Cherry and Celine Latulipe. 2014. Quantifying the creativity support of digital tools through the Creativity Support Index. ACM Transactions on ComputerHuman Interaction (TOCHI) 21, 4 (2014). - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Bhat et al. [8] find that LLM-supplied next-phrase suggestions may alter the form and content of a human user's writing even when the user dislikes these suggestions.”
What the sources say
- Not checked yetcites [8] [8] Advait Bhat, Saaket Agashe, Parth Oberoi, Niharika Mohile, Ravi Jangir, and Anirudha Joshi. 2023. Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing. In Proceedings of the 28th International Conference on Intelligent User Interfaces . 436-452. - source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Lee et al. [54] find that LLM support tends to increase the diversity of a user's vocabulary but may reduce their feelings of ownership for the text they produce.”
What the sources say
- Not checked yetcites [54] [54] Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a humanAI collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems . - source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Jakesch et al. [41] find that an opinionated LLM-based CST influences the opinions expressed by its users in argumentative writing.”
What the sources say
- Not checked yetcites [41] [41] Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-writing with opinionated language models affects users' views. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“though it may still be difficult for them to adapt their prompting strategies to elicit more diverse responses from the LLM.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For an LLM to directly transform a brief expression of intent into a substantially larger artifact (e.g., a passage of text), it will necessarily have to make a large number of creative decisions (e.g., individual word choices) on the user's behalf, and these decisions will be made by reference to the relative likelihood of different alternatives.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This may be because participants can see all of their own ideas, so they can actively modulate the diversity of the set of ideas that they have generated”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Third, we infer that LLM users might be able to resist the homogenization effect if they are given information about what kinds of stereotyped outputs the LLM tends to produce in a particular creative context”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Because our study made use of a general-purpose LLM-based chatbot (ChatGPT) rather than an LLM-based CST specifically designed to support divergent ideation”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most organic divergent ideation scenarios are not subject to these constraints.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study was conducted in a lab context and consequently may not reflect some aspects of how people use LLMs for creativity support 'in the wild'.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“additionally, the ideation prompts in our study were supplied by us, rather than being formulated independently by CST users.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to keep study sessions to a manageable length, participants were only allowed a fixed and relatively short amount of time to respond to each ideation prompt”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the LLM itself functions most directly as a means of refracting user-supplied inputs through the lens of statistical patterns learned from its training data.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, participants cannot easily gauge the similarity or dissimilarity of their own responses to other people's responses, so they cannot actively intervene to modulate diversity at the group level.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These findings suggest that interventions at both the AI and UI layers may help to mitigate homogenization effects in future LLM-based CSTs.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This limits the replicability of research results derived from OpenAI models and therefore the suitability of these models for research pipelines.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: OpenAI's sentence embedding models are proprietary, cost money to use, cannot be run locally, and are not open source.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Altogether, we conclude that users viewing ChatGPT as authoritative may have led them to accept LLM-suggested ideas as valid, but it did not seem to constrain the process of ideation by displacing ideas that were somehow incompatible with those suggested by ChatGPT.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Padmakumar and He have found evidence that feedbacktuned LLMs may induce creative homogenization where base (nonfeedback-tuned) LLMs do not [69]”
What the sources say
- Not checked yetcites [69] [69] Vishakh Padmakumar and He He. 2023. Does Writing with Language Models Reduce Content Diversity? arXiv preprint arXiv:2309.05196 (2023). -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it should be noted that alternative deployments of LLMs may be able to avoid homogenizing user ideas.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, the rapid pace of progress in machine learning research means that transformerbased sentence embeddings have not yet been validated against human judgments of semantic similarity in the context of creativity research specifically.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“There also exists some evidence that cosine similarity between sentence embeddings is more strongly influenced by overlap in the set of nouns than by other similarities [66]”
What the sources say
- Not checked yetcites [66] [66] Dmitry Nikolaev and Sebastian Padó. 2023. Representation biases in sentence transformers. arXiv preprint arXiv:2301.13039 (2023). -
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The Oblique Strategies deck is a valid alternative creativity support tool for comparison with ChatGPT.
“We directed participants to a web app version of the deck [76] as an alternative CST to ChatGPT in our control condition.”
“We conducted a within-subjects experiment to evaluate the effects of using two different CSTs for idea generation: ChatGPT and the Oblique Strategies (OS) deck.”
The paper relies on this without support.
ChatGPT 3.5 is representative of LLM-based creativity support tools for generalizing homogenization effects.
“Participants in this study used the versions of ChatGPT 3.5 released on May 3rd 2023 (n=6, 16.6%) and on August 3rd, 2023 (30, 83.3%).”
“Collectively, our results suggest that LLM-driven homogenization stems from the LLM providing different users with similar ideas”
The paper relies on this without support.
The sample size of 33 participants is sufficient to detect the reported effects.
“Our sample included 33 participants, ranging in age from 22 to 44 (M=28.36, SD=6.84)”
“Of the 36 participants originally recruited, three were excluded from all analyses for failure to follow direction”
The paper relies on this without support.
An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.
“Many of our findings follow naturally from the fact that an LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.”
The paper relies on this without support.
The Oblique Strategies deck consists of a relatively small set of fixed text strings.
“The deck consists of a relatively small set of fixed text strings”
The paper relies on this without support.
Aggregated word embeddings are generally outperformed on semantic similarity tasks by more recent transformer-based sentence embeddings, which can take sentence structure into account.
“Aggregated word embeddings are generally outperformed on semantic similarity tasks by more recent transformer-based sentence embeddings, which (unlike aggregated word embeddings) are able to take sentence structure into account.”
The authors state this as an assumption.
Human-constructed idea categories from the flexibility analysis are a valid ground truth for semantic similarity judgments.
“Our experiment took the human-constructed idea categories produced by our flexibility analysis (Section 4.3.2) as a source of ground truth for semantic similarity judgments”
The paper relies on this without support.
OpenAI's sentence embedding models are proprietary, cost money to use, cannot be run locally, and are not open source.
“OpenAI models (unlike the SentenceTransformers models) are proprietary: they cost money to use, cannot be run locally, and are not open source.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“State-ofthe-art generative AI systems have reached proficiency levels that not only match human creativity in certain evaluations [20,33], but can also enhance the creative output of knowledge workers [1,2,30].”
From Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Emerging evidence shows measurable shifts in human writing styles, creative ideation, and divergent thinking following the widespread adoption of systems like ChatGPT [46, 6, 7, 24, 44, 79].”
From Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Other studies have found evidence of a homogenization effect (i.e., less semantically distinct outputs) from LLM assistance across a variety of GenAI text and image generation tools [8, 12, 31, 99, 139].”
From The Impact of Generative AI on Creativity in Software Development: A Research Agenda (2025) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“Moreover, most existing studies on AI's impact on creativity have focused on textual outputs using text-to-text models, like writing essays [75], ctional stories [28], or imaginative and experimental scenarios [9, 10].”
From Creativity in the Age of AI: Evaluating the Impact of Generative AI on Design Outputs and Designers' Creative Thinking (2024) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Across both experiments, we observed that LLM assistance was helpful during the exposure phase, aligning with previous research that highlights LLMs' ability to enhance creative task performance when users have access to AI-generated ideas or strategies [2,14].”
From Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking (2025) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Existing research has shown that LLM usage can lead to the homogenization of ideas within groups, where participants tend to converge on similar outcomes when using AIgenerated suggestions [2,19,45].”
From Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking (2025) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Literature shows that GenAI can homogenize the ideas generated by different users [8] and limit human agency and lead to creative fixation [5, 21, 101].”
From Decoupling of Usefulness and Novelty: Evaluating the Impact of Generative AI on Design Outputs and Novice Designers' Creative Thinking (2026) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“People who use AI tend to produce more similar outputs compared to their counterparts who do not—an effect that has been shown in a range of contexts involving open-ended output, such as product design [1, 18] and short-story writing [4].”
From The Role of Human Agency in Human-AI Co-Creativity (2026) · cites this paper as [1]
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“t (32) = 2.32, p = 0.237”
Reported p = .237 · recomputed p = .027
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“r (32)=.46, p = .004”
Reported p = .004 · recomputed p = .006
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“r (32)=.43, p = .008”
Reported p = .008 · recomputed p = .011
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t (32) = 2.154, p = 0.038”
Reported p = .038 · recomputed p = .039
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upt(32) = 2.32reported p = .237 · recomputed p = .027The two fall on opposite sides of .05
“t (32) = 2.32, p = .237”
- Doesn’t add upt(32) = 2.154reported p = .038 · recomputed p = .039
“t (32) = 2.154, p = .038”
- Doesn’t add upr(32) = 0.43reported p = .008 · recomputed p = .011
“r (32)=.43, p = .008”
- Doesn’t add upr(32) = 0.46reported p = .004 · recomputed p = .006
“r (32)=.46, p = .004”
- Not confirmedt(35) = 1.028reported p = .312 · recomputed p = .311
“t (35) = 1.028, p = .312”
- Not confirmedt(32) = 0.091reported p = .927 · recomputed p = .928
“t (32) = .091, p = .927”
- Consistentt(32) = 0.944reported p = .352 · recomputed p = .352
“t (32) = .944, p = .352”
- Consistentt(32) = 3.21reported p = .003 · recomputed p = .003
“t (32) = 3.21, p = .003”
- Consistentt(32) = 2.1reported p = .044 · recomputed p = .044
“t (32) = 2.10, p = .044”
- Consistentt(32) = 3.5reported p = .001 · recomputed p = .001
“t (32) = 3.50, p = .001”
- Consistentt(32) = 1.65reported p = .108 · recomputed p = .109
“t (32) = 1.65, p = .108”
- Consistentt(32) = 0.134reported p = .894 · recomputed p = .894
“t (32) = .134, p = .894”
- Consistentt(32) = 1.424reported p = .164 · recomputed p = .164
“t (32) = 1.424, p = .164”
- Consistentr(32) = 0.23reported p = .185 · recomputed p = .191
“r (32)=.23, p = .185”
- Consistentt(32) = 1.016reported p = .317 · recomputed p = .317
“t (32) = 1.016, p = .317”
- Consistentt(32) = 0.276reported p = .784 · recomputed p = .784
“t (32) = .276, p = .784”
- Consistentt(32) = 0.222reported p = .826 · recomputed p = .826
“t (32) = .222, p = .826”
- Consistentt(32) = 0.208reported p = .836 · recomputed p = .837
“t (32) = .208, p = .836”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“including 63.63% (n=21) men and 33.36% (n=12) women”
Reported 33.36 · recomputed 36.363636
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 Retrospective Reflections · p. 7
“t (35) = 1.028, p = 0.312”
§ 3.1 Participants · p. 3
“Of the 36 participants originally recruited, three were excluded from all analyses for failure to follow direction”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 Retrospective Reflections · p. 7
“| Low Engagement (ChatGPT) | ChatGPT allowed me to turn my brain off. It did more of the heavy lifting. | 8 | 30.56% |”
§ 4.4 Retrospective Reflections · p. 6
“ChatGPT to be less engaging (22.22%, n =8)”
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Confidence Intervals from Normalized Data: A correction to Cousineau (2005)
Morey, R. D. (2008) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.04.2.p061 · Cited by 1,991
Ambiguity as a resource for design
Gaver, W. W., Beaver, J., & Benford, S. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642653 · Cited by 1,474
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
Jansson, D. G., & Smith, S. M. (1991) · Design Studies · doi:10.1016/0142-694x(91)90003-f · Cited by 1,087
Silvia, P. J., Winterstein, B. P., Willse, J. T., Barona, C. M., Cram, J. T., Hess, K. I., Martinez, J. L., & Richard, C. A. (2008) · Psychology of Aesthetics, Creativity, and the Arts · doi:10.1037/1931-3896.2.2.68 · Cited by 912
Creativity support tools: accelerating discovery and innovation
Shneiderman, B. (2007) · Communications of the ACM · doi:10.1145/1323688.1323689 · Cited by 765
Text and Code Embeddings by Contrastive Pre-TrainingPreprint
Neelakantan, A., Xu, T., Puri, R., Radford, A., Han, J. M., Tworek, J., Yuan, Q.-M., Tezak, N., Kim, J. W., Hallacy, C., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2201.10005 · Cited by 661
AI-Mediated Communication: Definition, Research Agenda, and Ethical Considerations
Hancock, J. T., Naaman, M., & Levy, K. (2020) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmz022 · Cited by 658
Research works printed but not linked: 22
- Margaret A Boden. 2004. The Creative Mind: Myths and Mechanisms. Routledge.
- Rishi Bommasani Kathleen A Creel Ananya Kumar Dan Jurafsky and Percy S Liang. 2022. Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? In Advances in Neural Information Processing Systems Vol. 35. 3663–3678.
- Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel Ziegler Jeffrey Wu Clemens Winter Chris Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems Vol. 33. 1877–1901.
- Daniel Buschek, Lukas Mecke, Florian Lehmann, and Hai Dang. 2021. Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems. In Joint Proceedings of the ACM IUI 2021 Workshops.
- Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020. How Novelists Use Generative Language Models: An Exploratory User Study. In Workshop on Human-AI Co-Creation with Generative Models (HAI-GEN 2020).
- Kathy Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Sage.
- John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching stories with generative pretrained language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems.
- Anil R Doshi and Oliver Hauser. 2023. Generative artificial intelligence enhances creativity. Available at SSRN (2023).
- Ziv Epstein, Aaron Hertzmann, and Investigators of Human Creativity. 2023. Art and the science of generative AI. Science 380, 6650 (2023), 1110–1111.
- Agnes Guenther, Boris Eisenbart, and Andy Dong. 2017. Creativity as a way to innovate successfully. In DS 87-8 Proceedings of the 21st International Conference on Engineering Design (ICED 17) Vol 8: Human Behaviour in Design, Vancouver, Canada, 21-25.08. 2017. 389–398.
- Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in Psychology. Vol. 52. Elsevier, 139–183.
- Tom Hewett, Mary Czerwinski, Michael Terry, Jay Nunamaker, Linda Candy, Bill Kules, and Elisabeth Sylvan. 2005. Creativity support tool evaluation methods and metrics. In Creativity Support Tools: A workshop sponsored by the National Science Foundation. 10–24.
- David Kirsh. 2011. Creative cognition in choreography. In Proceedings of the 2nd International Conference on Computational Creativity.
- Max Kreminski and John Joon Young Chung. 2024. Intent Elicitation in Mixed-Initiative Co-Creativity. In Joint Proceedings of the ACM IUI Workshops.
- Max Kreminski, Isaac Karth, Michael Mateas, and Noah Wardrip-Fruin. 2022. Evaluating mixed-initiative creative interfaces via expressive range coverage analysis. In IUI Workshops. 34–45.
- Max Kreminski and Michael Mateas. 2021. Reflective creators. In International Conference on Computational Creativity. 309–318.
- Isabelle Levent and Lila Shroff. 2023. The Model is the Message. In The Second Workshop on Intelligent and Interactive Writing Assistants.
- Roberta M Milgram and Norman A Milgram. 1976. Creative thinking and creative performance in Israeli students.Journal of educational psychology 68, 3 (1976), 255.
- OpenAI. 2023. ChatGPT: A Large-Scale Conversational Language Model. https://www.openai.com/research/chatgpt. Accessed: August 3rd, 2023.
- Long Ouyang Jeffrey Wu Xu Jiang Diogo Almeida Carroll Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray John Schulman Jacob Hilton Fraser Kelton Luke Miller Maddie Simens Amanda Askell Peter Welinder Paul F Christiano Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems Vol. 35. 27730–27744.
- Melissa Roemmele. 2021. Inspiration through observation: Demonstrating the influence of automatically generated text on creative writing. In Proceedings of the 12th International Conference on Computational Creativity (ICCC ’21).
- E Paul Torrance. 1966. Torrance tests of creative thinking. Educational and Psychological Measurement (1966).
Cited by
Sorted by citations
A Design Space for Intelligent and Interactive Writing Assistants
Lee, M., Gero, K. I., Chung, J. J. Y., Shum, S. B., Raheja, V., Shen, H., Venugopalan, S., Wambsganss, T., Zhou, D., Alghamdi, E. A., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642697 · Cited by 136
“The most common mechanism is to use text formatting such”
Self-citationAssessing and understanding creativity in large language models
Zhao, Y., Zhang, R., Li, W., & Li, L. (2025) · Machine Intelligence Research · doi:10.1007/s11633-025-1546-4 · Cited by 57
Divergent creativity in humans and large language models
Bellemare-Pepin, A., Lespinasse, F., Thölke, P., Harel, Y., Mathewson, K. W., Olson, J. A., Bengio, Y., & Jerbi, K. (2026) · Scientific Reports · doi:10.1038/s41598-025-25157-3 · Cited by 121
Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking
Kumar, H., Vincentius, J., Jordan, E., & Anderson, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714198 · Cited by 129
“Across both experiments, we observed that LLM assistance was helpful during the exposure phase, aligning with previous research that highlights LLMs’ ability to enhance creative task performance when users have access to AI-generated ideas or strategies [2, 14].”
InfluentialGenerative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization
Zhang, S., Xu, J., & Alvero, A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251327130 · Cited by 68
Metadata
- Authors
- Anderson, B. R., Shah, J. H., & Kreminski, M.
- Affiliations
- Independent Researcher, United States
- Santa Clara University (US)
- Venue
- Creativity and Cognition
- Publisher
- ACM
- Published
- 2024-06-23
- Pages
- 413-425
- OpenAlex
- W4391556516
- Cited by
- 188
- References
- 50
- Citations by year
- 2026: 97 · 2025: 79 · 2024: 12
- FWCI
- 117.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Creativity and Cognition (publishedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- Scholar Commons (Santa Clara University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
- Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- large language models, creativity support tools, creative idea generation, ChatGPT, idea generation, AI-assisted creativity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
