The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index; none of the 40 sources its claims cite is checked for support yet.
Homogenization Effects of Large Language Models on Human Creative Ideation
Open accessAbstract
Large language models (LLMs) are now being used in a wide variety of contexts, including as creativity support tools (CSTs) intended to help their users come up with new ideas. But do LLMs actually support user creativity? We hypothesized that the use of an LLM as a CST might make the LLM’s users feel more creative, and even broaden the range of ideas suggested by each individual user, but also homogenize the ideas suggested by different users. We conducted a 36-participant comparative user study and found, in accordance with the homogenization hypothesis, that different users tended to produce less semantically distinct ideas with ChatGPT than with an alternative CST. Additionally, ChatGPT users generated a greater number of more detailed ideas, but felt less responsible for the ideas they generated. We discuss potential implications of these findings for users, designers, and developers of LLM-based CSTs.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
A 36-participant within-subjects study compared ChatGPT and the Oblique Strategies deck for divergent ideation, finding that ChatGPT users produced more ideas but more semantically homogeneous ideas at the group level, not at the individual level, and felt less responsible for their ideas. ChatGPT also increased fluency, flexibility, and elaboration, but did not increase unique ideas.
Key findings
- Group-level homogenization: ideas generated with ChatGPT were less divergent from the average embedding of all ideas for that task (M = .24, SD = .07) than ideas generated with Oblique Strategies (M = .28, SD = .08), t(32) = 2.154, p = 0.038, d = .47, 95% CI [.00, .07].
- Individual-level homogenization: no observable difference between ChatGPT (M = .65, SD = .07) and Oblique Strategies (M = .66, SD = .08), t(32) = .944, p = 0.352, d = .12, 95% CI [-.04, .01].
- Responsibility: participants assigned less responsibility to themselves and more to the tool with ChatGPT (M = 48.17%, SD = 26.22%) than with Oblique Strategies (M = 63.63%, SD = 17.36%), t(32) = 3.21, p = 0.003, d = .67, 95% CI [-24.60%, -5.51%].
- Other creativity facets: ChatGPT produced more ideas (M = 8.39, SD = 3.39 vs. M = 7.32, SD = 3.22; about 15% increase; p = 0.044), more idea categories (M = 8.58, SD = 2.90 vs. M = 6.77, SD = 3.69; about 27% more; p = 0.001), and greater stoplisted word count (M = 8.25, SD = 4.61 vs. M = 6.46, SD = 2.55; t(32) = 2.32, p = 0.237, d = .48), while unique ideas did not differ (M = .74, SD = .69 vs. M = .97, SD = 1.00; p = 0.108).
Methods
- Within-subjects experiment: 36 participants recruited, 33 analyzed after excluding 3 for failure to follow directions; participants completed four divergent ideation tasks based on a subset of the Torrance Tests of Creative Thinking, half with ChatGPT (GPT-3.5 versions from May and August 2023) and half with the Oblique Strategies web app, producing 1271 ideas total, with 8 minutes per prompt and randomized/balanced order.
- Measures: semantic similarity via sentence embeddings (all-MiniLM-L6-v2) for homogenization; Creativity Support Index and bespoke responsibility ratings; fluency by idea count, flexibility by human coding into 181 idea categories, elaboration by stoplisted word count; plus talk-aloud process observations and post-session interviews.
- Analysis: paired comparisons reported with t(32), p-values, effect sizes, and 95% confidence intervals; semantic embedding model validated against human idea categorization, with all-MiniLM-L6-v2 averaging 58.92% agreement.
Limitations
- Stated: the lab context may not reflect real-world LLM creativity support; participants had fixed, relatively short time per prompt; ideation prompts were supplied by experimenters rather than formulated by users.
- Stated: the study used general-purpose ChatGPT rather than an LLM-based CST designed specifically for divergent ideation, so alternative deployments may avoid homogenization; prior evidence suggests feedback-tuned LLMs may homogenize where base LLMs do not.
- Evident from the text: the paper did not evaluate idea quality, and its semantic similarity measure is imperfect—even the best embedding model achieved only 59% agreement with human coders on average, and cosine similarity between sentence embeddings may be influenced more by noun overlap than by all nuances of sentence meaning.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
54 Own result · 31 Not checked yet · 31 Own interpretation · 8 Own method · 4 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“different users tended to produce less semantically distinct ideas with ChatGPT than with an alternative CST”
- Also stated in: § 4.1 Homogenization · p. 5 · plain
“When participants used ChatGPT, the ideas they produced were less divergent from the average embedding of all ideas generated for that task, ( M = .24, SD = .07), compared to the ideas that they produced when using OS, ( M = .28, SD = .08), t (32) = 2.154, p = 0.038, d = .47, 95% CI [.00,.07].”
- Also stated in: § 4.1 Homogenization · p. 5 · plain
“At the group level, ideas produced with the help of ChatGPT were more homogenized.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.1 Homogenization · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With which CST do participants produce more semantically similar ideas at the individual level? (A: no difference )”
- Also stated in: § 4.1 Homogenization · p. 5 · plain
“When participants used ChatGPT, the ideas they produced were not observably more divergent from the average embedding of all of the other ideas that they themselves generated for the same ideation prompt, ( M = .65, SD = .07), compared to semantic dissimilarity for ideas that they produced when using OS, ( M = .66, SD = .08), t (32) = .944, p = 0.352, d = .12, 95% CI [-.04,.01].”
- Also stated in: § 4.1 Homogenization · p. 5 · plain
“At the individual level, we did not observe a difference in homogenization for ideas generated with the help of ChatGPT.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.1 Homogenization · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Beyond originality, do ChatGPT and non-ChatGPT users differ in terms of other facets of creativity, such as fluency, flexibility, and elaboration? (A: ChatGPT → higher fluency, flexibility, and elaboration )”
- Also stated in: § 4.3 Other Facets of Creativity · p. 6 · plain
“Participants generated ideas that hit about 27% more categories when using ChatGPT ( M = 8.58, SD =2.90), compared to the number of categories covered when using OS ( M = 6.77, SD = 3.69), t (32) = 3.50, p = 0.001, d = .54, 95% CI [.75,2.86].”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4.3 Other Facets of Creativity · p. 6 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT users generated a greater number of more detailed ideas”
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
“Beyond originality, do ChatGPT and non-ChatGPT users differ in terms of other facets of creativity, such as fluency, flexibility, and elaboration? (A: ChatGPT → higher fluency, flexibility, and elaboration )”
- Also stated in: § 4.3 Other Facets of Creativity · p. 5 · plain
“Participants generated about one additional idea-approximately a 15% increase-when using ChatGPT ( M = 8.39, SD =3.39), compared to the number of ideas they generated when using OS ( M = 7.32, SD = 3.22), t (32) = 2.10, p = 0.044, d = .32, 95% CI [.03,2.11].”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“felt less responsible for the ideas they generated”
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
“Do ChatGPT users feel more or less responsible for the ideas they produce? (A: less responsible )”
- Also stated in: § 4.2 Sense of Responsibility · p. 5 · plain
“Participants assigned less responsibility to themselves (and more to the tool) for ideas generated while using ChatGPT ( M =48.17%, SD =26.22%), compared to ideas generated while using OS ( M =63.63%, SD =17.36%), t (32) = 3.21, p = 0.003, d = .67, 95% CI [-24.60%, -5.51%,].”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We did not observe any relationship between the number of LLM prompts entered and average homogeneity of ideas generated, r (32)=.23, p = .185.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applying a 5% uniqueness threshold to idea categories, we again did not observe a difference in the number of unique responses (with a 5% threshold) produced by participants using ChatGPT ( M = .87, SD = .88) or OS ( M = .90, SD = 1.07), t (32) = .134, p = 0.894, d = .02, 95% CI [-.34,.39].”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and between the number of prompts entered and weighted uniqueness scores, r (32)=.46, p = .004.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This was similar to the interaction pattern we observed with OS”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants also reported finding ChatGPT to be less engaging (22.22%, n =8)”
plainn = 8- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The single most common theme was that ChatGPT was easier to use but less rewarding (27.78%, n = 10).”
plainn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We did not observe any difference in the number of ideas generated from participants who used any creative prompting strategy (e.g. role-based, creative constraints, etc.) ( M = 9.18, SD = 3.77), compared to those who directly copied the creative ideation prompt, ( M = 8.01, SD = 3.13), t (32) = 1.016, p = 0.317, 95% CI [-3.51,.1.17].”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participant ideas generated while using ChatGPT had a greater stoplisted word count ( M = 8.25, SD = 4.61) than ideas generated while using OS, ( M = 6.46, SD = 2.55), t (32) = 2.32, p = 0.237, d = .48, 95% CI [.22,3.37]”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observed no differences for any of the CSI sub-scales ( Exploration, Engagement, Effort/Reward Tradeoff, Tool Transparency, Expressiveness ).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, as measured by our weighted flexibility score, participants using ChatGPT did not generate more unique ideas ( M = 6.34, SD = 3.31) compared to participants using OS, ( M = 5.50, SD = 3.01), t (32) = 1.424, p = 0.164, d = .26, 95% CI [-.35,2.02].”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“WhenusingChatGPT,abouttwo-thirds (63.89%, n =23) of our participants began by providing a number of their own ideas ( M =5.65, SD =3.90) before interacting with the tool at all.”
plainn = 23- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“and contrary to our expectation that participants might rely on the CST to overcome a blank page [50, 52].”
What the sources say
- Not checked yetcites [50] [50] Max Kreminski and Chris Martens. 2022. Unmet creativity support needs in computationally supported creative writing. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022) . 74-82. - source
- Not checked yetcites [52] [52] Max Kreminski and Noah Wardrip-Fruin. 2019. Generative games as storytelling partners. In Proceedings of the 14th International Conference on the Foundations of Digital Games . - source
hedgedn = 23- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A substantial fraction of participants combined both approaches, copying a few ideas directly, modifying others, and later adding more of their own ideas with no obvious connection to the ChatGPT output (38.89%, n = 14).”
plainn = 14- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A minority of participants entirely avoided direct copying, providing ideas either paraphrased from or inspired by the ChatGPT output (16.67%, n = 6).”
plainn = 6- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We did not observe any differences in Creativity Support Index (CSI) ratings for the LLM-based tool ( M =78.03%, SD =18.82%) and for OS ( M =73.98%, SD =15.35 %), t (35) = 1.028, p = 0.312, d = .24, 95% CI [-3.94%, 12.02%].”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When using ideas from ChatGPT, a slight plurality of participants copy and pasted from the output directly (41.67%, n = 15).”
plainn = 15- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observed a significant positive relationship between the number of prompts a participant entered into ChatGPT and the number of ideas that they generated, r (32)=.43, p = .008”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When interacting with ChatGPT, the majority of participants (86.11%, n =31) directly copy/pasted the creative ideation prompt for their current task.”
plainn = 31- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We did not observe a difference in the number of unique ideas (i.e., ideas that no other participant also generated) 2 between participants using ChatGPT ( M = .74, SD = .69) and OS, ( M = .97, SD = 1.00), t (32) = 1.65, p = 0.108, d = .26, 95% CI [-.05,.50].”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observed no difference in the weighted number of unique ideas, (Unusual Prompt: M = 7.98, SD = 3.12, Copied Prompt: M = 6.82, SD = 3.12), p = .278, 95% CI [-3.29,.98]”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We did not observe any difference in the number of ideas generated from participants who started by entering their own ideas first ( M = 8.38, SD = 3.39), compared to those who went directly to ChatGPT, ( M = 8.71, SD = 3.58), t (32) = .276, p = .784, 95% CI [-2.11,2.77].”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We have further clarified that the homogenization effect is group-level rather than individual-level”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When supported by ChatGPT, participants produced a larger number of ideas”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even among participants who viewed ChatGPT as an authority, they still generally suggested some ideas unlike those generated by the LLM”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also consistently outperforms all-mpnet-base-v2 , the theoretically best overall general-purpose SentenceTransformers model, by a small margin.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Human-constructed idea categories from the flexibility analysis are a valid ground truth for semantic similarity judgments.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The ability of participants to observe and work around homogenization at the individual level is backed up to some extent by the fact that a few ChatGPT users explicitly noted the repetitiveness of the model's responses.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, our analysis of prompting process (Section 4.5.4) found that-although different ChatGPT users in our study did employ different prompting strategies-there were no reliable differences in homogenization to be found between users of different approaches to prompting.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“or in the homogeneity of their ideas, between participants who took these different approaches, (Own Ideas First: M = .99, SD = .03, LLM First: M = .99, SD = .03), t (32) = .208, p = .836, 95% CI [-.01,.02].”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also observed no difference in the weighted number of unique ideas (Own Ideas First: M = 7.21, SD = 3.12, LLM First: M = 7.46, SD = 3.12), t (32) = .222, p = .826, 95% CI [-1.98,2.47]”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it therefore substantially outperforms both GloVe 840B (an aggregate word embedding model previously assessed as state-ofthe-art for creativity research [6, 26]) and the random baseline”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Human-constructed idea categories from the flexibility analysis are a valid ground truth for semantic similarity judgments.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We have presented evidence that LLM-based CSTs exert a stronger homogenization effect on human-in-the-loop divergent ideation processes than at least some plausible alternative CSTs.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and the random baseline (which GloVe itself beats by more than an order of magnitude).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Human-constructed idea categories from the flexibility analysis are a valid ground truth for semantic similarity judgments.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“or in the homogeneity of their ideas between participants who took these different approaches, (Unusual Prompt: M = .99, SD = .03, Copied Prompt: M = .99, SD = .03), t (32) = .091, p = 0.927, 95% CI [-.02,.02].”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, we found that-in line with to our expectation of LLMinduced homogenization-ideas generated with assistance from ChatGPT were significantly less semantically diverse at the group level than ideas generated with assistance from the non-AI-based CST.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it therefore substantially outperforms both GloVe 840B (an aggregate word embedding model previously assessed as state-ofthe-art for creativity research [6, 26]) and the random baseline”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Human-constructed idea categories from the flexibility analysis are a valid ground truth for semantic similarity judgments.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to Oblique Strategies users, ChatGPT users reported feeling less responsible for the ideas they produced”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In practice, however, this did not turn out to be the case.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and several mentioned their own felt sense of non-responsibility for ideas produced with ChatGPT in post-test interview responses.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, our chosen sentence embedding model ( all-MiniLM-L6-v2 ) agrees with human idea categorizations more than half the time across all four creativity tasks”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Human-constructed idea categories from the flexibility analysis are a valid ground truth for semantic similarity judgments.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but the set of ideas generated by each individual participant was similarly diverse in the ChatGPT and non-ChatGPT conditions”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In our categorization experiment, even the bestperforming embeddings model achieved only 59% agreement with human coders on average.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Human-constructed idea categories from the flexibility analysis are a valid ground truth for semantic similarity judgments.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Existing evidence seems to support the existence of homogenization effects overall, but more research is needed to gauge the severity of these effects; to confirm their existence outside of writing tasks specifically; and to clarify how they emerge.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Collectively, our results suggest that LLM-driven homogenization stems from the LLM providing different users with similar ideas, rather than by increasing individual user-level fixation”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“that users may be able to resist homogenization effects if they are given a sense of what the model tends to suggest in similar contexts”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“that low inferential distance between LLM outputs and apparently finished creative products may contribute to homogenization”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Creative design processes are subject to measurable fixation effects [21, 42, 71], through which the final proposed solutions to a design problem are inappropriately constrained by features of earlier solution candidates [2]”
What the sources say
- Not checked yetcites [21] [21] Nathan Crilly. 2019. Creativity and fixation in the real world: A literature review of case study research. Design Studies 64 (2019), 154-168. - source
- Not checked yetcites [42] [42] David G Jansson and Steven M Smith. 1991. Design fixation. Design Studies 12, 1 (1991), 3-11. - source
- Not checked yetcites [71] [71] ATerry Purcell and John S Gero. 1996. Design and other types of fixation. Design Studies 17, 4 (1996), 363-383. - source
- Not checked yetcites [2] [2] Leyla Alipour, Mohsen Faizi, Asghar Mohammad Moradi, and Gholamreza Akrami. 2018. A review of design fixation: Research directions and key factors. International Journal of Design Creativity and Innovation 6, 1-2 (2018), 22-35. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Among several potential computational means of gauging idea elaboration (which is roughly synonymous with complexity or level of detail), stoplisted word count (i.e., counting the words used to express the idea while excluding a 'stoplist' of common low-information filler words) demonstrates the strongest correlation with human judgments of elaboration [27].”
What the sources say
- Not checked yetcites [27] [27] Denis Dumas, Peter Organisciak, Shannon Maio, and Michael Doherty. 2021. Four text-mining methods for measuring elaboration. The Journal of Creative Behavior 55, 2 (2021), 517-531. - source
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The use of a single consistent AI system to perform tasks that used to be performed by many very different humans or systems can lead to increased homogenization of outcomes [10]; similar effects might be expected on creative processes if many different people all begin using ChatGPT (or another singular LLMbased system) as a CST.”
What the sources say
- Not checked yetcites [10] [10] Rishi Bommasani, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. 2022. Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? In Advances in Neural Information Processing Systems , Vol. 35. 3663-3678. -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Because uniqueness is sensitive to sample size [79], we also attempted two common alternative approaches to assessing originality: counting idea categories that less than 5% of the participants produced as unique [58], and weighting each idea category by the frequency of its occurrence in the entire sample for this study to produce a weighted flexibility score [34, 75].”
What the sources say
- Not checked yetcites [79] [79] Paul J Silvia, Beate P Winterstein, John T Willse, Christopher M Barona, Joshua T Cram, Karl I Hess, Jenna L Martinez, and Crystal A Richard. 2008. Assessing creativity with divergent thinking tasks: exploring the reliability and validity of new subjective scoring methods. Psychology of Aesthetics, Creativity, and the Arts 2, 2 (2008), 68. - source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we performed a small experiment to validate the agreement of these embeddings with human judgments of semantic similarity on our dataset.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“we investigate originality primarily by means of semantic similarity, using a well-performing sentence embedding model [72] whose direct predecessors [70] have been found to agree well with human judgments of originality in creativity research [6, 26].”
What the sources say
- Not checked yetcites [72] [72] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics. https://arxiv.org/abs/1908.10084 - source
- Not checked yetcites [70] [70] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP) . 1532-1543. http://www.aclweb.org/anthology/D141162 - source
- Not checked yetcites [6] [6] Roger E Beaty and Dan R Johnson. 2021. Automating creativity assessment with SemDis: An open platform for computing semantic distance. Behavior Research Methods 53, 2 (2021), 757-780. - source
- Not checked yetcites [26] [26] Denis Dumas, Peter Organisciak, and Michael Doherty. 2021. Measuring divergent thinking originality with human raters and text-mining models: A psychometric comparison of methods. Psychology of Aesthetics, Creativity, and the Arts 15, 4 (2021), 645. - source
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Meanwhile, Doshi and Hauser evaluate GPT-4 in the context of short-form fictional narrative writing and observe a similar homogenization effect [25].”
What the sources say
- Not checked yetcites [25] [25] Anil R Doshi and Oliver Hauser. 2023. Generative artificial intelligence enhances creativity. Available at SSRN (2023). -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We did not witness this in practice, suggesting that LLMs do not increase homogenization by causing or worsening creative fixation.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“First, we find that homogenization stems not from individual-level increases in fixation when working with the LLM, but from grouplevel suggestion of similar ideas to different users by the LLM.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and that it may be partly attributable to low inferential distance between LLM outputs and apparently finished ideas.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“a metric which has been found to agree well with human judgments of semantic similarity [6, 26].”
What the sources say
- Not checked yetcites [6] [6] Roger E Beaty and Dan R Johnson. 2021. Automating creativity assessment with SemDis: An open platform for computing semantic distance. Behavior Research Methods 53, 2 (2021), 757-780. - source
- Not checked yetcites [26] [26] Denis Dumas, Peter Organisciak, and Michael Doherty. 2021. Measuring divergent thinking originality with human raters and text-mining models: A psychometric comparison of methods. Psychology of Aesthetics, Creativity, and the Arts 15, 4 (2021), 645. - source
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A vaguely defined creative intent, expressed as (for instance) a short text string containing only a handful of words, will often tend-for level-of-detail reasons alone-to be similar to vaguely defined intents expressed by other users.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Consequently, user adoption of more sophisticated prompting strategies may not yield reliable improvements in terms of group-level homogenization without additional interventions at the level of CST design or model development.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“these results suggest that current general-purpose instructiontuned LLMs (such as ChatGPT) are capable of functioning as useful CSTs by enabling the rapid enumeration of relatively obvious possibilities that users might otherwise fail, or take longer, to consider.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, these systems are not currently well-suited to helping users develop truly original ideas.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“and changes in prompt structure can dramatically alter both the relative and absolute performance of different LLMs on a single task [60]”
What the sources say
- Not checked yetcites [60] [60] Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2023. State of What Art? A Call for Multi-Prompt LLM Evaluation. arXiv preprint arXiv:2401.00595 (2023). -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“We believe that this might be due to higher inferential distance [43] between the text on the OS cards and the expected form of the task responses”
What the sources say
- Not checked yetcites [43] [43] David Kirsh. 2011. Creative cognition in choreography. In Proceedings of the 2nd International Conference on Computational Creativity . -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, we suggest that the homogenization effect of LLMs on creative ideation is attributable in part to the low inferential distance between LLM outputs and apparently complete ideas.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“because the user has to do more work to interpret the text of a particular OS card in relation to a particular ideation prompt (compared to text generated by ChatGPT in response to that prompt), the OS deck creates more room within the overall ideation process for the user's psychological uniqueness to influence the eventual output ideas.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“suggesting that even low-engagement users may be experiencing something closer to 'inspiration through observation' [74] than to total subsumption of the ideation process by the CST.”
What the sources say
- Not checked yetcites [74] [74] Melissa Roemmele. 2021. Inspiration through observation: Demonstrating the influence of automatically generated text on creative writing. In Proceedings of the 12th International Conference on Computational Creativity (ICCC '21) . -
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“suggesting that the increase in quantity of ideas generated did not result in a commensurate increase in diversity as might be expected if diversity is a linear function of quantity [80].”
What the sources say
- Not checked yetcites [80] [80] Dean Keith Simonton. 2010. Creative thought as blind-variation and selectiveretention: Combinatorial models of exceptional creativity. Physics of Life Reviews 7, 2 (2010), 156-179. - source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“suggesting that what makes for a good LLM prompt is likely opaque to users in general.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The absence of individual-level homogenization in our study suggests that participants were able to gauge the diversity of their own responses and continue producing ideas until they felt the set of ideas they have produced is sufficiently diverse.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Brief statements of creative intent therefore yield 'samey' artifacts, which exhibit a phenomenon characterized elsewhere as the dearth of the author [45].”
What the sources say
- Not checked yetcites [45] [45] Max Kreminski. 2024. The Dearth of the Author in AI-Supported Writing. arXiv preprint arXiv:2404.10289 (2024). -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.
In the paper
“and task-specific CST design has previously shown promise for reducing conformity effects in crowdsourced divergent ideation [89].”
What the sources say
- Not checked yetcites [89] [89] Lixiu Yu, Aniket Kittur, and Robert E Kraut. 2014. Distributed analogical idea generation: inventing with crowds. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems . 1245-1254. - source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Additionally, LLM prompt design is difficult for novice users [90]”
What the sources say
- Not checked yetcites [90] [90] JD Zamfirescu-Pereira, Richmond Y Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny can't prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the fact that LLM outputs are strongly reflective of direct user inputs raises a key question for designers of LLM-based CSTs”
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For at least some users, the apparent completeness of ChatGPT responses resulted in users feeling disengaged from the process of idea generation”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, to get highly original outputs from an LLM, the user must supply the LLM with an input that contains enough detail to strongly differentiate it from other users' inputs”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Sentence embedding models remain imperfect arbiters of semantic similarity.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“suggesting that these models do not take all of the nuances of sentence meaning into account when computing similarity scores.”
What the sources say
- Not checked yetcites [66] [66] Dmitry Nikolaev and Sebastian Padó. 2023. Representation biases in sentence transformers. arXiv preprint arXiv:2301.13039 (2023). -
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the agreement between models like all-MiniLM-L6-v2 and human judgments of semantic similarity strike us as high enough to justify the use of these sentence embedding models for homogenization analysis in creativity research, in particular for the increased scale of analysis that these models enable.”
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Aggregated word embeddings are generally outperformed on semantic similarity tasks by more recent transformer-based sentence embeddings, which can take sentence structure into account.
- The authors state this as an assumption: Human-constructed idea categories from the flexibility analysis are a valid ground truth for semantic similarity judgments.
In the paper
“we consult the Massive Text Embedding Benchmark [63], which shows that the most powerful OpenAI embeddings model ( text-embedding-ada-002 ) is not clearly much better or worse than a variety of SentenceTransformers models (including all-MiniLM-L6-v2 ) across a range of tasks.”
What the sources say
- Not checked yetcites [63] [63] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive Text Embedding Benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics . Association for Computational Linguistics, 2014-2037. https://doi.org/10.18653/v1/2023.eaclmain.148 - source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The quote only partly states this claim. 1
- Probably not a substantial claim. 38
- It may state two separate claims as one. 14
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT users generated a greater number of more detailed ideas”
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
“Beyond originality, do ChatGPT and non-ChatGPT users differ in terms of other facets of creativity, such as fluency, flexibility, and elaboration? (A: ChatGPT → higher fluency, flexibility, and elaboration )”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Other prompting strategies included asking for relevant scientific or factual information (e.g. 'What are the most popular dolls?', 'How common is face blindness?', etc.), asking for responses from a specific perspective (e.g. project manager, scientist, athlete, etc.), or adding creative constraints (e.g. make this toy for a dog, respond in the form of a story, etc.).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants also made self-deprecating remarks regarding their familiarity and technical ability with ChatGPT (5.56%, n = 2), but made no such remarks regarding OS.”
plainn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The majority of these participants copied selectively: only 1 copied the entirety of the ChatGPT output without any attempt at curation or editing.”
plainn = 15- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most participants (72.22%, n = 26) also iterated on their prompt, either by adding more context to their initial prompt, probing with more specific questions or variations, or by asking ChatGPT to regenerate answers or provide additional responses.”
plainn = 26- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and that the LLM responses became too specific too quickly (5.56%, n = 2).”
plainn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The second most common theme was a positive sentiment regarding the speed and accuracy of the LLM-provided responses (25.00%, n = 9).”
plainn = 9- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“that the LLM responses were too repetitive (8.33%, n = 3)”
plainn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Coupled with evidence that ChatGPT users exhibit greater fluency, flexibility, and elaboration than users of an alternative CST”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we only observed one participant in the entire study directly copy-paste the entirety of a ChatGPT response without any further curation or modification”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“some participants in our study viewed ChatGPT as a 'very fast and accurate' authoritative source that 'gave [...] the right answers'”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Coding was conducted while blind to the CST condition (i.e., coders did not know if an idea was generated with or without LLM support).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The order of CSTs and prompts was randomized per participant and balanced across the entire experiment.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Begus [7] compares human-written to GPT-generated short stories, observing that GPTgenerated stories are less diverse than human-written stories in structure but exhibit greater gender and sexual diversity in character description.”
What the sources say
- Not checked yetcites [7] [7] Nina Begus. 2023. Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytelling. arXiv preprint arXiv:2310.12902 (2023). -
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“We did not collect responses for the Collaboration subscale, which is irrelevant and often omitted in exclusively single-user contexts like that of our study (e.g., [5, 87]).”
What the sources say
- Not checked yetcites [5] [5] Ilhan Aslan, Katharina Weitz, Ruben Schlagowski, Simon Flutura, Susana Garcia Valesco, Marius Pfeil, and Elisabeth André. 2019. Creativity support and multimodal pen-based interaction. In 2019 International Conference on Multimodal Interaction . 135-144. - source
- Not checked yetcites [87] [87] Qian Wan and Zhicong Lu. 2023. GANCollage: A GAN-Driven Digital Mood Board to Facilitate Ideation in Creativity Support. In Proceedings of the 2023 ACM Designing Interactive Systems Conference . 136-146. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Chakrabarty et al. [15] compare LLM-generated short stories to stories by expert human writers and find that the human-written stories substantially outperform the LLM-generated stories on all dimensions of creativity measured by the TTCT, including the originality dimension.”
What the sources say
- Not checked yetcites [15] [15] Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. 2023. Art or Artifice? Large Language Models and the False Promise of Creativity. arXiv preprint arXiv:2309.14556 (2023). -
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The Torrance Tests of Creative Thinking (TTCT) [85] evaluate the creativity of test-takers according to four facets of creative output: fluency , or sheer quantity of artifacts created; flexibility , or quantity of distinct categories of artifacts created; originality , or dissimilarity of created artifacts to others' creations; and elaboration , or level of detail in created artifacts.”
What the sources say
- Not checked yetcites [85] [85] E Paul Torrance. 1966. Torrance tests of creative thinking. Educational and Psychological Measurement (1966). -
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Padmakumar and He evaluate the baseline GPT-3 model [12] versus the instructionfinetuned variant InstructGPT [68] in the context of a short-form argumentative essay writing task and find a homogenization effect from LLM assistance at both the lexical and content levels, but only for the instruction-tuned LLM [69].”
What the sources say
- Not checked yetcites [69] [69] Vishakh Padmakumar and He He. 2023. Does Writing with Language Models Reduce Content Diversity? arXiv preprint arXiv:2309.05196 (2023). -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Of the 36 participants originally recruited, three were excluded from all analyses for failure to follow direction (e.g., not using ChatGPT when prompted to do so).”
plainn = 36- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Roemmele [74] proposes and provides evidence for an 'inspiration through observation' model of how human writers are influenced by LLMs: observation of LLM-generated text shapes the ideas expressed by the LLM's user even when the user does not incorporate LLM-generated text directly into their writing.”
What the sources say
- Not checked yetcites [74] [74] Melissa Roemmele. 2021. Inspiration through observation: Demonstrating the influence of automatically generated text on creative writing. In Proceedings of the 12th International Conference on Computational Creativity (ICCC '21) . -
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Arnold et al. found that 'predictive text encourages predictable writing' [4] in the context of singleword suggestions given by smartphone keyboards.”
What the sources say
- Not checked yetcites [4] [4] Kenneth C Arnold, Krysta Chauncey, and Krzysztof Z Gajos. 2020. Predictive text encourages predictable writing. In Proceedings of the 25th International Conference on Intelligent User Interfaces . 128-138. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study protocol and recruitment materials were approved by the Santa Clara University IRB.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“groups tend to be less creative than individuals when majoritarian convergence processes are used, and minority dissent within group creative processes tends to lead to greater creativity [30]”
What the sources say
- Not checked yetcites [30] [30] Umer Farooq, John M Carroll, and Craig H Canoe. 2008. Designing for creativity in computer-supported cooperative work. International Journal of e-Collaboration (IJeC) 4, 4 (2008), 51-75. - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“The Creativity Support Index (CSI) [17] is a widely used and psychometrically validated survey instrument that attempts to standardize some aspects of this experience reporting process across different CSTs.”
What the sources say
- Not checked yetcites [17] [17] Erin Cherry and Celine Latulipe. 2014. Quantifying the creativity support of digital tools through the Creativity Support Index. ACM Transactions on ComputerHuman Interaction (TOCHI) 21, 4 (2014). - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Bhat et al. [8] find that LLM-supplied next-phrase suggestions may alter the form and content of a human user's writing even when the user dislikes these suggestions.”
What the sources say
- Not checked yetcites [8] [8] Advait Bhat, Saaket Agashe, Parth Oberoi, Niharika Mohile, Ravi Jangir, and Anirudha Joshi. 2023. Interacting with Next-Phrase Suggestions: How Suggestion Systems Aid and Influence the Cognitive Processes of Writing. In Proceedings of the 28th International Conference on Intelligent User Interfaces . 436-452. - source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
In the paper
“Lee et al. [54] find that LLM support tends to increase the diversity of a user's vocabulary but may reduce their feelings of ownership for the text they produce.”
What the sources say
- Not checked yetcites [54] [54] Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a humanAI collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems . - source
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Jakesch et al. [41] find that an opinionated LLM-based CST influences the opinions expressed by its users in argumentative writing.”
What the sources say
- Not checked yetcites [41] [41] Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co-writing with opinionated language models affects users' views. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . - source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“though it may still be difficult for them to adapt their prompting strategies to elicit more diverse responses from the LLM.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For an LLM to directly transform a brief expression of intent into a substantially larger artifact (e.g., a passage of text), it will necessarily have to make a large number of creative decisions (e.g., individual word choices) on the user's behalf, and these decisions will be made by reference to the relative likelihood of different alternatives.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This may be because participants can see all of their own ideas, so they can actively modulate the diversity of the set of ideas that they have generated”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Third, we infer that LLM users might be able to resist the homogenization effect if they are given information about what kinds of stereotyped outputs the LLM tends to produce in a particular creative context”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Because our study made use of a general-purpose LLM-based chatbot (ChatGPT) rather than an LLM-based CST specifically designed to support divergent ideation”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most organic divergent ideation scenarios are not subject to these constraints.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study was conducted in a lab context and consequently may not reflect some aspects of how people use LLMs for creativity support 'in the wild'.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“additionally, the ideation prompts in our study were supplied by us, rather than being formulated independently by CST users.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In order to keep study sessions to a manageable length, participants were only allowed a fixed and relatively short amount of time to respond to each ideation prompt”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the LLM itself functions most directly as a means of refracting user-supplied inputs through the lens of statistical patterns learned from its training data.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, participants cannot easily gauge the similarity or dissimilarity of their own responses to other people's responses, so they cannot actively intervene to modulate diversity at the group level.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These findings suggest that interventions at both the AI and UI layers may help to mitigate homogenization effects in future LLM-based CSTs.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This limits the replicability of research results derived from OpenAI models and therefore the suitability of these models for research pipelines.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: OpenAI's sentence embedding models are proprietary, cost money to use, cannot be run locally, and are not open source.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Altogether, we conclude that users viewing ChatGPT as authoritative may have led them to accept LLM-suggested ideas as valid, but it did not seem to constrain the process of ideation by displacing ideas that were somehow incompatible with those suggested by ChatGPT.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Padmakumar and He have found evidence that feedbacktuned LLMs may induce creative homogenization where base (nonfeedback-tuned) LLMs do not [69]”
What the sources say
- Not checked yetcites [69] [69] Vishakh Padmakumar and He He. 2023. Does Writing with Language Models Reduce Content Diversity? arXiv preprint arXiv:2309.05196 (2023). -
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it should be noted that alternative deployments of LLMs may be able to avoid homogenizing user ideas.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, the rapid pace of progress in machine learning research means that transformerbased sentence embeddings have not yet been validated against human judgments of semantic similarity in the context of creativity research specifically.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“There also exists some evidence that cosine similarity between sentence embeddings is more strongly influenced by overlap in the set of nouns than by other similarities [66]”
What the sources say
- Not checked yetcites [66] [66] Dmitry Nikolaev and Sebastian Padó. 2023. Representation biases in sentence transformers. arXiv preprint arXiv:2301.13039 (2023). -
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The Oblique Strategies deck is a valid alternative creativity support tool for comparison with ChatGPT.
“We directed participants to a web app version of the deck [76] as an alternative CST to ChatGPT in our control condition.”
“We conducted a within-subjects experiment to evaluate the effects of using two different CSTs for idea generation: ChatGPT and the Oblique Strategies (OS) deck.”
The paper relies on this without support.
ChatGPT 3.5 is representative of LLM-based creativity support tools for generalizing homogenization effects.
“Participants in this study used the versions of ChatGPT 3.5 released on May 3rd 2023 (n=6, 16.6%) and on August 3rd, 2023 (30, 83.3%).”
“Collectively, our results suggest that LLM-driven homogenization stems from the LLM providing different users with similar ideas”
The paper relies on this without support.
The sample size of 33 participants is sufficient to detect the reported effects.
“Our sample included 33 participants, ranging in age from 22 to 44 (M=28.36, SD=6.84)”
“Of the 36 participants originally recruited, three were excluded from all analyses for failure to follow direction”
The paper relies on this without support.
An LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.
“Many of our findings follow naturally from the fact that an LLM will tend to produce similar outputs in response to similar inputs, even when those inputs come from different users.”
The paper relies on this without support.
The Oblique Strategies deck consists of a relatively small set of fixed text strings.
“The deck consists of a relatively small set of fixed text strings”
The paper relies on this without support.
Aggregated word embeddings are generally outperformed on semantic similarity tasks by more recent transformer-based sentence embeddings, which can take sentence structure into account.
“Aggregated word embeddings are generally outperformed on semantic similarity tasks by more recent transformer-based sentence embeddings, which (unlike aggregated word embeddings) are able to take sentence structure into account.”
The authors state this as an assumption.
Human-constructed idea categories from the flexibility analysis are a valid ground truth for semantic similarity judgments.
“Our experiment took the human-constructed idea categories produced by our flexibility analysis (Section 4.3.2) as a source of ground truth for semantic similarity judgments”
The paper relies on this without support.
OpenAI's sentence embedding models are proprietary, cost money to use, cannot be run locally, and are not open source.
“OpenAI models (unlike the SentenceTransformers models) are proprietary: they cost money to use, cannot be run locally, and are not open source.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“State-ofthe-art generative AI systems have reached proficiency levels that not only match human creativity in certain evaluations [20,33], but can also enhance the creative output of knowledge workers [1,2,30].”
From Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking (2025) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Emerging evidence shows measurable shifts in human writing styles, creative ideation, and divergent thinking following the widespread adoption of systems like ChatGPT [46, 6, 7, 24, 44, 79].”
From Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Other studies have found evidence of a homogenization effect (i.e., less semantically distinct outputs) from LLM assistance across a variety of GenAI text and image generation tools [8, 12, 31, 99, 139].”
From The Impact of Generative AI on Creativity in Software Development: A Research Agenda (2025) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“Moreover, most existing studies on AI's impact on creativity have focused on textual outputs using text-to-text models, like writing essays [75], ctional stories [28], or imaginative and experimental scenarios [9, 10].”
From Creativity in the Age of AI: Evaluating the Impact of Generative AI on Design Outputs and Designers' Creative Thinking (2024) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Across both experiments, we observed that LLM assistance was helpful during the exposure phase, aligning with previous research that highlights LLMs' ability to enhance creative task performance when users have access to AI-generated ideas or strategies [2,14].”
From Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking (2025) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Existing research has shown that LLM usage can lead to the homogenization of ideas within groups, where participants tend to converge on similar outcomes when using AIgenerated suggestions [2,19,45].”
From Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking (2025) · cites this paper as [2]
The cited source says what the claim attributes to it.
In the citing paper
“Literature shows that GenAI can homogenize the ideas generated by different users [8] and limit human agency and lead to creative fixation [5, 21, 101].”
From Decoupling of Usefulness and Novelty: Evaluating the Impact of Generative AI on Design Outputs and Novice Designers' Creative Thinking (2026) · cites this paper as [8]
The cited source says what the claim attributes to it.
In the citing paper
“People who use AI tend to produce more similar outputs compared to their counterparts who do not—an effect that has been shown in a range of contexts involving open-ended output, such as product design [1, 18] and short-story writing [4].”
From The Role of Human Agency in Human-AI Co-Creativity (2026) · cites this paper as [1]
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“t (32) = 2.32, p = 0.237”
Reported p = .237 · recomputed p = .027
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“r (32)=.46, p = .004”
Reported p = .004 · recomputed p = .006
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“r (32)=.43, p = .008”
Reported p = .008 · recomputed p = .011
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t (32) = 2.154, p = 0.038”
Reported p = .038 · recomputed p = .039
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upt(32) = 2.32reported p = .237 · recomputed p = .027The two fall on opposite sides of .05
“t (32) = 2.32, p = .237”
- Doesn’t add upt(32) = 2.154reported p = .038 · recomputed p = .039
“t (32) = 2.154, p = .038”
- Doesn’t add upr(32) = 0.43reported p = .008 · recomputed p = .011
“r (32)=.43, p = .008”
- Doesn’t add upr(32) = 0.46reported p = .004 · recomputed p = .006
“r (32)=.46, p = .004”
- Not confirmedt(35) = 1.028reported p = .312 · recomputed p = .311
“t (35) = 1.028, p = .312”
- Not confirmedt(32) = 0.091reported p = .927 · recomputed p = .928
“t (32) = .091, p = .927”
- Consistentt(32) = 0.944reported p = .352 · recomputed p = .352
“t (32) = .944, p = .352”
- Consistentt(32) = 3.21reported p = .003 · recomputed p = .003
“t (32) = 3.21, p = .003”
- Consistentt(32) = 2.1reported p = .044 · recomputed p = .044
“t (32) = 2.10, p = .044”
- Consistentt(32) = 3.5reported p = .001 · recomputed p = .001
“t (32) = 3.50, p = .001”
- Consistentt(32) = 1.65reported p = .108 · recomputed p = .109
“t (32) = 1.65, p = .108”
- Consistentt(32) = 0.134reported p = .894 · recomputed p = .894
“t (32) = .134, p = .894”
- Consistentt(32) = 1.424reported p = .164 · recomputed p = .164
“t (32) = 1.424, p = .164”
- Consistentr(32) = 0.23reported p = .185 · recomputed p = .191
“r (32)=.23, p = .185”
- Consistentt(32) = 1.016reported p = .317 · recomputed p = .317
“t (32) = 1.016, p = .317”
- Consistentt(32) = 0.276reported p = .784 · recomputed p = .784
“t (32) = .276, p = .784”
- Consistentt(32) = 0.222reported p = .826 · recomputed p = .826
“t (32) = .222, p = .826”
- Consistentt(32) = 0.208reported p = .836 · recomputed p = .837
“t (32) = .208, p = .836”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“including 63.63% (n=21) men and 33.36% (n=12) women”
Reported 33.36 · recomputed 36.363636
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 Retrospective Reflections · p. 7
“t (35) = 1.028, p = 0.312”
§ 3.1 Participants · p. 3
“Of the 36 participants originally recruited, three were excluded from all analyses for failure to follow direction”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4 Retrospective Reflections · p. 7
“| Low Engagement (ChatGPT) | ChatGPT allowed me to turn my brain off. It did more of the heavy lifting. | 8 | 30.56% |”
§ 4.4 Retrospective Reflections · p. 6
“ChatGPT to be less engaging (22.22%, n =8)”
Citations
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Confidence Intervals from Normalized Data: A correction to Cousineau (2005)
Morey, R. D. (2008) · Tutorials in Quantitative Methods for Psychology · doi:10.20982/tqmp.04.2.p061 · Cited by 1,991
Ambiguity as a resource for design
Gaver, W. W., Beaver, J., & Benford, S. (2003) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642653 · Cited by 1,474
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
Jansson, D. G., & Smith, S. M. (1991) · Design Studies · doi:10.1016/0142-694x(91)90003-f · Cited by 1,087
Silvia, P. J., Winterstein, B. P., Willse, J. T., Barona, C. M., Cram, J. T., Hess, K. I., Martinez, J. L., & Richard, C. A. (2008) · Psychology of Aesthetics, Creativity, and the Arts · doi:10.1037/1931-3896.2.2.68 · Cited by 912
Creativity support tools: accelerating discovery and innovation
Shneiderman, B. (2007) · Communications of the ACM · doi:10.1145/1323688.1323689 · Cited by 765
Text and Code Embeddings by Contrastive Pre-TrainingPreprint
Neelakantan, A., Xu, T., Puri, R., Radford, A., Han, J. M., Tworek, J., Yuan, Q.-M., Tezak, N., Kim, J. W., Hallacy, C., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2201.10005 · Cited by 661
AI-Mediated Communication: Definition, Research Agenda, and Ethical Considerations
Hancock, J. T., Naaman, M., & Levy, K. (2020) · Journal of Computer-Mediated Communication · doi:10.1093/jcmc/zmz022 · Cited by 658
Research works printed but not linked: 22
- Margaret A Boden. 2004. The Creative Mind: Myths and Mechanisms. Routledge.
- Rishi Bommasani Kathleen A Creel Ananya Kumar Dan Jurafsky and Percy S Liang. 2022. Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? In Advances in Neural Information Processing Systems Vol. 35. 3663–3678.
- Tom Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared D Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel Ziegler Jeffrey Wu Clemens Winter Chris Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems Vol. 33. 1877–1901.
- Daniel Buschek, Lukas Mecke, Florian Lehmann, and Hai Dang. 2021. Nine Potential Pitfalls when Designing Human-AI Co-Creative Systems. In Joint Proceedings of the ACM IUI 2021 Workshops.
- Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020. How Novelists Use Generative Language Models: An Exploratory User Study. In Workshop on Human-AI Co-Creation with Generative Models (HAI-GEN 2020).
- Kathy Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. Sage.
- John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching stories with generative pretrained language models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems.
- Anil R Doshi and Oliver Hauser. 2023. Generative artificial intelligence enhances creativity. Available at SSRN (2023).
- Ziv Epstein, Aaron Hertzmann, and Investigators of Human Creativity. 2023. Art and the science of generative AI. Science 380, 6650 (2023), 1110–1111.
- Agnes Guenther, Boris Eisenbart, and Andy Dong. 2017. Creativity as a way to innovate successfully. In DS 87-8 Proceedings of the 21st International Conference on Engineering Design (ICED 17) Vol 8: Human Behaviour in Design, Vancouver, Canada, 21-25.08. 2017. 389–398.
- Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in Psychology. Vol. 52. Elsevier, 139–183.
- Tom Hewett, Mary Czerwinski, Michael Terry, Jay Nunamaker, Linda Candy, Bill Kules, and Elisabeth Sylvan. 2005. Creativity support tool evaluation methods and metrics. In Creativity Support Tools: A workshop sponsored by the National Science Foundation. 10–24.
- David Kirsh. 2011. Creative cognition in choreography. In Proceedings of the 2nd International Conference on Computational Creativity.
- Max Kreminski and John Joon Young Chung. 2024. Intent Elicitation in Mixed-Initiative Co-Creativity. In Joint Proceedings of the ACM IUI Workshops.
- Max Kreminski, Isaac Karth, Michael Mateas, and Noah Wardrip-Fruin. 2022. Evaluating mixed-initiative creative interfaces via expressive range coverage analysis. In IUI Workshops. 34–45.
- Max Kreminski and Michael Mateas. 2021. Reflective creators. In International Conference on Computational Creativity. 309–318.
- Isabelle Levent and Lila Shroff. 2023. The Model is the Message. In The Second Workshop on Intelligent and Interactive Writing Assistants.
- Roberta M Milgram and Norman A Milgram. 1976. Creative thinking and creative performance in Israeli students.Journal of educational psychology 68, 3 (1976), 255.
- OpenAI. 2023. ChatGPT: A Large-Scale Conversational Language Model. https://www.openai.com/research/chatgpt. Accessed: August 3rd, 2023.
- Long Ouyang Jeffrey Wu Xu Jiang Diogo Almeida Carroll Wainwright Pamela Mishkin Chong Zhang Sandhini Agarwal Katarina Slama Alex Ray John Schulman Jacob Hilton Fraser Kelton Luke Miller Maddie Simens Amanda Askell Peter Welinder Paul F Christiano Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems Vol. 35. 27730–27744.
- Melissa Roemmele. 2021. Inspiration through observation: Demonstrating the influence of automatically generated text on creative writing. In Proceedings of the 12th International Conference on Computational Creativity (ICCC ’21).
- E Paul Torrance. 1966. Torrance tests of creative thinking. Educational and Psychological Measurement (1966).
Cited by
Sorted by citations
A Design Space for Intelligent and Interactive Writing Assistants
Lee, M., Gero, K. I., Chung, J. J. Y., Shum, S. B., Raheja, V., Shen, H., Venugopalan, S., Wambsganss, T., Zhou, D., Alghamdi, E. A., et al. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642697 · Cited by 136
“The most common mechanism is to use text formatting such”
Self-citationAssessing and understanding creativity in large language models
Zhao, Y., Zhang, R., Li, W., & Li, L. (2025) · Machine Intelligence Research · doi:10.1007/s11633-025-1546-4 · Cited by 57
Divergent creativity in humans and large language models
Bellemare-Pepin, A., Lespinasse, F., Thölke, P., Harel, Y., Mathewson, K. W., Olson, J. A., Bengio, Y., & Jerbi, K. (2026) · Scientific Reports · doi:10.1038/s41598-025-25157-3 · Cited by 121
Human Creativity in the Age of LLMs: Randomized Experiments on Divergent and Convergent Thinking
Kumar, H., Vincentius, J., Jordan, E., & Anderson, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714198 · Cited by 129
“Across both experiments, we observed that LLM assistance was helpful during the exposure phase, aligning with previous research that highlights LLMs’ ability to enhance creative task performance when users have access to AI-generated ideas or strategies [2, 14].”
InfluentialGenerative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization
Zhang, S., Xu, J., & Alvero, A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251327130 · Cited by 68
Metadata
- Authors
- Anderson, B. R., Shah, J. H., & Kreminski, M.
- Affiliations
- Independent Researcher, United States
- Santa Clara University (US)
- Venue
- Creativity and Cognition
- Publisher
- ACM
- Published
- 2024-06-23
- Pages
- 413-425
- OpenAlex
- W4391556516
- Cited by
- 188
- References
- 50
- Citations by year
- 2026: 97 · 2025: 79 · 2024: 12
- FWCI
- 117.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Creativity and Cognition (publishedVersion, cc-by-nc-sa)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- Scholar Commons (Santa Clara University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
- Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- large language models, creativity support tools, creative idea generation, ChatGPT, idea generation, AI-assisted creativity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
