The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
Art or Artifice? Large Language Models and the False Promise of Creativity
Open accessAbstract
Researchers have argued that large language models (LLMs) exhibit high-quality writing capabilities from blogs to stories. However, evaluating objectively the creativity of a piece of writing is challenging. Inspired by the Torrance Test of Creative Thinking (TTCT) [64], which measures creativity as a process, we use the Consensual Assessment Technique [3] and propose Torrance Test of Creative Writing (TTCW) to evaluate creativity as product. TTCW consists of 14 binary tests organized into the original dimensions of Fluency, Flexibility, Originality, and Elaboration. We recruit 10 creative writers and implement a human assessment of 48 stories written either by professional authors or LLMs using TTCW. Our analysis shows that LLM-generated stories pass 3-10X less TTCW tests than stories written by professionals. In addition, we explore the use of LLMs as assessors to automate the TTCW evaluation, revealing that none of the LLMs positively correlate with the expert assessments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“recent work evaluating LLMs' creative writing showed that they are not, after all, creative - as deemed by expert standards (Chakrabarty et al., 2024), creativity tests (Wenger & Kenett, 2025), with respect to its training data (Lu et al., 2025), or its generation of story continuations (Xu et al., 2024).”
From QUDsim: Quantifying Discourse Similarities in LLM-Generated Text (2025) · cites this paper as Chakrabarty et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“existing studies of GenAI for other types of creative work document mixed results (e.g., [22])”
From The Impact of Generative AI on Creativity in Software Development: A Research Agenda (2025) · cites this paper as [22]
The cited source says what the claim attributes to it.
In the citing paper
“For example, seasoned writers have been found to outperform (by a large margin) current LLMs (e.g., Claude, GPT-4) across a diverse range of creativity measures11.”
From Establishing the importance of co-creation and self-efficacy in creative collaboration with artificial intelligence (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“First, we begin with the observation that generative AI has meaningful limitations in its production of creative work, which suggests that publicity surrounding its creative prowess may not yet map onto reality.”
From Establishing the importance of co-creation and self-efficacy in creative collaboration with artificial intelligence (2024) · cites this paper as [11]
The cited source says what the claim attributes to it.
In the citing paper
“their weakness at generating unique and creative content becomes a major liability”
From QUDsim: Quantifying Discourse Similarities in LLM-Generated Text (2025) · cites this paper as Chakrabarty et al., 2024
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A General Inductive Approach for Analyzing Qualitative Evaluation Data
Thomas, D. R. (2006) · American Journal of Evaluation · doi:10.1177/1098214005283748 · Cited by 10,858
Postmodernism, or, The Cultural Logic of Late Capitalism
Jameson, F. (1992) · Duke University Press eBooks · doi:10.1215/9780822378419 · Cited by 6,954
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
Social psychology of creativity: A consensual assessment technique.
Amabile, T. M. (1982) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.43.5.997 · Cited by 2,278
Kittur, A., Nickerson, J. V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman, J., Lease, M., & Horton, J. (2013) · Proceedings of the 2013 conference on Computer supported cooperative work · doi:10.1145/2441776.2441923 · Cited by 1,034 · cited 1×
“Prior work has shown the potential of crowd-sourcing (through platforms such as Amazon Mechanical Turk) and the ability of non-experts to accomplish complex tasks as a crowd [38], when following an appropriate workflow that iterates and validates the work on individual non-experts.”
A cognitive process theory of writing
Flower, L., & Hayes, J. R. (1981) · College Composition & Communication · doi:10.58680/ccc198115885 · Cited by 1,025
Currie, G. (1990) · doi:10.1017/cbo9780511897498 · Cited by 975
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
Research works printed but not linked: 14
- Anthropic. 2022. Introducing Claude. (2022). https://www.anthropic.com/index/introducing-claude
- Michael M Boardman. 1992. Narrative Innovation and Incoherence: Ideology in Defoe, Goldsmith, Austen, Eliot, and Hemingway. Duke University Press.
- David Fishelov. 1990. Types of character, characteristics of types. Style (1990), 422–439.
- Nigel Fountain. 2012. Clichés: Avoid them like the plague. Michael O’Mara Books.
- 10.18653/v1 · doi:10.18653/v1
- Norman Norwood Holland. 2009. Literature and the Brain. PsyArt Foundation.
- 10.18653/v1 · doi:10.18653/v1
- Maria Kochis. 2007. Baxter, Charles. The Art of Subtext: Beyond Plot. Library Journal 132, 14 (2007), 135–136.
- Tim Mayers. 2007. (Re) Writing craft: composition, creative writing, and the future of English studies. University of Pittsburgh Pre.
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
- Ellis Paul Torrance. 1966. Torrance tests of creative thinking: Norms-technical manual: Verbal tests, forms a and b: Figural tests, forms a and b. Personal Press, Incorporated.
- 10.18653/v1 · doi:10.18653/v1
- 10.18653/v1 · doi:10.18653/v1
Cited by
Artificial intelligence and consumer behavior: From predictive to generative AI
Hermann, E., & Puntoni, S. (2024) · Journal of Business Research · doi:10.1016/j.jbusres.2024.114720 · Cited by 230
McGuire, J., De Cremer, D., & Van de Cruys, T. (2024) · Scientific Reports · doi:10.1038/s41598-024-69423-2 · Cited by 150
Safeguarding large language models: a survey
Dong, Y., Mu, R., Zhang, Y., Sun, S., Zhang, T., Wu, C., Jin, G., Qi, Y., Hu, J., Meng, J., et al. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11389-2 · Cited by 317
Self-citationMulti-Modal LLMs in Agriculture: A Comprehensive Review
Sapkota, R., Qureshi, R., Hadi, M. U., Hassan, S. Z., Sadak, F., Shoman, M., Sajjad, M., Dharejo, F. A., Paudel, A., Li, J., et al. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3612154 · Cited by 106
Multi-Modal LLMs in Agriculture: A Comprehensive Review
Sapkota, R., Qureshi, R., Hassan, S. Z., Shutske, J., Shoman, M., Sajjad, M., Dharejo, F. A., Paudel, A., Li, J., Meng, Z., et al. (2024) · doi:10.36227/techrxiv.172651082.24507804/v1 · Cited by 68
Metadata
- Authors
- Chakrabarty, T., Laban, P., Agarwal, D., Muresan, S., & Wu, C.-S.
- Affiliations
- Columbia University (US)
- Salesforce (United States) (US)
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-34
- OpenAlex
- W4396833426
- Cited by
- 135
- References
- 33
- Citations by year
- 2026: 50 · 2025: 52 · 2024: 29
- FWCI
- 38.72
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Keywords
- large language models, creativity assessment, Consensual Assessment Technique, Torrance Tests of Creative Thinking, fluency, flexibility, originality, elaboration, authorship attribution
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Art or Artifice? Large Language Models and the False Promise of Creativity. https://app.cochat.ai/papers/10.1145/3613904.3642731
- The paper
- Chakrabarty, T., Laban, P., Agarwal, D., Muresan, S., & Wu, C.-S. (2024). Art or Artifice? Large Language Models and the False Promise of Creativity. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3642731
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.