Assessing and understanding creativity in large language models
Open accessAbstract
In the field of natural language processing, the rapid development of large language model (LLM) has attracted increasing attention. LLMs have shown a high level of creativity in various tasks, but the methods for assessing such creativity are inadequate. Assessment of LLM creativity needs to consider differences from humans, requiring multiple dimensional measurement while balancing accuracy and efficiency. This paper aims to establish an efficient framework for assessing the level of creativity in LLMs. By adapting the modified Torrance tests of creative thinking, the research evaluates the creative performance of various LLMs across 7 tasks, emphasizing 4 criteria including fluency, flexibility, originality, and elaboration. In this context, we develop a comprehensive dataset of 700 questions for testing and an LLM-based evaluation method. In addition, this study presents a novel analysis of LLMs’ responses to diverse prompts and role-play situations. We found that the creativity of LLMs primarily falls short in originality, while excelling in elaboration. In addition, the use of prompts and role-play settings of the model significantly influence creativity. Additionally, the experimental results also indicate that collaboration among multiple LLMs can enhance originality. Notably, our findings reveal a consensus between human evaluations and LLMs regarding the personality traits that influence creativity. The findings underscore the significant impact of LLM design on creativity and bridge artificial intelligence and human creativity, offering insights into LLMs’ creativity and potential applications.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The lack of creativity is concretely manifested in the templatic nature of LLM-generated texts.
“This lack of creativity is concretely manifested in the templatic nature of LLM-generated texts (Spangher et al., 2024; Tian et al., 2024; Zhao et al., 2025).”
From QUDsim: Quantifying Discourse Similarities in LLM-Generated Text (2025) · cites this paper as Zhao et al., 2025
The Abstract reports that LLM creativity falls short primarily in originality, and the Introduction notes that LLM use may homogenize creativity. Neither passage establishes that LLM-generated texts are templatic; the paper also says LLMs show considerable creativity. Page numbers are unavailable in our copy. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
(1999) · Cited by 7,995 · see citing works
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 1× · see citing works
“The practical manifestations of this creativity have penetrated various sectors, including science research, where they assist in idea generation and suggestion[10]; education, by providing personalized learning experiences[11]; and in the entertainment industry, creating music and art[12, 13].”
BackgroundThe standard definition of creativity
Runco, M. A., & Jaeger, G. J. (2012) · Creativity Research Journal · doi:10.1080/10400419.2012.650092 · Cited by 3,895 · see citing works
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · cited 1× · see citing works
“The emergence of abilities from LLMs continually surpass people’s expectations, and the evaluation of various abilities of LLMs has received widespread attention[28].”
BackgroundProceedings of the 13th International Conference on Neural Information Processing Systems
Leen, T. K., Dietterich, T. G., & Tresp, V. (2000) · Cited by 2,190 · see citing works
Handbook of Personality Theory and Research.
Schwartz, M. L., Borgatta, E. F., & Lambert, W. W. (1970) · American Sociological Review · doi:10.2307/2093310 · Cited by 2,182 · see citing works
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · see citing works
Diffusion Models in Vision: A Survey
Croitoru, F.-A., Hondru, V., Ionescu, R. T., & Shah, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3261988 · Cited by 1,965 · see citing works
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · see citing works
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Proceedings for the 5th International Conference on Learning Representations
Srivastava, A., & Sutton, C. A. (2017) · Cited by 1,829 · see citing works
Measures in Health Psychology: A User's Portfolio
Johnston, M., Wright, S., & Weinman, J. A. (1995) · Cited by 1,300 · see citing works
Spreng*, R. N., McKinnon*, M. C., Mar, R. A., & Levine, B. (2009) · Journal of Personality Assessment · doi:10.1080/00223890802484381 · Cited by 1,161 · see citing works
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · see citing works
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948 · see citing works
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948 · see citing works
Models of the Creative Process: Past, Present and Future
Lubart, T. (2001) · Creativity Research Journal · doi:10.1207/s15326934crj1334_07 · Cited by 907 · see citing works
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · cited 1× · see citing works
“Similar findings can also be observed in other LLM evaluation papers[40, 41, 42].”
ResultCan We Trust Creativity Tests? A Review of the Torrance Tests of Creative Thinking (TTCT)
Kim, K. H. (2006) · Creativity Research Journal · doi:10.1207/s15326934crj1801_2 · Cited by 830 · cited 1× · see citing works
“Its ability to tap into various facets of creative thinking has made TTCT a reliable and respected tool[25].”
BackgroundBang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682 · see citing works
PaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · see citing works
New paradigms for assessing emotional intelligence: Theory and data.
MacCann, C., & Roberts, R. D. (2008) · Emotion · doi:10.1037/a0012746 · Cited by 481 · see citing works
Using large language models in psychology
Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00241-5 · Cited by 369 · see citing works
Automating creativity assessment with SemDis: An open platform for computing semantic distance
Beaty, R. E., & Johnson, D. R. (2021) · Behavior Research Methods · doi:10.3758/s13428-020-01453-w · Cited by 341 · see citing works
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.16609 · Cited by 332 · see citing works
Batey, M. (2012) · Creativity Research Journal · doi:10.1080/10400419.2012.649181 · Cited by 285 · see citing works
Divergent thinking: New methods, recent research, and extended theory.
Acar, S., & Runco, M. A. (2019) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/aca0000231 · Cited by 248 · see citing works
Piffer, D. (2012) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2012.04.009 · Cited by 228 · see citing works
The Effect Size of Variables Associated With Creativity: A Meta-Analysis
Ma, H.-H. (2009) · Creativity Research Journal · doi:10.1080/10400410802633400 · Cited by 222 · see citing works
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.149 · Cited by 211 · see citing works
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211 · see citing works
Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory
Li, S., Yu, W., Gu, T., Lin, C., Wang, Q., Qian, C., Loy, C. C., & Liu, Z. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01077 · Cited by 208 · see citing works
Collaborative Creativity—Group Creativity and Team Innovation
Paulus, P. B., Dzindolet, M. T., & Kohn, N. W. (2011) · Elsevier eBooks · doi:10.1016/b978-0-12-374714-3.00014-8 · Cited by 202 · see citing works
Homogenization Effects of Large Language Models on Human Creative Ideation
Anderson, B. R., Shah, J. H., & Kreminski, M. (2024) · Creativity and Cognition · doi:10.1145/3635636.3656204 · Cited by 188 · see citing works
Cseh, G. M., & Jeffries, K. K. (2019) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/aca0000220 · Cited by 171 · see citing works
The originality of machines: AI takes the Torrance test
Guzik, E. E., Byrge, C., & Gilde, C. (2023) · Journal of Creativity · doi:10.1016/j.yjoc.2023.100065 · Cited by 165 · see citing works
Personal factors of creativity: A second order meta-analysis
da Costa, S., Páez, D., Sánchez, F., Garaigordobil, M., & Gondim, S. M. G. (2015) · Journal of Work and Organizational Psychology · doi:10.1016/j.rpto.2015.06.002 · Cited by 162 · see citing works
Computational Creativity and Music Generation Systems: An Introduction to the State of the Art
Carnovalini, F., & Rodà, A. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00014 · Cited by 149 · see citing works
Art or Artifice? Large Language Models and the False Promise of Creativity
Chakrabarty, T., Laban, P., Agarwal, D., Muresan, S., & Wu, C.-S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642731 · Cited by 135 · see citing works
Self-assessments of creativity: Not ideal, but better than you think.
Kaufman, J. C. (2019) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/aca0000217 · Cited by 134 · see citing works
Creativity assessment in psychological research: (Re)setting the standards.
Barbot, B., Hass, R. W., & Reiter‐Palmon, R. (2019) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/aca0000233 · Cited by 134 · see citing works
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.29 · Cited by 117 · see citing works
On the creativity of large language models
Franceschelli, G., & Musolesi, M. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02127-3 · Cited by 113 · see citing works
Kim, K. H. (2011) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/a0021917 · Cited by 104 · see citing works
Guilford's Structure of Intellect Model and Model of Creativity: Contributions and Limitations
Sternberg, R. J., & Grigorenko, E. L. (2001) · Creativity Research Journal · doi:10.1207/s15326934crj1334_08 · Cited by 93 · see citing works
Creative Collaboration and Collaborative Creativity: A Systematic Literature Review
Barrett, M. S., Creech, A., & Zhukov, K. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.713445 · Cited by 91 · see citing works
Qian, M., Plucker, J. A., & Yang, X. (2019) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2019.100571 · Cited by 77 · see citing works
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., & Hashimoto, T. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1308 · Cited by 76 · see citing works
Probing the psychology of AI models
Shiffrin, R., & Mitchell, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2300963120 · Cited by 70 · see citing works
Applying Automated Originality Scoring to the Verbal Form of Torrance Tests of Creative Thinking
Acar, S., Berthiaume, K., Grajzel, K., Dumas, D., Flemister, C. “., & Organisciak, P. (2021) · Gifted Child Quarterly · doi:10.1177/00169862211061874 · Cited by 69 · see citing works
The Idea Machine: LLM-based Expansion, Rewriting, Combination, and Suggestion of Ideas
Fede, G. D., Rocchesso, D., Dow, S. P., & Andolina, S. (2022) · Creativity and Cognition · doi:10.1145/3527927.3535197 · Cited by 68 · see citing works
Autoformalization with Large Language ModelsPreprint
Wu, Y., Jiang, A. Q., Li, W., Rabe, M. N., Staats, C., Jamnik, M., & Szegedy, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.12615 · Cited by 62 · see citing works
Self-citationLiu, Y., Chen, S., Cheng, H., Yu, M., Ran, X., Mo, A., Tang, Y., & Huang, Y. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642698 · Cited by 61 · see citing works
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., & Liu, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.07201 · Cited by 55 · cited 1× · see citing works
“…from traditional human annotation to reference-based automated methods, and now, to meth-ods based on LLMs. LLMs are increasingly playing the role of judges in tasks such as question-answering, translation, and text quality assessment[35, 36], giving rise to various evaluation framework[37].”
BackgroundPutting GPT-3's Creativity to the (Alternative Uses) Test
Stevenson, C., Smal, I., Baas, M., Grasman, R., & van der Maas, H. (2022) · arXiv · doi:10.48550/arxiv.2206.08932 · Cited by 51 · see citing works
More human than human: LLM-generated narratives outperform human-LLM interleaved narratives
Zhao, Z., Song, S., Duah, B., Macbeth, J., Carter, S., Van, M. P., Bravo, N. S., Klenk, M., Sieck, K., & Filipowicz, A. L. S. (2023) · Creativity and Cognition · doi:10.1145/3591196.3596612 · Cited by 45 · see citing works
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
Wang, Y., Yu, Z., Zeng, Z., Yang, L., Wang, C., Chen, H., Jiang, C., Xie, R., Wang, J., Xie, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05087 · Cited by 34 · see citing works
Benchmarking Foundation Models with Language-Model-as-an-Examiner
Bai, Y., Ying, J., Cao, Y., Lv, X., He, Y., Wang, X., Yu, J., Zeng, K., Xiao, Y., Lyu, H., et al. (2023) · neural information processing systems · doi:10.52202/075280-3414 · Cited by 33 · see citing works
Creativity assessment: Pitfalls, solutions, and standards.
Barbot, B., & Reiter‐Palmon, R. (2019) · Psychology of Aesthetics Creativity and the Arts · doi:10.1037/aca0000251 · Cited by 24 · see citing works
A Systematic Evaluation of GPT-2-Based Music Generation
Banar, B., & Colton, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-03789-4_2 · Cited by 13 · see citing works
On the Tool Manipulation Capability of Open-source Large Language Models
Xu, Q., Hong, F., Li, B., Hu, C., Chen, Z., & Zhang, J. (2023) · arXiv · doi:10.48550/arxiv.2305.16504 · Cited by 11 · see citing works
Longitudinal Studies of Creativity
(2013) · Routledge eBooks · doi:10.4324/9780203063330 · Cited by 5 · see citing works
Judging the creative prowess of AI
Chakraborty, T., & Masud, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00664-y · Cited by 4 · see citing works
CARE-MI: Chinese Benchmark for Misinformation Evaluation in Maternity and Infant Care
Xiang, T., Li, L., Li, L., Bai, M., Wei, L., Wang, B., & García, N. (2023) · neural information processing systems · doi:10.52202/075280-1836 · Cited by 4 · see citing works
Self-citationCreative Data Generation: A Review Focusing on Text and PoetryPreprint
Elzohbi, M., & Zhao, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.08493 · Cited by 4 · see citing works
Naeini, S. A., Saqur, R., Saeidi, M., Giorgi, J., & Taati, B. (2023) · neural information processing systems · doi:10.52202/075280-0246 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Z. Li, S. Lund- berg, H. Nori, H. Palangi, M. T. Ribeiro, Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4, [Online], Available: https://arxiv.org/abs/2303. 12712, 2023.
- H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhos- ale, D. Bikel, L. Blecher, C. C. Ferrer, M. Y. Chen, G. Cu- curull, D. Esiobu, J. Fernandes, J. Fu, W. Y. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. H. Lu, Y. N. Mao, X. Mar- tinet, T. Mihaylov, P. Mishra, I. Molybog, Y. X. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. X. Xu, Z. Yan, I. Zarov, Y. C. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, T. Scialom. Llama 2: Open foundation and fine-tuned chat models, [Online], Available: https://arxiv.org/abs/2307.09288, 2023. · arXiv
- J. Devlin, M. W. Chang, L. Kenton, K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies,
- H. Shin, S. Choi, J. Y. Cho, S. Admoni, H. Lim, T. Kim, H. Hong, M. Lee, Kim, J. Towards an evaluation of LLM-gen- erated inspiration by developing and validating inspira- tion scale. In Proceedings of the 1st HEAL Workshop at CHI Conference on Human Factors in Computing Sys- tems, Honolulu, USA, 2024.
- E. P. Torrance. Torrance Test of Creative Thinking: Dir- ections Manual and Scoring Guide, Lexington, USA: Per- sonnel Press, 1966.
- D. Summers-Stay, S. Lukin, C. Voss. Brainstorm, then se- lect: A generative language model improves its creativity score. In Proceedings of AAAI Workshop on Creative AI Across Modalities, 2023.
- Ling Li received the B. Sc. degree in com- puter science and technology from Wuhan University, China in 2004, and the Ph. D. degree in computer architecture from the Institute of Computing Technology, Chinese Academy of Sciences, China in 2009. She is currently a professor at the In- stitute of Software, Chinese Academy of Sciences, China. She has authored four books (including AI Computing Systems) and over 70 papers on journals and conferences (including TIP, TC, TPDS, ICML, NeurIPS, AAAI, ISCA, MICRO). Her research interest is intelligent computing. E-mail: liling@iscas.ac.cn
Cited by
Sorted by citations
Divergent creativity in humans and large language models
Bellemare-Pepin, A., Lespinasse, F., Thölke, P., Harel, Y., Mathewson, K. W., Olson, J. A., Bengio, Y., & Jerbi, K. (2026) · Scientific Reports · doi:10.1038/s41598-025-25157-3 · Cited by 52 · see citing works
Generative Artificial Intelligence in Anatomic Pathology
Brodsky, V., Ullah, E., Bychkov, A., Song, A. H., Walk, E. E., Louis, P., Rasool, G., Singh, R. S., Mahmood, F., Bui, M. M., et al. (2025) · Archives of Pathology & Laboratory Medicine · doi:10.5858/arpa.2024-0215-ra · Cited by 48 · see citing works
A Primer for Evaluating Large Language Models in Social-Science Research
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251325174 · Cited by 57 · see citing works
RAGVA: Engineering retrieval augmented generation-based virtual assistants in practice
Yang, R., Fu, M. C., Tantithamthavorn, C. K., Arora, C., Vandenhurk, L., & Chua, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112436 · Cited by 34 · see citing works
Evaluating AI-assisted creative ideation: A crossover study in higher education
Baltà‐Salvador, R., Brasó‐Vives, E., & Peña, M. (2025) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2025.101958 · Cited by 35 · see citing works
The Impact of Generative AI on Creativity in Software Development: A Research Agenda
Jackson, V., Vasilescu, B., Russo, D., Ralph, P., Prikladnicki, R., Izadi, M., D’Angelo, S., Inman, S., Andrade, A., & van der Hoek, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708523 · Cited by 23 · see citing works
CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation
DeLorenzo, M. C., Gohil, V., & Rajendran, J. (2024) · IEEE LLM Aided Design Workshop (LAD) · doi:10.1109/lad62341.2024.10691798 · Cited by 23 · see citing works
You’ve Got AI Friend in Me: LLMs as Collaborative Learning Partners
Oppenheimer, D. M., Cash, T. N., & Pensky, A. C. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00521-6 · Cited by 18 · see citing works
Computational Hermeneutics: Evaluating Generative AI as a Cultural TechnologyPreprint
Kommers, C., Ahnert, R., Antoniak, M., Benetos, E., Benford, S., Bunz, M., Caramiaux, B., Concannon, S. J., Disley, M., Dobson, J. E., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5409144 · Cited by 10 · see citing works
Computational hermeneutics: evaluating generative AI as a cultural technology
Kommers, C., Ahnert, R., Antoniak, M., Benetos, E., Benford, S. A., Bunz, M., Caramiaux, B., Concannon, S. J., Disley, M., Dobson, J. E., et al. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1753041 · Cited by 19 · see citing works
Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons
Hu, R., Cheng, Y., Meng, L., Xia, J., Zong, Y., Shi, X., & Lin, W. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715265 · Cited by 7 · see citing works
AI delivers creative output but struggles with thinking processes
Zhang, M., Li, Y., Peng, Y., Sun, Y., Guo, W., Hu, H., Chen, S., & Zhao, Q. (2026) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2026.102196 · Cited by 10 · see citing works
Evaluating LLMs' divergent thinking capabilities for scientific idea generation with minimal context
Ruan, K., Wang, X., Hong, J., Wang, P., Liu, Y., & Sun, H. (2026) · Nature Communications · doi:10.1038/s41467-026-70245-1 · Cited by 12 · see citing works
Tütüncü, E. K., & Izzouzi, L. (2024) · IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) · doi:10.1109/ismar-adjunct64951.2024.00048 · Cited by 4 · see citing works
Zhang, J., Han, S., Weng, Y., Hu, Y., Zhang, M., Cheng, M., & Zhang, G. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106594 · Cited by 4 · see citing works
Explorando a Aplicação de Modelos de Linguagem de Grande Escala (LLMs) para o Design de Game Stories
Sarinho, V. T. (2025) · Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames) · doi:10.5753/sbgames.2025.9858 · Cited by 3 · see citing works
Rosenfelder, A., Levitin, M., & Gilead, M. (2025) · Comput. Hum. Behav. Artif. Humans · doi:10.1016/j.chbah.2025.100228 · Cited by 4 · see citing works
Semantic stability protocol: intercoder reliability for zero-shot classification via AI coders
Hsu, H.-Y., & Hsu, M. (2026) · Quality & Quantity · doi:10.1007/s11135-026-02832-9 · Cited by 3 · see citing works
Detecting Creative States From Emotional Cues: Insights From Speech With Focus on Text Modality
Kalateh, S., Nikghadam-Hojjati, S., & Barata, J. (2025) · IEEE Access · doi:10.1109/access.2025.3564499 · Cited by 3 · see citing works
Dynamic Oral English Assessment System Based on Large Language Models for Learners
Yu, J., & Burhanudeen, H. (2025) · IEEE International Conference on Social Sciences and Intelligence Management (SSIM) · doi:10.3390/engproc2025098032 · Cited by 2 · see citing works
Xu, T., Piao, Z. H., Mukai, T., Murayama, Y., & Izumi, K. (2025) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2025.200566 · Cited by 3 · see citing works
Lee, H.-S., Chang, C.-C., Chen, C.-Y., & Hsu, Y.-H. (2025) · The Electronic Library · doi:10.1108/el-04-2025-0136 · Cited by 2 · see citing works
Generative AI Can Be Creative Too
Agrawal, P., Yagnik, A. S., & Dong, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-65572-2_1 · Cited by 2 · see citing works
An Empirical Study of Production Incidents in Generative AI Cloud Services
Yan, H., Chen, Y., Ma, M., Wen, M., Lu, S., Zhang, S., Xu, T., Wang, R., Bansal, C., Rajmohan, S., et al. (2025) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre66568.2025.00044 · Cited by 2 · see citing works
Evaluating Large Language Models in Psychological Research: A Guide for ReviewersPreprint
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2024) · doi:10.31234/osf.io/ag7hy · Cited by 2 · see citing works
Peng, H., Liu, S., & Li, L. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0865-2_17 · Cited by 1 · see citing works
Self-citationCheng, Y., & Choi, I. (2025) · KNOWLEDGE ORGANIZATION · doi:10.31083/ko39046 · Cited by 1 · see citing works
Dynamic reinforcement learning for actors
Shibata, K. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107895 · Cited by 1 · see citing works
Rahimi, S., Li, H., Esmaeiligoujar, S., Ercan, D., & Botelho, A. F. (2026) · Creativity Research Journal · doi:10.1080/10400419.2026.2638384 · Cited by 1 · see citing works
🧑🍳 Cooking Up Creativity : Enhancing LLM Creativity through Structured Recombination
Mizrahi, M., Shani, C., Stanovsky, G., Jurafsky, D., & Shahaf, D. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.638 · Cited by 3 · see citing works
Hernández-Espinosa, A., Ozelim, L., Abrahão, F. S., & Zenil, H. (2026) · Nature Communications · doi:10.1038/s41467-026-73289-5 · Cited by 1 · see citing works
Measuring Group Creativity of Dialogic Interaction Systems by Means of Remote Entailment Analysis
Baumartz, D., Konca, M., Mehler, A., Schrottenbacher, P., & Braunheim, D. (2024) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/3648188.3675140 · Cited by 1 · see citing works
Cho, B. C., & Bang, J. (2024) · Journal of Broadcast Engineering · doi:10.5909/jbe.2024.29.5.596 · Cited by 1 · see citing works
Outlines of a Fuzzy Approach for Characterizing Purposefulness in Human and LLM Generated Text
Csapó, Á. (2024) · IEEE International Conference on Cognitive Infocommunications · doi:10.1109/coginfocom63007.2024.10894733 · Cited by 1 · see citing works
QUDsim: Quantifying Discourse Similarities in LLM-Generated Text
Namuduri, R., Wu, Y., Zheng, A. A., Wadhwa, M., Durrett, G., & Li, J. J. (2025) · arXiv · doi:10.48550/arxiv.2504.09373
Creating Engaging Chatbot Experiences by Tailoring Personalities in Text-Based Interactions
-, D. C., -, M. D., -, A. N., -, G. B., -, S. P., Gudadhe, M. M., & Kubade, H. M. (2024) · International Journal For Multidisciplinary Research · doi:10.36948/ijfmr.2024.v06i03.19804
RuSimulBench a Benchmark for Assessing the Quality of Language Models in Russian
Абдурахимов, М. А. У., & Khodorchenko, M. (2025) · Conference of Open Innovations Association (FRUCT) · doi:10.23919/fruct65909.2025.11008128
Autonomous Measure of Creativity in Large Language Models (LLM)
Mora-Merchán, J. M., Larios, D. F., Personal, E., & León, C. (2025) · Springer Proceedings in Materials · doi:10.1007/978-3-031-99987-1_24
Emakporuena, D., & Tunde, B. D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5418659
Lee, M.-Y., Kim, Y., Jean, S.-H., & Oh, D.-S. (2026) · Journal of Digital Contents Society · doi:10.9728/dcs.2026.27.1.13
Grammenos, D., & Lubart, T. (2026) · Research Square · doi:10.21203/rs.3.rs-8718021/v1
Wallius, E., & Lehtonen, E. (2026) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2026.101917
Exploring Creative Inflection Points: A Comparative Study of Human and GenAI Design Task Performance
Zou, W., Luo, S., & Tang, X. (2026) · Creativity Research Journal · doi:10.1080/10400419.2026.2636533
CAD score: context-aware semantic diversity for scalable creativity evaluation of LLMs
Kim, J., Jeong, G., & Kim, H. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3624
From Completion to Creation: Observing Synthesis in a Large Language ModelPreprint
Hedberg, A. (2026) · Research Square · doi:10.21203/rs.3.rs-8768251/v1
Cloud-Type Classification Using Vision-Language Model With Cloud Multimodal Adapter
Liu, S., Chu, Y., Zhang, Z., & Xiao, B. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3691532
An exploration of safety analytics for proactivity in workplace safety using large language models
Bayramova, A., Edwards, D. J., & Rillie, I. (2026) · Quality & Quantity · doi:10.1007/s11135-026-02842-7
Lemire, M., & Hudon, A. (2026) · Santé mentale au Québec · doi:10.7202/1125488ar
Countering Adversarial Attacks with Multimodal Image Fusion
Dong, Y., Zhang, C., Zhang, X., Zhang, G., & Zhao, Y. (2026) · Machine Intelligence Research · doi:10.1007/s11633-025-1613-x
Self-citationDoes GPT-5 dream of (il)literate minds? A network analysis of semantic fluency
Cinar, G., Robinson, M. N., & Gedik, T. A. (2026) · Yearbook of the German Cognitive Linguistics Association · doi:10.1515/gcla-2025-0103
Rzepka, R., Hosono, J., & Nakamura, J. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-33384-1_13
Pérez, C. R. (2026) · Digital Translation · doi:10.1075/dt.25027.ric
ijlessjournal (2026) · مجلة القانون والأعمال الدولية · doi:10.63585/ozsp2793 · Cited by 15 · see citing works
Thomas, B., Venkat, R., Ramasamy, S., & Tharuman, S. (2026) · International Conference on Educational Technology, Language and Technical Communication (ETLTC) · doi:10.3390/engproc2026143068
Extracting creativity from hallucination: Rethinking large language models
Jiang, X., Liu, Y., Shen, Y., Xu, C., Tian, Y., Chen, J., Zhai, X., Liu, Y., Hua, F., Zhao, B., et al. (2026) · AI Plus · doi:10.59717/ipj.aiplus.2026.100008
الدولية, م. ا. و. (2026) · مجلة القانون والأعمال الدولية · doi:10.63585/vlqv1532
Le, D. T. N., Ngo, A. T., Ho, D. T. T., & Trinh, K. A. (2026) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2026.102400
A Hierarchical Memory-Based Exploration Mechanism for Creative Language Model Agents
He, C., Bi, Z., Lou, J., Hu, Z., Xue, Z., Shen, Q., Liu, Z., & Zhao, K. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729301
- Authors
- Zhao, Y., Zhang, R., Li, W., & Li, L.
- Affiliations
- University of Science and Technology of China (CN)
- Chinese Academy of Sciences (CN)
- Corresponding author
- Zhao, Yunpu
- Zhang, Rui
- Li, Wenyi
- Li, Ling
- Venue
- Machine Intelligence Research
- Published
- 2025-04-28
- Volume
- 22
- Issue
- 3
- Pages
- 417-436
- ISSN
- 1476-8186 (ISSN-L)
- 2731-538X
- 2731-5398
- OpenAlex
- W4409892328
- Cited by
- 57
- References
- 64
- Citations by year
- 2026: 27 · 2025: 20 · 2024: 10
- FWCI
- 49.63
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Machine Intelligence Research (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62102399, 62302482, 61925208, 62372436, 62302480, 62222214, U22A2028, 62302483, 62302478)
- Youth Innovation Promotion Association of the Chinese Academy of Sciences
- Youth Innovation Promotion Association
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, creativity assessment, Torrance Tests of Creative Thinking, originality, fluency, flexibility, elaboration, LLM-based evaluation, prompt effects, human creativity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
