CoChat
Sign Up
PreprintCited by 51Creativity in Education and NeuroscienceTopic ModelingArtificial Intelligence Applications

Putting GPT-3's Creativity to the (Alternative Uses) Test

Open access
Stevenson, Claire, Smal, Iris, Baas, Matthijs, Grasman, Raoul, van der Maas, Han·2022·arXiv
DOI 10.48550/arxiv.2206.08932arXiv 2206.08932
Open this paper in CoChat — free accountOpen PDF

Abstract

AI large language models have (co-)produced amazing written works from newspaper articles to novels and poetry. These works meet the standards of the standard definition of creativity: being original and useful, and sometimes even the additional element of surprise. But can a large language model designed to predict the next text fragment provide creative, out-of-the-box, responses that still solve the problem at hand? We put Open AI's generative natural language model, GPT-3, to the test. Can it provide creative solutions to one of the most commonly used tests in creativity research? We assessed GPT-3's creativity on Guilford's Alternative Uses Test and compared its performance to previously collected human responses on expert ratings of originality, usefulness and surprise of responses, flexibility of each set of ideas as well as an automated method to measure creativity based on the semantic distance between a response and the AUT object in question. Our results show that -- on the whole -- humans currently outperform GPT-3 when it comes to creative output. But, we believe it is only a matter of time before GPT-3 catches up on this particular task. We discuss what this work reveals about human and AI creativity, creativity testing and our definition of creativity.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2206.08932

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The survey questions were slightly adapted from Yuan et al. [125] and Stevenson et al. [108].

    “These questions are slightly adapted from Yuan et al. [125] and Stevenson et al. [108]”

    From Co-Writing Screenplays and Theatre Scripts with Language Models: Evaluation by Industry Professionals (2023) · cites this paper as Stevenson et al. [108]

    The Alternative Uses Task sections (p. 2) provide a question posed to GPT-3 and instructions given to human participants, so the paper contains wording that could be adapted into questions. Our copy does not show the citing document’s survey questions, however, and cannot establish that they were only slightly adapted from this wording. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 19

  1. Baer, J. and McKool, S.S., 2009. Assessing creativity using the consensual assessment technique. In Handbook of research on assessment technologies, methods, and applications in higher education (pp. 65-77). IGI Global.
  2. Beaty, R.E. and Johnson, D.R., 2021. Automating creativity assessment with SemDis: An open platform for computing semantic distance. Behavior research methods , 53 (2):757780.
  3. Bender, E.M., Gebru, T., McMillan-Major, A. and Mitchell, S., 2021, March. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 610-623. https://doi.org/10.1145/3442188. 3445922 · DOI · Paper page
  4. Boden, M. 2004. The creative mind: Myths and mechanisms. Routledge.
  5. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. and Agarwal, S., 2020. Language models are fewshot learners. Advances in neural information processing systems , 33 , pp.1877-1901. https://arxiv.org/abs/2005. 14165
  6. Diedrich, J., Benedek, M., Jauk, E. and Neubauer, A.C., 2015. Are creative ideas novel and useful?. Psychology of Aesthetics, Creativity, and the Arts, 9 (1), p.35-40. https://doi.org/10.1037/a0038688 · DOI · Paper page
  7. Guilford, J.P., 1967. Creativity: Yesterday, today and tomorrow. The Journal of Creative Behavior , 1 (1):3-14.
  8. Hass, R.W., 2017. Semantic search during divergent thinking. Cognition, 166 , pp.344-357. https://doi.org/ 10.1016/j.cognition.2017.05.039 · DOI
  9. Mednick, S.A., 1968. The remote associates test. The Journal of Creative Behavior. 2 :213-214.
  10. Mitchell, M., 2021. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 1505 (1):79-101. https://doi.org/10.1111/nyas. 14619 · DOI
  11. Nijstad, B.A., De Dreu, C.K., Rietzschel, E.F. and Baas, M., 2010. The dual pathway to creativity model: Creative ideation as a function of flexibility and persistence. European review of social psychology, 21 (1), pp.34-77. https://doi.org/10.1080/10463281003765323 · DOI
  12. Rietzschel, E. F., Nijstad, B. A., and Stroebe, W. 2019. Why great ideas are often overlooked. The Oxford handbook of group creativity and innovation, 179-197.
  13. Runco, M.A., 2008. Commentary: Divergent thinking is not synonymous with creativity. Psychology of Aesthetics, Creativity, and the Arts, 2 (2):93-96. https://doi.org/10. 1037/1931-3896.2.2.93
  14. Runco, M.A. and Jaeger, G.J., 2012. The standard definition of creativity. Creativity research journal , 24 (1):92-96. https://doi.org/10.1080/10400419.2012.650092 · DOI · Paper page
  15. Silvia, P.J., Martin, C. and Nusbaum, E.C., 2009. A snapshot of creativity: Evaluating a quick and simple method for assessing divergent thinking. Thinking Skills and Creativity, 4 (2):79-85. https://doi.org/10.1016/j.tsc.2009. 06.005 · DOI
  16. Simonton, D.K., 2018. Defining creativity: Don't we also need to define what is not creative?. The Journal of Creative Behavior, 52(1), pp.80-90. https://doi.org/10.1002/jocb.137 · DOI
  17. Sternberg, R.J. and Davidson, J.E., 1982. The mind of the puzzler. Psychology Today , 16 (6):37-44.
  18. Stevenson, C.E., Baas, M. and van der Maas, H., 2021. A minimal theory of creative ability. Journal of Intelligence , 9 (1):9. https://doi.org/10.3390/jintelligence9010009 · DOI
  19. van der Maas, H.L., Snoek, L. and Stevenson, C.E., 2021. How much intelligence is there in artificial intelligence? A 2020 update. Intelligence , 87 :101548. https://doi.org/ 10.1016/j.intell.2021.101548 · DOI

Cited by

Sorted by citations

Authors
Stevenson, C., Smal, I., Baas, M., Grasman, R., & van der Maas, H.
Affiliations
University of Amsterdam (NL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2206.08932
arXiv
2206.08932
OpenAlex
W4283321795
Cited by
51
References
0
Citations by year
2026: 6 · 2025: 10 · 2024: 16 · 2023: 18 · 2022: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • UvA-DARE (University of Amsterdam) (submittedVersion, cc-by)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
Jacobs Foundation
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Creativity in Education and Neuroscience
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
GPT-3, large language models, computational creativity, human creativity, Alternative Uses Test, originality, usefulness, surprise, semantic distance, creativity assessment, human-AI comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review