CoChat

Drawing with Reframer: Emergence and Control in Co-Creative AI

Open access
Lawton, Tomas, Ibarrola, Francisco J, Ventura, Dan, Grace, Kazjon · 2023 · Proceedings of the 28th International Conference on Intelligent User Interfaces
Cited by 58

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Abstract

Over the past few years, rapid developments in AI have resulted in new models capable of generating high-quality images and creative artefacts, most of which seek to fully automate the process of creation. In stark contrast, creative professionals rely on iteration—to change their mind, to modify their sketches, and to re-imagine. For that reason, end-to-end generative approaches limit application to real-world design workflows. We present a novel human-AI drawing interface called Reframer, along with a new survey instrument for evaluating co-creative systems. Based on a co-creative drawing model called the Collaborative, Interactive Context-Aware Design Agent (CICADA), Reframer uses CLIP-guided synthesis-by-optimisation to support real-time synchronous drawing with AI. We present two versions of Reframer’s interface, one that prioritises emergence and system agency and the other control and user agency. To begin exploring how these different interaction models might influence the user experience, we also propose the Mixed-Initiative Creativity Support Index (MICSI). MICSI rates co-creative systems along experiential axes relevant to AI co-creation. We administer MICSI and a short qualitative interview to users who engaged with the Reframer variants on two distinct creative tasks. The results show overall broad efficacy of Reframer as a creativity support tool, but MICSI also allows us to begin unpacking the complex interactions between learning effects, task type, visibility, control, and emergent behaviour. We conclude with a discussion of how these findings highlight challenges for future co-creative systems design.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3581641.3584095

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 36

  1. Teresa M Amabile, Mary Ann Collins, Regina Conti, Elise Phillips, Martha Picariello, John Ruscio, and Dean Whitney. 2018. Creativity in Context: Update to the Social Psychology of Creativity. Routledge.
  2. Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473(2014). · arXiv
  3. Margaret A Boden. 2004. The Creative Mind: Myths and Mechanisms. Routledge.
  4. Simon Colton Geraint A Wiggins 2012. Computational creativity: The final frontier?. In ECAI Vol. 12. Montpelier 21–26.
  5. Mihaly Csikszentmihalyi. 1997. Flow and creativity.Namta Journal 22, 2 (1997), 60–97.
  6. 10.1007/978-94-017-9088-8_15 · DOI · Paper page
  7. 10.1145/2856767.2856795 · DOI · Paper page
  8. Nicholas Mark Davis. 2013. Human-computer co-creativity: Blending human and computational creativity. In Ninth Artificial Intelligence and Interactive Digital Entertainment Conference.
  9. 10.7551/mitpress/10096.001.0001 · DOI · Paper page
  10. Paul Dourish. 1999. Embodied interaction: Exploring the foundations of a new approach to HCI. Work (1999), 1–16.
  11. Kevin Frans, LB Soros, and Olaf Witkowski. 2021. Clipdraw: Exploring text-to-drawing synthesis through language-image encoders. arXiv preprint arXiv:2106.14843(2021). · arXiv
  12. 10.1016/j.neunet.2021.07.019 · DOI · Paper page
  13. 10.1609/aimag.v11i4.854 · doi:10.1609/aimag.v11i4.854
  14. John S Gero. 2003. Situated computing a new paradigm for design computing. (2003). · doi:10.52842/conf.caadria.2003.579
  15. 10.1080/21650349.2014.943295 · DOI
  16. Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in Psychology. Vol. 52. Elsevier, 139–183.
  17. Francisco Ibarrola Tomas Lawton and Kazjon Grace. 2022. A collaborative interactive and context-aware drawing agent for co-creative design. arXiv preprint arXiv:2209.12588(2022). · arXiv
  18. Pegah Karimi, Mary Lou Maher, Kazjon Grace, and Nicholas Davis. 2018. A computational model for visual conceptual blends. IBM Journal of Research and Development 63, 1 (2018), 5–1.
  19. Max Kreminski, Isaac Karth, Michael Mateas, and Noah Wardrip-Fruin. 2022. Evaluating mixed-initiative creative interfaces via expressive range coverage analysis.. In IUI Workshops. 34–45.
  20. 10.1145/3167476 · DOI · Paper page
  21. 10.1145/3414685.3417763 · DOI
  22. Michael Muller, Justin D Weisz, and Werner Geyer. 2020. Mixed initiative generative AI interfaces: An analytic framework for generative AI applications. In Proceedings of the Workshop The Future of Co-Creative Systems-A Workshop on Human-Computer Co-Creativity of the 11th International Conference on Computational Creativity (ICCC 2020).
  23. Allen Newell, J Clifford Shaw, and Herbert A Simon. 1962. The processes of creative thinking. In Contemporary Approaches to Creative Thinking,. Atherton Press.
  24. 10.1145/3569219.3569352 · DOI · Paper page
  25. 10.1016/S0954-1810(96)00047-7 · DOI
  26. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748–8763.
  27. Aditya Ramesh Prafulla Dhariwal Alex Nichol Casey Chu and Mark Chen. 2022. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125(2022). · arXiv
  28. Mel Rhodes. 1961. An analysis of creativity. The Phi Delta Kappan 42, 7 (1961), 305–310.
  29. 10.1016/0950-7051(92)90020-G · DOI · Paper page
  30. Herbert A Simon. 2019. The Sciences of the Artificial. MIT press.
  31. C. W. Taylor. 1988. Various approaches to the definition of creativity. In The Nature of Creativity, R. J. Sternberg (Ed.). Cambridge University Press, 99–124.
  32. John W Tukey 1977. Exploratory Data Analysis. Vol. 2. Reading, MA.
  33. 10.1145/2639189.2639220 · DOI
  34. 10.1109/TPAMI.2016.2587640 · DOI · Paper page
  35. 10.1016/j.knosys.2006.04.009 · DOI · Paper page
  36. Yuhao Zhang Hang Jiang Yasuhide Miura Christopher D Manning and Curtis P Langlotz. 2020. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747(2020). · arXiv

Cited by

See all 58

Metadata

Authors
Lawton, T., Ibarrola, F. J., Ventura, D., & Grace, K.
Venue
Proceedings of the 28th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2023-03-27
Pages
264-277
OpenAlex
W4360991290
Cited by
58
References
18
FWCI
17.47
Type
conference paper
Language
en
Open access
bronze
Funders
Australian Research Council (DP200101059)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.