Drawing with Reframer: Emergence and Control in Co-Creative AI
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Over the past few years, rapid developments in AI have resulted in new models capable of generating high-quality images and creative artefacts, most of which seek to fully automate the process of creation. In stark contrast, creative professionals rely on iteration—to change their mind, to modify their sketches, and to re-imagine. For that reason, end-to-end generative approaches limit application to real-world design workflows. We present a novel human-AI drawing interface called Reframer, along with a new survey instrument for evaluating co-creative systems. Based on a co-creative drawing model called the Collaborative, Interactive Context-Aware Design Agent (CICADA), Reframer uses CLIP-guided synthesis-by-optimisation to support real-time synchronous drawing with AI. We present two versions of Reframer’s interface, one that prioritises emergence and system agency and the other control and user agency. To begin exploring how these different interaction models might influence the user experience, we also propose the Mixed-Initiative Creativity Support Index (MICSI). MICSI rates co-creative systems along experiential axes relevant to AI co-creation. We administer MICSI and a short qualitative interview to users who engaged with the Reframer variants on two distinct creative tasks. The results show overall broad efficacy of Reframer as a creativity support tool, but MICSI also allows us to begin unpacking the complex interactions between learning effects, task type, visibility, control, and emergent behaviour. We conclude with a discussion of how these findings highlight challenges for future co-creative systems design.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Supervised Contrastive Learning
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., & Krishnan, D. (2020) · arXiv · doi:10.48550/arxiv.2004.11362 · Cited by 3,569
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2022) · arXiv · doi:10.48550/arxiv.2208.12242 · Cited by 2,442
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295 · cited 1×
“of fine control on the user’s part [39, 42].”
BackgroundJansson, D. G., & Smith, S. M. (1991) · Design Studies · doi:10.1016/0142-694x(91)90003-f · Cited by 1,087
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Design Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501825 · Cited by 639
Design Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2021) · arXiv · doi:10.48550/arxiv.2109.06977 · Cited by 635
Quantifying the Creativity Support of Digital Tools through the Creativity Support Index
Cherry, E., & Latulipe, C. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2617588 · Cited by 498 · cited 5×
“The Creativity Support Index (CSI) [5] is a simple and flexible survey-based method for evaluating user perceptions of a creativity support tool.”
BackgroundMethodInfluentialCreating creativity: user interfaces for supporting innovation
Shneiderman, B. (2000) · ACM Transactions on Computer-Human Interaction · doi:10.1145/344949.345077 · Cited by 432
Mapping the Landscape of Creativity Support Tools in HCI
Frich, J., MacDonald Vermeulen, L., Remy, C., Biskjaer, M. M., & Dalsgaard, P. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300619 · Cited by 410 · cited 2×
“Computational creativity is a research field concerned with “creative systems”, or AI systems that enact some part of the creative process by “taking on some creative responsibility” [6, 17], either to produce creative outputs or to help us understand human creativity.”
BackgroundSuwa, M., Gero, J., & Purcell, T. (2000) · Design Studies · doi:10.1016/s0142-694x(99)00034-4 · Cited by 338 · cited 2×
“This gives the system a fluid, improvisational feeling, as users can describe new goals as they discover them, a creatively useful process known as “situated invention of requirements” [47].”
BackgroundSocially Responsible AI Algorithms: Issues, Purposes, and Challenges
Cheng, L., Varshney, K. R., & Liu, H. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12814 · Cited by 261
CLIPasso: semantically-aware object sketching
Vinker, Y., Pajouheshgar, E., Bo, J. Y., Bachmann, R. C., Bermano, A. H., Cohen-Or, D., Zamir, A., & Shamir, A. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530068 · Cited by 217
Analyzing the expressive range of a level generator
Smith, G., & Whitehead, J. (2010) · Proceedings of the 2010 Workshop on Procedural Content Generation in Games · doi:10.1145/1814256.1814260 · Cited by 204 · cited 1×
“An alternative approach for evaluating creativity support tools is assessing their capacity to generate diverse artefacts [46].”
BackgroundMixed-Initiative Creative Interfaces
Deterding, S., Hook, J., Fiebrink, R., Gillies, M., Gow, J., Akten, M., Smith, G., Liapis, A., & Compton, K. (2017) · Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/3027063.3027072 · Cited by 117 · cited 1×
“The related notion of mixed-initiative creative interfaces [13, 54] involves creativity support tools that can actively interject on the user’s process with suggestions or modifications, which brings them into conflict with the idea that creativity tools should be unobtrusive and serve to maximise…”
BackgroundDesigning Co-Creative AI for Public Spaces
Long, D., Jacob, M., & Magerko, B. (2019) · Proceedings of the 2019 on Creativity and Cognition · doi:10.1145/3325480.3325504 · Cited by 70 · cited 1×
“This challenge is even greater when the tool is intended for use outside the creative professions, such as in casual [7] or public [33] contexts.”
BackgroundDrawing Apprentice: An Enactive Co-Creative Agent for Artistic Collaboration
Davis, N., Hsiao, C.-P., Singh, K. Y., Li, L., Moningi, S., & Magerko, B. (2015) · Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition · doi:10.1145/2757226.2764555 · Cited by 67
Research works printed but not linked: 36
- Teresa M Amabile, Mary Ann Collins, Regina Conti, Elise Phillips, Martha Picariello, John Ruscio, and Dean Whitney. 2018. Creativity in Context: Update to the Social Psychology of Creativity. Routledge.
- Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473(2014). · arXiv
- Margaret A Boden. 2004. The Creative Mind: Myths and Mechanisms. Routledge.
- Simon Colton Geraint A Wiggins 2012. Computational creativity: The final frontier?. In ECAI Vol. 12. Montpelier 21–26.
- Mihaly Csikszentmihalyi. 1997. Flow and creativity.Namta Journal 22, 2 (1997), 60–97.
- 10.1007/978-94-017-9088-8_15 · DOI · Paper page
- 10.1145/2856767.2856795 · DOI · Paper page
- Nicholas Mark Davis. 2013. Human-computer co-creativity: Blending human and computational creativity. In Ninth Artificial Intelligence and Interactive Digital Entertainment Conference.
- 10.7551/mitpress/10096.001.0001 · DOI · Paper page
- Paul Dourish. 1999. Embodied interaction: Exploring the foundations of a new approach to HCI. Work (1999), 1–16.
- Kevin Frans, LB Soros, and Olaf Witkowski. 2021. Clipdraw: Exploring text-to-drawing synthesis through language-image encoders. arXiv preprint arXiv:2106.14843(2021). · arXiv
- 10.1016/j.neunet.2021.07.019 · DOI · Paper page
- 10.1609/aimag.v11i4.854 · doi:10.1609/aimag.v11i4.854
- John S Gero. 2003. Situated computing a new paradigm for design computing. (2003). · doi:10.52842/conf.caadria.2003.579
- 10.1080/21650349.2014.943295 · DOI
- Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in Psychology. Vol. 52. Elsevier, 139–183.
- Francisco Ibarrola Tomas Lawton and Kazjon Grace. 2022. A collaborative interactive and context-aware drawing agent for co-creative design. arXiv preprint arXiv:2209.12588(2022). · arXiv
- Pegah Karimi, Mary Lou Maher, Kazjon Grace, and Nicholas Davis. 2018. A computational model for visual conceptual blends. IBM Journal of Research and Development 63, 1 (2018), 5–1.
- Max Kreminski, Isaac Karth, Michael Mateas, and Noah Wardrip-Fruin. 2022. Evaluating mixed-initiative creative interfaces via expressive range coverage analysis.. In IUI Workshops. 34–45.
- 10.1145/3167476 · DOI · Paper page
- 10.1145/3414685.3417763 · DOI
- Michael Muller, Justin D Weisz, and Werner Geyer. 2020. Mixed initiative generative AI interfaces: An analytic framework for generative AI applications. In Proceedings of the Workshop The Future of Co-Creative Systems-A Workshop on Human-Computer Co-Creativity of the 11th International Conference on Computational Creativity (ICCC 2020).
- Allen Newell, J Clifford Shaw, and Herbert A Simon. 1962. The processes of creative thinking. In Contemporary Approaches to Creative Thinking,. Atherton Press.
- 10.1145/3569219.3569352 · DOI · Paper page
- 10.1016/S0954-1810(96)00047-7 · DOI
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 8748–8763.
- Aditya Ramesh Prafulla Dhariwal Alex Nichol Casey Chu and Mark Chen. 2022. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125(2022). · arXiv
- Mel Rhodes. 1961. An analysis of creativity. The Phi Delta Kappan 42, 7 (1961), 305–310.
- 10.1016/0950-7051(92)90020-G · DOI · Paper page
- Herbert A Simon. 2019. The Sciences of the Artificial. MIT press.
- C. W. Taylor. 1988. Various approaches to the definition of creativity. In The Nature of Creativity, R. J. Sternberg (Ed.). Cambridge University Press, 99–124.
- John W Tukey 1977. Exploratory Data Analysis. Vol. 2. Reading, MA.
- 10.1145/2639189.2639220 · DOI
- 10.1109/TPAMI.2016.2587640 · DOI · Paper page
- 10.1016/j.knosys.2006.04.009 · DOI · Paper page
- Yuhao Zhang Hang Jiang Yasuhide Miura Christopher D Manning and Curtis P Langlotz. 2020. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747(2020). · arXiv
Cited by
Suh, S., Chen, M., Min, B., Li, T. J.-J., & Xia, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642400 · Cited by 148
When is a Tool a Tool? User Perceptions of System Agency in Human–AI Co-Creative Drawing
Lawton, T., Grace, K., & Ibarrola, F. J. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3595977 · Cited by 72
Self-citationA User-centered Framework for Human-AI Co-creativity
Moruzzi, C., & Margarido, S. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650929 · Cited by 68
Zhang, S., Wang, H., & Yi, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757594 · Cited by 36
Metadata
- Authors
- Lawton, T., Ibarrola, F. J., Ventura, D., & Grace, K.
- Venue
- Proceedings of the 28th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2023-03-27
- Pages
- 264-277
- OpenAlex
- W4360991290
- Cited by
- 58
- References
- 18
- FWCI
- 17.47
- Type
- conference paper
- Language
- en
- Open access
- bronze
- Funders
- Australian Research Council (DP200101059)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.