CoChat
Sign Up
Cited by 72Art, Technology, and CultureDigital Media and Visual ArtMusic Technology and Sound Studies

Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts

Open access
Angert, Tyler, Suzara, Miroslav, Han, Jenny, Pondoc, Christopher, Subramonyam, Hariharan·2023·Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
DOI 10.1145/3586183.3606719
Open this paper in CoChat — free accountOpen PDF

Abstract

Creative coding tasks are often exploratory in nature. When producing digital artwork, artists usually begin with a high-level semantic construct such as a “stained glass filter” and programmatically implement it by varying code parameters such as shape, color, lines, and opacity to produce visually appealing results. Based on interviews with artists, it can be effortful to translate semantic constructs to program syntax, and current programming tools don’t lend well to rapid creative exploration. To address these challenges, we introduce Spellburst, a large language model (LLM) powered creative-coding environment. Spellburst provides (1) a node-based interface that allows artists to create generative art and explore variations through branching and merging operations, (2) expressive prompt-based interactions to engage in semantic programming, and (3) dynamic prompt-driven interfaces and direct code editing to seamlessly switch between semantic and syntactic exploration. Our evaluation with artists demonstrates Spellburst’s potential to enhance creative coding practices and inform the design of computational creativity tools that bridge semantic and syntactic spaces.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3586183.3606719

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Sketchar allows designers to prototype game characters and generate images based on conceptual input, providing visual outcomes that can give immediate feedback and enhance communication with illustrators' next step in the design cycle.

    “we created Sketchar, a Generative AI (GenAI) tool that allows designers to prototype game characters and generate images based on conceptual input, providing visual outcomes that can give immediate feedback and enhance communication with illustrators' next step in the design cycle.”

    From Sketchar: Supporting Character Design and Illustration Prototyping Using Generative AI (2024) · cites this paper as [4]

    The Introduction (section 1; page unavailable) describes visual outputs as feedback for artists’ subsequent exploration, and the User Experience section (section 4; page unavailable) describes generating sketches from natural-language prompts. The paper describes Spellburst, not Sketchar; it does not support the specific claims about prototyping game characters or improving communication with illustrators. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 39 · Cited by 72

Research works printed but not linked: 44

  1. Mary Beth Kery and Brad A. Myers. 2017. Exploring exploratory programming. In 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 25-29. https://doi.org/10.1109/VLHCC.2017.8103446 · DOI · Paper page
  2. Margaret A Boden. 2010. Creativity and art: Three roads to surprise. Oxford University Press.
  3. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77-101.
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877-1901. https://proceedings.neurips.cc/paper_files/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models Are Few-Shot Learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS'20).
  6. Cameron Burgess, Dan Lockton, Maayan Albert, and Daniel Cardoso Llach. 2020. Stamper: An Artboard-Oriented Creative Coding Environment. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA '20). Association for Computing Machinery, New York, NY, USA, 1-9. https://doi.org/10.1145/3334480.3382994 · DOI
  7. Minsuk Chang, Stefania Druga, Alex Fiannaca, Pedro Vergani, Chinmay Kulkarni, Carrie Cai, and Michael Terry. 2023. The Prompt Artists. arXiv:2303.12253 [cs.HC] · arXiv
  8. Charles Forceville. 2008. Metaphor in pictures and multimodal representations. The Cambridge handbook of metaphor and thought (2008), 462-482.
  9. Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. 2022. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion. arXiv:2208.01618 [cs.CV] · arXiv
  10. Thomas R. G. Green and Marian Petre. 1996. Usability analysis of visual pro- gramming environments: a 'cognitive dimensions' framework. Journal of Visual Languages & Computing 7, 2 (1996), 131-174.
  11. Stephen Halliwell. 2009. The aesthetics of mimesis. In The Aesthetics of Mimesis. Princeton University Press.
  12. Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psy- chology. Vol. 52. Elsevier, 139-183.
  13. Björn Hartmann, Loren Yu, Abel Allison, Yeonsoo Yang, and Scott R Klemmer. 2008. Design as exploration: creating interface alternatives through parallel authoring and runtime tuning. In Proceedings of the 21st annual ACM symposium on User interface software and technology. 91-100.
  14. Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A. Lemley, and Percy Liang. 2023. Foundation Models and Fair Use. arXiv:2303.15715 [cs.CY] · arXiv
  15. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2018. Image-to-Image Translation with Conditional Adversarial Networks. arXiv:1611.07004 [cs.CV] · arXiv
  16. Jennifer Jacobs, Joel Brandt, Radomír Mech, and Mitchel Resnick. 2018. Extending manual drawing practices with artist-centric programming tools. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1-13.
  17. Jennifer Jacobs, Sumit Gogia, Radomír Mȇch, and Joel R Brandt. 2017. Supporting expressive procedural art creation through direct manipulation. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 6330-6341.
  18. Vanita Jain, Piyush Agrawal, Subham Banga, Rishabh Kapoor, and Shashwat Gulyani. 2019. Sketch2Code: transformation of sketches to UI in real-time using deep neural network. arXiv preprint arXiv:1910.08930 (2019). · arXiv
  19. Mary Beth Kery, Amber Horvath, and Brad A Myers. 2017. Variolite: Supporting Exploratory Programming by Data Scientists.. In CHI, Vol. 10. 3025453-3025626.
  20. Kevin Gonyop Kim, Richard Lee Davis, Alessia Eletta Coppi, Alberto Cattaneo, and Pierre Dillenbourg. 2022. Mixplorer: Scaffolding Design Space Exploration through Genetic Recombination of Multiple Peoples' Designs to Support Novices' Creativity. In CHI Conference on Human Factors in Computing Systems. 1-13.
  21. Kevin Gonyop Kim, Richard Lee Davis, Alessia Eletta Coppi, Alberto Catta- neo, and Pierre Dillenbourg. 2022. Mixplorer: Scaffolding Design Space Explo- ration through Genetic Recombination of Multiple Peoples' Designs to Sup- port Novices' Creativity. In Proceedings of the 2022 CHI Conference on Hu- man Factors in Computing Systems (New Orleans, LA, USA) (CHI '22). Asso- ciation for Computing Machinery, New York, NY, USA, Article 308, 13 pages. https://doi.org/10.1145/3491102.3501854 · DOI
  22. Oded M Kleinmintz, Tal Ivancovsky, and Simone G Shamay-Tsoory. 2019. The two-fold model of creativity: the neural underpinnings of the generation and evaluation of creative ideas. Current Opinion in Behavioral Sciences 27 (2019), 131-138.
  23. Aaron D Knochel and Ryan M Patton. 2015. If art education then critical digital making: Computational thinking and creative code. Studies in Art Education 57, 1 (2015), 21-38.
  24. Chinmay Kulkarni, Stefania Druga, Minsuk Chang, Alex Fiannaca, Carrie Cai, and Michael Terry. 2023. A Word is Worth a Thousand Pictures: Prompts as AI Design Material. arXiv:2303.12647 [cs.HC] · arXiv
  25. James R Lewis. 1992. Psychometric evaluation of the post-study system usability questionnaire: The PSSUQ. In Proceedings of the human factors society annual meeting, Vol. 36. Sage Publications Sage CA: Los Angeles, CA, 1259-1260.
  26. Jingyi Li, Sonia Hashim, and Jennifer Jacobs. 2021. What We Can Learn From Visual Artists About Software Development. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1-14.
  27. John Maloney, Mitchel Resnick, Natalie Rusk, Brian Silverman, and Evelyn East- mond. 2010. The scratch programming language and environment. ACM Trans- actions on Computing Education (TOCE) 10, 4 (2010), 1-15.
  28. Hiroaki Mikami, Daisuke Sakamoto, and Takeo Igarashi. 2017. Micro-Versioning Tool to Support Experimentation in Exploratory Programming. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 6208-6219. https://doi.org/10.1145/3025453.3025597 · DOI
  29. Brad A Myers. 1990. Taxonomies of visual programming and program visualiza- tion. Journal of Visual Languages & Computing 1, 1 (1990), 97-123.
  30. Bernard A Nijstad, Wolfgang Stroebe, and Hein FM Lodewijkx. 2002. Cognitive stimulation and interference in groups: Exposure effects in an idea generation task. Journal of experimental social psychology 38, 6 (2002), 535-544.
  31. Fatih Kursat Ozenc, Miso Kim, John Zimmerman, Stephen Oney, and Brad Myers. 2010. How to Support Designers in Getting Hold of the Immaterial Material of Software. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '10). Association for Computing Machinery, New York, NY, USA, 2513-2522. https://doi.org/10.1145/1753326.1753707 · DOI
  32. Kylie Peppler and Yasmin Kafai. 2005. Creative coding: Programming for personal expression. Retrieved August 30, 2008 (2005), 314.
  33. Eric Rawn, Jingyi Li, Eric Paulos, and Sarah Chasins. 2023. Understanding Version Control as Material Interaction with Quickpose. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems.
  34. Casey Reas, Ben Fry, and John Maeda. 2007. Processing: A Programming Handbook for Visual Designers and Artists. The MIT Press.
  35. Evan Shimizu, Matthew Fisher, Sylvain Paris, James McCann, and Kayvon Fa- tahalian. 2020. Design Adjectives: A Framework for Interactive Model-Guided Exploration of Parameterized Design Spaces. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 261-278.
  36. Ben Shneiderman. 2007. Creativity support tools: accelerating discovery and innovation. Commun. ACM 50, 12 (2007), 20-32.
  37. Sarah Sterman, Molly Jane Nicholas, and Eric Paulos. 2022. Towards Creative Version Control. In CM Conference on Computer-Supported Cooperative Work and Social Computing.
  38. Kesler Tanner, Naomi Johnson, and James A. Landay. 2019. Poirot: A Web Inspector for Designers. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1-12. https://doi.org/10.1145/ 3290605.3300758 · DOI
  39. Michael Tempel. 2017. Generative art for all. Journal of Innovation and En- trepreneurship 6, 1 (2017), 1-14.
  40. Mark Turner. 2006. The artful mind: Cognitive science and the riddle of human creativity. Oxford University Press.
  41. Adrian Ward and Geoff Cox. 1999. How I Drew One of My Pictures: or, The Authorship of Generative Art. In International Conference on Generative Art. Generative Design Lab Milan.
  42. Nathaniel Weinman, Steven M. Drucker, Titus Barik, and Robert DeLine. 2021. Fork It: Supporting Stateful Alternatives in Computational Notebooks. In Pro- ceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 307, 12 pages. https://doi.org/10.1145/3411764.3445527 · DOI
  43. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 [cs.SE] · arXiv
  44. Loutfouz Zaman, Wolfgang Stuerzlinger, Christian Neugebauer, Rob Woodbury, Maher Elkhaldi, Naghmi Shireen, and Michael Terry. 2015. Gem-ni: A system for creating and managing alternatives in generative design. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 1201-1210.

Authors
Angert, T., Suzara, M., Han, J., Pondoc, C., & Subramonyam, H.
Affiliations
  • Replit, Inc., United States
  • Stanford University (US)
ORCID
Venue
Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology
Publisher
ACM
Published
2023-10-29
Pages
1-22
DOI
10.1145/3586183.3606719
OpenAlex
W4385714563
Cited by
72
References
39
Citations by year
2026: 26 · 2025: 31 · 2024: 13 · 2023: 2
FWCI
195.82
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Arts and Humanities › Visual Arts and Performing Arts › Art, Technology, and Culture
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Digital Media and Visual Art
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Music Technology and Sound Studies
Keywords
creative coding, generative art, large language models, natural language prompts, computational creativity, code editing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review