CoChat
Sign Up
Cited by 16Design Education and PracticeInnovative Human-Technology InteractionMusic Technology and Sound Studies

Compositional Structures as Substrates for Human-AI Co-creation Environment: A Design Approach and A Case Study

Open access
Cao, Yining, Huang, Yiyi, Truong, Anh, Shin, Hijung Valentina, Xia, Haijun·2025·Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3706598.3713401
Open this paper in CoChat — free account

Abstract

It has been increasingly recognized that effective human-AI co-creation requires more than prompts and results, but an environment with empowering structures that facilitate exploration, planning, iteration, as well as control and inspection of AI generation. Yet, a concrete design approach to such an environment has not been established. Our literature analysis highlights that compositional structures-which organize and visualize individual elements into meaningful wholes-are highly effective in granting creators control over the essential aspects of their content. However, efficiently aggregating and connecting these structures to support the full creation process remains challenging. Therefore, we propose a design approach of leveraging compositional structures as the substrates and infusing AI within and across these structures to enable a controlled and fluid creation process. We evaluate this approach through a case study of developing a video co-creation environment using this approach. User evaluation shows that such an environment allowed users to stay oriented in their creation activity, remain aware and in control of AI's generation, and enable flexible human-AI collaborative workflows.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3706598.3713401

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Human designers often need to deconstruct and recompose AI outputs to meet professional standards.

    “As highlighted by Cao et al. [1], human designers often need to deconstruct and recompose AI outputs to meet professional standards.”

    From Human-AI Co-Creation in Visual Communication Design: A Framework and Empirical Study (2026) · cites this paper as [1]

    Section 7.1 (p. 12) reports that creators used compositional structures to understand AI-generated content, spot mismatches, and identify where to make changes; the Introduction (p. 2) likewise emphasizes iterative refinement beyond prompting. The paper does not establish that designers often need to deconstruct and recompose outputs specifically to meet professional standards. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 91

  1. [3][3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). - · arXiv
  2. [2][2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). - · arXiv
  3. [7][7] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for HumanAI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19) . Association for Computing Machinery, New York, NY, USA, 1-13. doi:10.1145/3290605.3300233 - · DOI · Paper page
  4. [11][11] Michel Beaudouin-Lafon. 2017. Towards Unified Principles of Interaction. In Proceedings of the 12th Biannual Conference on Italian SIGCHI Chapter (Cagliari, Italy) (CHItaly '17) . Association for Computing Machinery, New York, NY, USA, Article 1, 2 pages. doi:10.1145/3125571.3125602 - · DOI
  5. [10][10] Benjamin Bach, Zezhong Wang, Matteo Farinella, Dave Murray-Rust, and Nathalie Henry Riche. 2018. Design Patterns for Data Comics. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18) . Association for Computing Machinery, New York, NY, USA, 1-12. doi:10.1145/3173574.3173612 - · DOI
  6. [12][12] Floraine Berthouzoz, Wilmot Li, and Maneesh Agrawala. 2012. Tools for placing cuts and transitions in interview video. ACM Trans. Graph. 31, 4, Article 67 (jul 2012), 8 pages. doi:10.1145/2185520.2185563 - · DOI
  7. [14][14] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). - · arXiv
  8. [13][13] Susanne Bodker. 2021. Through the interface: A human activity approach to user interface design . CRC Press. -
  9. [15][15] Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman. 2023. Promptify: Text-to-image generation through interactive prompt exploration with large language models. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology . 1-14. -
  10. [17][17] Daniel Buschek. 2024. Collage is the New Writing: Exploring the Fragmentation of Text and User Interfaces in AI Tools. In Proceedings of the 2024 ACM Designing Interactive Systems Conference (Copenhagen, Denmark) (DIS '24) . Association for Computing Machinery, New York, NY, USA, 2719-2737. doi:10.1145/3643834. 3660681 - · DOI
  11. [18][18] Yining Cao, Jane L E, Zhutian Chen, and Haijun Xia. 2023. Dataparticles: Block-based and language-oriented authoring of animated unit visualizations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1-15. -
  12. [20][20] Vincent Cavez, Catherine Letondal, Emmanuel Pietriga, and Caroline Appert. 2024. Challenges of Music Score Writing and the Potentials of Interactive Surfaces (CHI '24) . Association for Computing Machinery, New York, NY, USA, Article 728, 16 pages. doi:10.1145/3613904.3642079 - · DOI
  13. [19][19] Yining Cao, Rubaiat Habib Kazi, Li-Yi Wei, Deepali Aneja, and Haijun Xia. 2024. Elastica: Adaptive Live Augmented Presentations with Elastic Mappings Across Modalities. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1-19. -
  14. [21][21] Zhutian Chen and Haijun Xia. 2022. CrossData: Leveraging Text-Data Connections for Authoring Data Documents. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22) . Association for Computing Machinery, New York, NY, USA, Article 95, 15 pages. doi:10.1145/3491102.3517485 - · DOI
  15. [23][23] Peggy Chi, Zheng Sun, Katrina Panovich, and Irfan Essa. 2020. Automatic video creation from a web page. In Proceedings of the 33rd annual ACM symposium on user interface software and technology . 279-292. -
  16. [22][22] Peggy Chi, Tao Dong, Christian Frueh, Brian Colonna, Vivek Kwatra, and Irfan Essa. 2022. Synthesis-Assisted Video Prototyping From a Document. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology . 1-10. -
  17. [26][26] Hai Dang, Karim Benharrak, Florian Lehmann, and Daniel Buschek. 2022. Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summaries. In Proceedings of the 35th Annual ACM Symposium on User Interface -
  18. [25][25] Matthew Conlen, Megan Vo, Alan Tan, and Jeffrey Heer. 2021. Idyll Studio: A Structured Editor for Authoring Interactive & Data-Driven Articles. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '21) . Association for Computing Machinery, New York, NY, USA, 1-12. doi:10.1145/3472749.3474731 - Software and Technology (Bend, OR, USA) (UIST '22) . Association for Computing Machinery, New York, NY, USA, Article 98, 13 pages. doi:10.1145/3526113. 3545672 - · DOI
  19. [27][27] Abe Davis and Maneesh Agrawala. 2018. Visual rhythm and beat. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1-11. -
  20. [29][29] Steven Dow, Julie Fortuna, Dan Schwartz, Beth Altringer, Daniel Schwartz, and Scott Klemmer. 2011. Prototyping dynamics: sharing multiple designs improves exploration, group rapport, and results. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI '11) . Association for Computing Machinery, New York, NY, USA, 2807-2816. doi:10.1145/1978942.1979359 - · DOI · Paper page
  21. [31][31] Marianela Ciolfi Felice, Nolwenn Maudet, Wendy E. Mackay, and Michel Beaudouin-Lafon. 2016. Beyond Snapping: Persistent, Tweakable Alignment and Distribution with StickyLines. In Symposium on User Interface Software and Technology (UIST) . -
  22. [30][30] Steven P. Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L. Schwartz, and Scott R. Klemmer. 2011. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Trans. Comput.-Hum. Interact. 17, 4, Article 18 (dec 2011), 24 pages. doi:10.1145/1879831.1879836 - · DOI · Paper page
  23. [34][34] Amy Rae Fox, Philip Guo, Clemens Nylandsted Klokmose, Peter Dalsgaard, Arvind Satyanarayan, Haijun Xia, and James D. Hollan. 2020. Towards a dynamic multiscale personal information space: beyond application and document centered views of information. In Companion Proceedings of the 4th International Conference on Art, Science, and Engineering of Programming (Porto, Portugal) (Programming '20) . Association for Computing Machinery, New York, NY, USA, 136-143. doi:10.1145/3397537.3397542 - · DOI
  24. [35][35] Amy Rae Fox, Philip Guo, Clemens Nylandsted Klokmose, Peter Dalsgaard, Arvind Satyanarayan, Haijun Xia, and James D. Hollan. 2020. Towards a dynamic multiscale personal information space: beyond application and document centered views of information. In Companion Proceedings of the 4th International Conference on Art, Science, and Engineering of Programming (Porto, Portugal) (Programming '20) . Association for Computing Machinery, New York, NY, USA, 136-143. doi:10.1145/3397537.3397542 - · DOI
  25. [37][37] Jérémie Garcia, Louis Bigo, Antoine Spicher, and Wendy E. Mackay. 2013. PaperTonnetz: supporting music composition with interactive paper. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (Paris, France) (CHI EA '13) . Association for Computing Machinery, New York, NY, USA, 3051-3054. doi:10.1145/2468356.2479608 - · DOI
  26. [36][36] Mahak Gambhir and Vishal Gupta. 2017. Recent automatic text summarization techniques: a survey. Artificial Intelligence Review 47, 1 (2017), 1-66. -
  27. [38][38] Jérémie Garcia, Theophanis Tsandilas, Carlos Agon, and Wendy Mackay. 2012. Interactive paper substrates to support musical creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI '12) . Association for Computing Machinery, New York, NY, USA, 1825-1828. doi:10.1145/2207676.2208316 - · DOI
  28. [40][40] Han L. Han, Junhang Yu, Raphael Bournet, Alexandre Ciorascu, Wendy E. Mackay, and Michel Beaudouin-Lafon. 2022. Passages: Interacting with Text Across Documents. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI '22) . Association for Computing Machinery, New York, NY, USA, Article 338, 17 pages. doi:10.1145/3491102.3502052 - · DOI
  29. [39][39] Jérémie Garcia, Theophanis Tsandilas, Carlos Agon, and Wendy E. Mackay. 2014. Structured observation with polyphony: a multifaceted tool for studying music composition. In Proceedings of the 2014 Conference on Designing Interactive Systems (Vancouver, BC, Canada) (DIS '14) . Association for Computing Machinery, New York, NY, USA, 199-208. doi:10.1145/2598510.2598512 - · DOI
  30. [41][41] John Hart. 2013. The Art of the Storyboard: A filmmaker's introduction . Routledge. -
  31. [43][43] Rorik Henrikson, Bruno Araujo, Fanny Chevalier, Karan Singh, and Ravin Balakrishnan. 2016. Multi-Device Storyboards for Cinematic Narratives in VR. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST '16) . Association for Computing Machinery, New York, NY, USA, 787-796. doi:10.1145/2984511.2984539 - · DOI
  32. [42][42] Jeffrey Heer, Matthew Conlen, Vishal Devireddy, Tu Nguyen, and Joshua Horowitz. 2023. Living Papers: A Language Toolkit for Augmented Scholarly Communication. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST '23) . Association for Computing Machinery, New York, NY, USA, Article 42, 13 pages. doi:10.1145/3586183.3606791 - · DOI
  33. [44][44] Rorik Henrikson, Bruno De Araujo, Fanny Chevalier, Karan Singh, and Ravin Balakrishnan. 2016. Storeoboard: Sketching Stereoscopic Storyboards. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16) . Association for Computing Machinery, New York, NY, USA, 4587-4598. doi:10.1145/2858036.2858079 - · DOI
  34. [45][45] Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI '99) . Association for Computing Machinery, New York, NY, USA, 159-166. doi:10.1145/302979.303030 - · DOI · Paper page
  35. [47][47] Edwin L Hutchins, James D Hollan, and Donald A Norman. 1985. Direct manipulation interfaces. Human-computer interaction 1, 4 (1985), 311-338. -
  36. [46][46] Bernd Huber, Hijung Valentina Shin, Bryan Russell, Oliver Wang, and Gautham J Mysore. 2019. B-script: Transcript-based b-roll video editing with recommendations. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems . 1-11. -
  37. [48][48] Peiling Jiang, Jude Rayan, Steven P. Dow, and Haijun Xia. 2023. Graphologue: Exploring Large Language Model Responses with Interactive Diagrams. In Symposium on User Interface Software and Technology (UIST) . https://doi.org/10. 1145/3586183.3606737 -
  38. [49][49] DaYe Kang, Tony Ho, Nicolai Marquardt, Bilge Mutlu, and Andrea Bianchi. 2021. ToonNote: Improving Communication in Computational Notebooks Using Interactive Data Comics. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 727, 14 pages. doi:10. 1145/3411764.3445434 -
  39. [51][51] Nam Wook Kim, Nathalie Henry Riche, Benjamin Bach, Guanpeng Xu, Matthew Brehmer, Ken Hinckley, Michel Pahud, Haijun Xia, Michael J. McGuffin, and Hanspeter Pfister. 2019. DataToon: Drawing Dynamic Network Comics With Pen + Touch Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19) . Association for Computing Machinery, New York, NY, USA, 1-12. doi:10.1145/3290605.3300335 - · DOI
  40. [52][52] Cynthia L King. 2012. Reverse outlining: A method for effective revision of document structure. IEEE Transactions on Professional Communication 55, 3 (2012), 254-261. -
  41. [54][54] Clemens N Klokmose, James R Eagan, Siemen Baader, Wendy Mackay, and Michel Beaudouin-Lafon. 2015. Webstrates: shareable dynamic media. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology . 280-290. -
  42. [56][56] Joseph CR Licklider. 1960. Man-computer symbiosis. IRE transactions on human factors in electronics 1 (1960), 4-11. -
  43. [55][55] Wilmot Li, Maneesh Agrawala, Brian Curless, and David Salesin. 2008. Automated generation of interactive 3D exploded view diagrams. ACM Trans. Graph. 27, 3 (aug 2008), 1-7. doi:10.1145/1360612.1360700 - · DOI
  44. [57][57] Zhicong Lu, Mingming Fan, Yun Wang, Jian Zhao, Michelle Annett, and Daniel Wigdor. 2019. InkPlanner: Supporting Prewriting via Intelligent Visual Diagramming. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2019), 277-287. doi:10.1109/TVCG.2018.2864887 - · DOI
  45. [59][59] Nolwenn Maudet, Ghita Jalal, Philip Tchernavskij, Michel Beaudouin-Lafon, and Wendy E. Mackay. 2017. Beyond Grids: Interactive Graphical Substrates to Structure Digital Layout. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17) . Association for Computing Machinery, New York, NY, USA, 5053-5064. doi:10.1145/3025453. 3025718 - · DOI
  46. [58][58] W. Mackay and D. Pagani. 1994. Video mosaic: laying out time in a physical space (MULTIMEDIA '94) . Association for Computing Machinery, New York, NY, USA, 165-172. doi:10.1145/192593.192646 - · DOI
  47. [63][63] Sharon Oviatt. 1999. Ten myths of multimodal interaction. Commun. ACM 42, 11 (nov 1999), 74-81. doi:10.1145/319382.319398 - · DOI
  48. [65][65] Amy Pavel, Colorado Reed, Björn Hartmann, and Maneesh Agrawala. 2014. Video digests: a browsable, skimmable format for informational lecture videos. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST '14) . Association for Computing Machinery, New York, NY, USA, 573-582. doi:10.1145/2642918.2647400 - · DOI
  49. [64][64] Amy Pavel, Dan B. Goldman, Björn Hartmann, and Maneesh Agrawala. 2015. SceneSkim: Searching and Browsing Movies Using Synchronized Captions, Scripts and Plot Summaries. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST '15) . Association for Computing Machinery, New York, NY, USA, 181-190. doi:10.1145/2807442.2807502 - · DOI · Paper page
  50. [66][66] Emilia Rosselli Del Turco and Peter Dalsgaard. 2023. "I wouldn't dare losing one": How music artists capture and manage ideas. In Proceedings of - the 15th Conference on Creativity and Cognition (Virtual Event, USA) (C&C '23) . Association for Computing Machinery, New York, NY, USA, 88-102. doi:10.1145/3591196.3593338 - · DOI
  51. [68][68] Steve Rubin, Floraine Berthouzoz, Gautham J. Mysore, Wilmot Li, and Maneesh Agrawala. 2013. Content-based tools for editing audio stories. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST '13) . Association for Computing Machinery, New York, NY, USA, 113-122. doi:10.1145/2501988.2501993 - · DOI
  52. [67][67] Steve Rubin, Floraine Berthouzoz, Gautham Mysore, Wilmot Li, and Maneesh Agrawala. 2012. UnderScore: musical underlays for audio stories. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (Cambridge, Massachusetts, USA) (UIST '12) . Association for Computing Machinery, New York, NY, USA, 359-366. doi:10.1145/2380116.2380163 - · DOI
  53. [69][69] Yong Rui, Thomas S Huang, and Sharad Mehrotra. 1999. Constructing table-ofcontent for videos. Multimedia systems 7 (1999), 359-368. -
  54. [71][71] Donald A Schön. 2017. The reflective practitioner: How professionals think in action . Routledge. -
  55. [70][70] Daniel M. Russell, Mark J. Stefik, Peter Pirolli, and Stuart K. Card. 1993. The cost structure of sensemaking. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (Amsterdam, The Netherlands) (CHI '93) . Association for Computing Machinery, New York, NY, USA, 269-276. doi:10.1145/169059.169209 - · DOI
  56. [72][72] Evan Schrier, Mira Dontcheva, Charles Jacobs, Geraldine Wade, and David Salesin. 2008. Adaptive layout for dynamically aggregated documents. In Proceedings of the 13th International Conference on Intelligent User Interfaces (Gran Canaria, Spain) (IUI '08) . Association for Computing Machinery, New York, NY, USA, 99-108. doi:10.1145/1378773.1378787 - · DOI
  57. [74][74] Hari Subramonyam, Roy Pea, Christopher Pondoc, Maneesh Agrawala, and Colleen Seifert. 2024. Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24) . Association for Computing Machinery, New York, NY, USA, Article 1039, 19 pages. doi:10. 1145/3613904.3642754 -
  58. [73][73] Leixian Shen, Yizhi Zhang, Haidong Zhang, and Yun Wang. 2023. Data player: Automatic generation of data videos with narration-animation interplay. IEEE Transactions on Visualization and Computer Graphics (2023). -
  59. [75][75] Lucille Alice Suchman. 1987. Plans and situated actions: The problem of humanmachine communication . Cambridge university press. -
  60. [77][77] Sangho Suh, Martinet Lee, Gracie Xia, et al. 2020. Coding strip: A pedagogical tool for teaching and learning programming concepts through comics. In 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) . IEEE, 1-10. -
  61. [76][76] Sangho Suh, Meng Chen, Bryan Min, Toby Jia-Jun Li, and Haijun Xia. 2024. Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2024). -
  62. [78][78] Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models. In Symposium on User Interface Software and Technology (UIST) . https: //doi.org/10.1145/3586183.3606756 - · DOI · Paper page
  63. [80][80] Nicole Sultanum, Fanny Chevalier, Zoya Bylinskii, and Zhicheng Liu. 2021. Leveraging Text-Chart Links to Support Authoring of Data-Driven Articles with VizFlow. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21) . Association for Computing Machinery, New York, NY, USA, Article 16, 17 pages. doi:10.1145/3411764. 3445354 - · DOI · Paper page
  64. [79][79] Sangho Suh, Jian Zhao, and Edith Law. 2022. CodeToon: Story Ideation, Auto Comic Generation, and Structure Mapping for Code-Driven Storytelling. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (Bend, OR, USA) (UIST '22) . Association for Computing Machinery, New York, NY, USA, Article 13, 16 pages. doi:10.1145/3526113.3545617 - · DOI
  65. [81][81] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). - · arXiv
  66. [83][83] Maryam Tohidi, William Buxton, Ronald Baecker, and Abigail Sellen. 2006. Getting the right design and the design right. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada) (CHI '06) . Association for Computing Machinery, New York, NY, USA, 1243-1252. doi:10.1145/1124772.1124960 - · DOI · Paper page
  67. [84][84] Anh Truong, Floraine Berthouzoz, Wilmot Li, and Maneesh Agrawala. 2016. Quickcut: An interactive tool for editing narrated video. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology . 497-507. -
  68. [85][85] Khai N. Truong, Gillian R. Hayes, and Gregory D. Abowd. 2006. Storyboarding: an empirical determination of best practices and effective guidelines. In Proceedings of the 6th Conference on Designing Interactive Systems (University Park, PA, USA) (DIS '06) . Association for Computing Machinery, New York, NY, USA, 12-21. doi:10.1145/1142405.1142410 - · DOI
  69. [87][87] Barbara Tversky, Julie Bauer Morrison, and Mireille Betrancourt. 2002. Animation: can it facilitate? International journal of human-computer studies 57, 4 (2002), 247-262. -
  70. [86][86] Theophanis Tsandilas, Catherine Letondal, and Wendy E. Mackay. 2009. Musink: composing music through augmented drawing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI '09) . Association for Computing Machinery, New York, NY, USA, 819-828. doi:10. 1145/1518701.1518827 -
  71. [90][90] April Yi Wang, Anant Mittal, Christopher Brooks, and Steve Oney. 2019. How Data Scientists Use Computational Notebooks for Real-Time Collaboration. Proc. ACM Hum. Comput. Interact. 3, CSCW (2019), 39:1-39:30. doi:10.1145/3359141 - · DOI
  72. [89][89] Stephen Voida, Elizabeth D. Mynatt, and W. Keith Edwards. 2008. Re-framing the desktop interface around the activities of knowledge work. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (Monterey, CA, USA) (UIST '08) . Association for Computing Machinery, New York, NY, USA, 211-220. doi:10.1145/1449715.1449751 - · DOI
  73. [91][91] Bryan Wang, Yuliang Li, Zhaoyang Lv, Haijun Xia, Yan Xu, and Raj Sodhi. 2024. LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing. arXiv preprint arXiv:2402.10294 (2024). - · arXiv
  74. [93][93] Fengjie Wang, Yanna Lin, Leni Yang, Haotian Li, Mingyang Gu, Min Zhu, and Huamin Qu. 2024. OutlineSpark: Igniting AI-powered Presentation Slides Creation from Computational Notebooks through Outlines. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24) . Association for Computing Machinery, New York, NY, USA, Article 159, 16 pages. doi:10.1145/3613904.3642865 - · DOI · Paper page
  75. [92][92] Bryan Wang, Yuliang Li, Zhaoyang Lv, Haijun Xia, Yan Xu, and Raj Sodhi. 2024. LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing. In Proceedings of the 29th International Conference on Intelligent User Interfaces (Greenville, SC, USA) (IUI '24) . Association for Computing Machinery, New York, NY, USA, 699-714. doi:10.1145/3640543.3645143 - · DOI
  76. [94][94] Fengjie Wang, Xuye Liu, Oujing Liu, Ali Neshati, Tengfei Ma, Min Zhu, and Jian Zhao. 2023. Slide4N: Creating Presentation Slides from Computational Notebooks with Human-AI Collaboration. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23) . Association for Computing Machinery, New York, NY, USA, Article 364, 18 pages. doi:10.1145/3544548.3580753 - · DOI
  77. [96][96] Sitong Wang, Samia Menon, Tao Long, Keren Henderson, Dingzeyu Li, Kevin Crowston, Mark Hansen, Jeffrey V Nickerson, and Lydia B Chilton. 2023. Reelframer: Co-creating news reels on social media with generative ai. arXiv preprint arXiv:2304.09653 (2023). - · arXiv
  78. [95][95] Miao Wang, Guo-Wei Yang, Shi-Min Hu, Shing-Tung Yau, Ariel Shamir, et al. 2019. Write-a-video: computational video montage from themed text. ACM Trans. Graph. 38, 6 (2019), 177-1. -
  79. [97][97] Zijie J. Wang, Katie Dai, and W. Keith Edwards. 2022. StickyLand: Breaking the Linear Presentation of Computational Notebooks. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA '22) . Association for Computing Machinery, New York, NY, USA, Article 269, 7 pages. doi:10.1145/3491101.3519653 - · DOI
  80. [99][99] Steve Whittaker and Brian Amento. 2004. Semantic speech editing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vienna, Austria) (CHI '04) . Association for Computing Machinery, New York, NY, USA, 527-534. doi:10.1145/985692.985759 - · DOI
  81. [98][98] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824-24837. -
  82. [100][100] Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI conference on human factors in computing systems . 1-22. -
  83. [101][101] Haijun Xia. 2020. Crosspower: Bridging Graphics and Linguistics. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '20) . Association for Computing Machinery, New York, NY, USA, 722-734. doi:10.1145/3379337.3415845 - · DOI
  84. [103][103] Haijun Xia, Jennifer Jacobs, and Maneesh Agrawala. 2020. Crosscast: adding visuals to audio travel podcasts. In Proceedings of the 33rd annual ACM symposium on user interface software and technology . 735-746. -
  85. [104][104] Haijun Xia, Jennifer Jacobs, and Maneesh Agrawala. 2020. Crosscast: Adding Visuals to Audio Travel Podcasts. In UIST '20: The 33rd Annual ACM Symposium on User Interface Software and Technology, Virtual Event, USA, October 20-23, 2020 . ACM, 735-746. doi:10.1145/3379337.3415882 - · DOI
  86. [106][106] Haijun Xia, Nathalie Henry Riche, Fanny Chevalier, Bruno Rodrigues De Araújo, and Daniel Wigdor. 2018. DataInk: Direct and Creative Data-Oriented Drawing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018, Montreal, QC, Canada, April 21-26, 2018 . ACM, 223. doi:10. 1145/3173574.3173797 -
  87. [105][105] Haijun Xia, Hui Xin Ng, Zhutian Chen, and James Hollan. 2022. Millions and Billions of Views: Understanding Popular Science and Knowledge Communication on Video-Sharing Platforms. In Proceedings of the Ninth ACM Conference on Learning @ Scale (New York City, NY, USA) (L@S '22) . Association for Computing Machinery, New York, NY, USA, 163-174. doi:10.1145/3491140.3528279 - · DOI
  88. [107][107] Haijun Xia, Tony Wang, Aditya Gunturu, Peiling Jiang, William Duan, and Xiaoshuo Yao. 2023. CrossTalk: Intelligent Substrates for Language-Oriented Interaction in Video-Based Communication and Collaboration. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST '23) . Association for Computing Machinery, New York, NY, USA, Article 60, 16 pages. doi:10.1145/3586183.3606773 - · DOI
  89. [108][108] J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23) . Association for Computing Machinery, New York, NY, USA, Article 437, 21 pages. doi:10.1145/3544548. 3581388 - · DOI · Paper page
  90. [110][110] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 3836-3847. -
  91. [111][111] Zheng Zhang, Jie Gao, Ranjodh Singh Dhaliwal, and Toby Jia-Jun Li. 2023. Visar: Ahuman-ai argumentative writing assistant with visual programming and rapid draft prototyping. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology . 1-30.

Cited by

Sorted by citations

Authors
Cao, Y., Huang, Y., Truong, A., Shin, H. V., & Xia, H.
Affiliations
  • University of California San Diego (US)
  • Adobe Research, New York, California, USA
  • Adobe Research, Cambridge, Massachusetts, USA
ORCID
Venue
Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2025-04-25
Pages
1-25
DOI
10.1145/3706598.3713401
OpenAlex
W4409735927
Cited by
16
References
7
Citations by year
2026: 13 · 2025: 3
FWCI
59.53
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-nc)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (2432644)
Topics
  • Physical Sciences › Engineering › Mechanical Engineering › Design Education and Practice
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Music Technology and Sound Studies
Keywords
compositional structure, human-AI co-creation, design approaches, human-AI interaction, AI-assisted design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review