Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models
Open accessThe DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index.
Abstract
While generative deep neural networks (DNNs) have demonstrated their capacity for creating novel musical compositions, less attention has been paid to the challenges and potential of co-creating with these musical AIs, especially for novices. In a needfinding study with a widely used, interactive musical AI, we found that the AI can overwhelm users with the amount of musical content it generates, and frustrate them with its non-deterministic output. To better match co-creation needs, we developed AI-steering tools, consisting of Voice Lanes that restrict content generation to particular voices; Example-Based Sliders to control the similarity of generated content to an existing example; Semantic Sliders to nudge music generation in high-level directions (happy/sad, conventional/surprising); and Multiple Alternatives of generated content to audition and choose from. In a summative study (N=21), we discovered the tools not only increased users' trust, control, comprehension, and sense of collaboration with the AI, but also contributed to a greater sense of self-efficacy and ownership of the composition relative to the AI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Creative industries employ vision-aligned proactive and semiproactive agency.”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [109]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Mapping interaction flows reveals prevalent patterns and underexplored areas, such as the sparse use of 'Auditory Interaction' [84, 109, 164], signaling innovation opportunities in domains like assistive technology or music co-creation.”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [109]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI agency patterns are typically co-operative or semi-active [66, 191], often involving a shared or dynamically negotiated locus of control [8, 47, 109].”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [109]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The locus of control can be dynamic, shifting between human guidance and AI generation [8, 109], often operating at a fine granularity for detailed adjustments [208].”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [109]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In contrast to software engineering, other domains including Human-Computer Interaction (HCI), organization and business management, and cognitive science have explored the impact of LLMs on creativity.”
From The Impact of Generative AI on Creativity in Software Development: A Research Agenda (2025) · cites this paper as [77]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482 · cited 1×
“To analyze the quantitative measures, we conducted paired t-tests using Benjamani-Hochberg correction [3] to account for the 15 planned-comparisons (using a false discovery rate Q = 0 .”
MethodDevelopment of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003 · cited 2×
“Effort: Users answered the effort question of the NASA-TLX [25], where 1=very low and 7=very high.”
Method(2016) · Cited by 11,679
An integrative model of organizational trust
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) · The Academy of Management Review · doi:10.2307/258792 · Cited by 10,736 · cited 1×
“Trust : Participants rated the system along Mayer’s dimensions of trust [35]: capability, benevolence, and integrity.”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 3×
“Previous research suggests benefits of educating users about the AI and its capabilities [1], or providing onboarding materials and exercises [6].”
BackgroundCai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594 · cited 1×
“Previous research suggests benefits of educating users about the AI and its capabilities [1], or providing onboarding materials and exercises [6].”
BackgroundSelf-citationBoulanger-Lewandowski, N., Bengio, Y., & Vincent, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6392 · Cited by 459 · cited 1×
“Research has demonstrated the potential for modeling and synthesizing music, ranging from singlevoice sequences [13] and multi-part music [19, 34], to music with variable parts at each time step [4] and music with longterm structure over minutes [30, 37, 26].”
BackgroundDeepBach: a Steerable Model for Bach Chorales GenerationPreprint
Hadjeres, G., Pachet, F., & Nielsen, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.01010 · Cited by 439 · cited 7×
“The conventional interface does include interactive control features via the infillmask feature (present in both conditions) which enables users to crop any region of music to be regenerated [11, 24].”
BackgroundMethodInfluentialOh, C., Song, J., Choi, J., Kim, S., Lee, S., & Suh, B. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174223 · Cited by 301
Enabling Factorized Piano Music Modeling and Generation with the MAESTRO DatasetPreprint
Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.-Z. A., Dieleman, S., Elsen, E., Engel, J., & Douglas Eck, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12247 · Cited by 301 · cited 1×
“Research has demonstrated the potential for modeling and synthesizing music, ranging from singlevoice sequences [13] and multi-part music [19, 34], to music with variable parts at each time step [4] and music with longterm structure over minutes [30, 37, 26].”
BackgroundSelf-citationA Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
Roberts, A., Engel, J., Raffel, C., Hawthorne, C., & Eck, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.05428 · Cited by 284 · cited 1×
“For example, there are emerging approaches aimed at learning a continuous latent space so that users can interpolate between music [38], or explore a space of musical alternatives [10].”
BackgroundCreative Writing with a Machine in the Loop: Case Studies on Slogans and Stories
Clark, E., Ross, A. S., Tan, C., Ji, Y., & Smith, N. A. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172983 · Cited by 257 · cited 3×
“For example, an AI might flesh out a half-sketched drawing [36], write the next paragraph of a story [7], or add an image to a design mood board [33].”
BackgroundFinding temporal structure in music: blues improvisation with LSTM recurrent networks
Eck, D., & Schmidhuber, J. (2003) · Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing · doi:10.1109/nnsp.2002.1030094 · Cited by 236 · cited 1×
“Research has demonstrated the potential for modeling and synthesizing music, ranging from single-voice sequences [13] and multi-part music [19, 34], to music with variable parts at each time step [4] and music with long-term structure over minutes [30, 37, 26].”
BackgroundGuzdial, M., Liao, N., Chen, J., Chen, S.-Y., Shah, S., Shah, V., Reno, J. O., Smith, G., & Riedl, M. O. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300854 · Cited by 176 · cited 3×
“The acceleration of AI capabilities has renewed interest in how AI can enable human-AI co-creation in domains such as drawing [36, 9, 32, 14], creative writing [18, 7], design ideation [33], video game content generation [23], and dance [31].”
BackgroundMay AI? Design Ideation with Cooperative Contextual Bandits
Koch, J., Lucero, A., Hegemann, L., & Oulasvirta, A. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300863 · Cited by 174
MySong: automatic accompaniment generation for vocal melodies
Simon, I., Morris, D., & Basu, S. (2008) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357169 · Cited by 159 · cited 1×
“Examples of such designs and systems include those that allow users to find chords to accompany a melody [41, 21], experiment with ad-venturous chord progressions [28, 17], control the similarity vs. otherness for retrieval of music samples [2], use custom gestural inputs to interpolate between…”
BackgroundMetaphoria: An Algorithmic Companion for Metaphor Creation
Gero, K. I., & Chilton, L. B. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300526 · Cited by 146 · cited 3×
“The acceleration of AI capabilities has renewed interest in how AI can enable human-AI co-creation in domains such as drawing [36, 9, 32, 14], creative writing [18, 7], design ideation [33], video game content generation [23], and dance [31].”
BackgroundDavis, N., Hsiao, C.-P., Yashraj Singh, K., Li, L., & Magerko, B. (2016) · Proceedings of the 21st International Conference on Intelligent User Interfaces · doi:10.1145/2856767.2856795 · Cited by 138 · cited 4×
“The acceleration of AI capabilities has renewed interest in how AI can enable human-AI co-creation in domains such as drawing [36, 9, 32, 14], creative writing [18, 7], design ideation [33], video game content generation [23], and dance [31].”
BackgroundInfluentialHyperscore: a graphical sketchpad for novice composers
Farbood, M. M., Pasztor, E. C., & Jennings, K. (2004) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2004.1255809 · Cited by 106 · cited 1×
“…a melody [41, 21], experiment with ad-venturous chord progressions [28, 17], control the similarity vs. otherness for retrieval of music samples [2], use custom gestural inputs to interpolate between synthesizer sounds [16], or turn free-hand sketches into harmonious musical textures [15].”
BackgroundCook, P. R., & Fiebrink, R. A. (2011) · Cited by 103
Huang, C.-Z. A., Cooijmans, T., Roberts, A., Courville, A., & Eck, D. (2019) · arXiv · doi:10.48550/arxiv.1903.07227 · Cited by 100 · cited 3×
“Examples include DeepBach [24] and Coconet [27], both trained on four-part Bach Chorales.”
BackgroundSelf-citationHuang, C.-Z. A., Vaswani, A., Uszkoreit, J., Shazeer, N., Simon, I., Hawthorne, C., Dai, A. M., Hoffman, M. D., Dinculescu, M., & Eck, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.04281 · Cited by 68
Self-citationLearning to Groove with Inverse Sequence TransformationsPreprint
Gillick, J., Roberts, A., Engel, J., Eck, D., & Bamman, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.06118 · Cited by 65 · cited 2×
“For example, by completing a drawing that a user has started [36, 9, 32, 14] or filling in a missing section of a song [27, 24], generative models could enable untrained lay users to take part in creative experiences that would otherwise be difficult to achieve without additional training or specialization [29, 12, 19…”
BackgroundChordRipple: Recommending Chords to Help Novice Composers Go Beyond the Ordinary
Huang, C.-Z. A., Duvenaud, D., & Gajos, K. Z. (2016) · Proceedings of the 21st International Conference on Intelligent User Interfaces · doi:10.1145/2856767.2856792 · Cited by 58 · cited 2×
“…designs and systems include those that allow users to find chords to accompany a melody [41, 21], experiment with ad-venturous chord progressions [28, 17], control the similarity vs. otherness for retrieval of music samples [2], use custom gestural inputs to interpolate between synthesizer…”
BackgroundResultSelf-citationcollabdraw: An Environment for Collaborative Sketching with an Artificial Agent
Fan, J. E., Dinculescu, M., & Ha, D. (2019) · Creativity and Cognition · doi:10.1145/3325480.3326578 · Cited by 57 · cited 3×
“For example, by completing a drawing that a user has started [36, 9, 32, 14] or filling in a missing section of a song [27, 24], generative models could enable untrained lay users to take part in creative experiences that would otherwise be difficult to achieve without additional training or specialization [29, 12, 19…”
BackgroundDeep Learning in a Computational Model for Conceptual Shifts in a Co-Creative Design SystemPreprint
Karimi, P., Maher, M. L., Davis, N. M., & Grace, K. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10188 · Cited by 50 · cited 2×
“The acceleration of AI capabilities has renewed interest in how AI can enable human-AI co-creation in domains such as drawing [36, 9, 32, 14], creative writing [18, 7], design ideation [33], video game content generation [23], and dance [31].”
BackgroundDonahue, C., Simon, I., & Dieleman, S. (2019) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3301275.3302288 · Cited by 49 · cited 1×
“…a drawing that a user has started [36, 9, 32, 14] or filling in a missing section of a song [27, 24], generative models could enable untrained lay users to take part in creative experiences that would otherwise be difficult to achieve without additional training or specialization [29, 12, 19].”
BackgroundThe Bach Doodle: Approachable music composition with machine learning at scalePreprint
Huang, C.-Z. A., Hawthorne, C., Roberts, A., Dinculescu, M., Wexler, J., Hong, L., & Howcroft, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.06637 · Cited by 41
Self-citationMidiMe: Personalizing a MusicVAE model with user data
Dinculescu, M., Engel, J., & Roberts, A. P. (2019) · Cited by 39
Interaction-based Authoring for Scalable Co-creative Agents
Jacob, M., & Magerko, B. S. (2015) · ICCC · Cited by 32
Magenta.js: A JavaScript API for Augmenting Creativity with Deep Learning
Roberts, A. P., Hawthorne, C., & Simon, I. (2018) · Cited by 28
Andersen, K., & Knees, P. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2892439 · Cited by 12 · cited 2×
“…a melody [41, 21], experiment with ad-venturous chord progressions [28, 17], control the similarity vs. otherness for retrieval of music samples [2], use custom gestural inputs to interpolate between synthesizer sounds [16], or turn free-hand sketches into harmonious musical textures [15].”
BackgroundCollaborating with Intelligent Machines: Interfaces for Creative Sound
Grote, F., Andersen, K., & Knees, P. (2015) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2702613.2702650 · Cited by 8 · cited 1×
“Efforts examining human-AI collaboration for creating music has been relatively nascent [22], particularly with generative DNN music agents of similar prowess.”
BackgroundFukayama, S., Yoshii, K., & Goto, M. (2013) · doi:10.5281/zenodo.1417700 · Cited by 7
Lumanote: A Real-Time Interactive Music Composition Assistant.
Granger, J., Aviles, M., Kirby, J., Griffin, A., Yoon, J., Lara-Garduno, R., & Hammond, T. (2018) · IUI Workshops · Cited by 7
Research works printed but not linked: 4
- Kate Compton and Michael Mateas. 2015. Casual Creators. In Proceedings of the Sixth International Conference on Computational Creativity (ICCC 2015), Hannu Toivonen, Simon Colton, Michael Cook, and Dan Ventura (Eds.). Brigham Young University, Park City, Utah, 228--235. http://computationalcreativity.net/iccc2015/proceedings/10_2Compton.pdf
- Monica Dinculescu and Cheng-Zhi Anna Huang. 2019. Coucou: An Expanded Interface for Interactive Composition with Coconet, through Flexible Inpainting. (2019). https://coconet.glitch.me/
- Feynman Liang. 2016. BachBot: Automatic Composition in the Style of Bach Chorales. Masters thesis, University of Cambridge (2016).
- Ralf Schwarzer and Matthias Jerusalem. 1995. Generalized Self-efficacy Scale. Measures in Health Psychology: A User's Portfolio. Causal and Control Beliefs 1, 1 (1995), 35--37.
Cited by
Design Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501825 · Cited by 639
“[30] introduced CoCoCo, an AI music creation system which lets users move sliders to tune their generated music to be "happier" or "sadder".”
BackgroundInfluentialDesign Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2021) · arXiv · doi:10.48550/arxiv.2109.06977 · Cited by 635
“[30] introduced CoCoCo, an AI music creation system which lets users move sliders to tune their generated music to be "happier" or "sadder".”
BackgroundInfluentialWu, T., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517582 · Cited by 483
“Contrary to recent work in human-AI interaction, which primarily examined how to increase AI controllability through exposing knobs within a model [44, 49], our work opens up the possibility of steering AI using the model itself as units to control.”
BackgroundSelf-citationWordcraft: Story Writing With Large Language Models
Yuan, A., Coenen, A., Reif, E., & Ippolito, D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511105 · Cited by 374
“There have been many studies of humanAI co-creation across domains, including drawing [8, 27, 31], music [15, 22, 25], video game content creation [24], and design ideation [16].”
BackgroundSelf-citationRezwana, J., & Maher, M. L. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3519026 · Cited by 342
“AI ability alone does not ensure a positive collaborative experience of users with the AI [100] and interaction is more critical than algorithms where interaction with the users is essential [152].”
Background
Metadata
- Authors
- Louie, R., Coenen, A., Huang, C. Z., Terry, M., & Cai, C. J.
- Affiliations
- Northwestern University (US)
- Google (United States) (US)
- Independent Researcher, Mountain View, CA, USA
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-13
- OpenAlex
- W3031000691
- Semantic Scholar
- 77a5e08f361b6f91cac8a24b380a14c12bb93383
- Cited by
- 271
- References
- 37
- Citations by year
- 2026: 40 · 2025: 75 · 2024: 65 · 2023: 40 · 2022: 34 · 2021: 14 · 2020: 3
- FWCI
- 7.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by-sa)
- Topics
- Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- novice users, generative music, user trust, self-efficacy, human-AI collaboration, music composition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.