Expressive Communication: Evaluating Developments in Generative Models and Steering Interfaces for Music Creation
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
There is an increasing interest from ML and HCI communities in empowering creators with better generative models and more intuitive interfaces with which to control them. In music, ML researchers have focused on training models capable of generating pieces with increasing long-range structure and musical coherence, while HCI researchers have separately focused on designing steering interfaces that support user control and ownership. In this study, we investigate how developments in both models and user interfaces are important for empowering co-creation where the goal is to create music that communicates particular imagery or ideas (e.g., as is common for other purposeful tasks in music creation like establishing mood or creating accompanying music for another media). Our study is distinguished in that it measures communication through both composer’s self-reported experiences, and how listeners evaluate this communication through the music. In an evaluation study with 26 composers creating 100+ pieces of music and listeners providing 1000+ head-to-head comparisons, we find that more expressive models and more steerable interfaces are important and complementary ways to make a difference in composers communicating through music and supporting their creative empowerment.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 1×
“Motivated by the importance of supporting efective, humancentered partnerships with AI [4, 36, 42] and supporting intrinsically-rewarding creative activities [13], we asked three questions about participants’ attitudes towards the generative tool and their composition experience using the tool.”
BackgroundA note on the evaluation of generative models
Theis, L., Oord, A. V. D., & Bethge, M. (2015) · arXiv · doi:10.48550/arxiv.1511.01844 · Cited by 1,026
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2020) · arXiv · doi:10.48550/arxiv.2004.09456 · Cited by 643
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
Quantifying the Creativity Support of Digital Tools through the Creativity Support Index
Cherry, E., & Latulipe, C. (2014) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2617588 · Cited by 498 · cited 1×
“For example, the Creativity Support Index (CSI) measures the impact on six dimensions such as exploration, expressiveness, immersion, en-joyment, results worth efort, and collaboration [11].”
BackgroundOh, C., Song, J., Choi, J., Kim, S., Lee, S., & Suh, B. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174223 · Cited by 301
Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models
Louie, R., Coenen, A., Huang, C. Z., Terry, M., & Cai, C. J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376739 · Cited by 271 · cited 5×
“When designing a steering interface for our experiments, we drew upon HCI research about building better interactive tools that provide users (e.g., novice composers) more control and agency when creating with deep generative models [37].”
BackgroundMethodInfluentialSelf-citation
Research works printed but not linked: 36
- Alex Alemi , Ben Poole , Ian Fischer , Josh Dillon , Rif A Saurus , and Kevin Murphy . 2018. An information-theoretic analysis of deep latent-variable models. OpenReview: https://openreview.net/forum?id=H1rRWl-Cb ( 2018 ). Alex Alemi, Ben Poole, Ian Fischer, Josh Dillon, Rif A Saurus, and Kevin Murphy. 2018. An information-theoretic analysis of deep latent-variable models. OpenReview: https://openreview.net/forum?id=H1rRWl-Cb (2018).
- Safinah Ali and Devi Parikh. 2021. Telling Creative Stories Using Generative Visual Aids. arXiv preprint arXiv:2110.14810(2021). Safinah Ali and Devi Parikh. 2021. Telling Creative Stories Using Generative Visual Aids. arXiv preprint arXiv:2110.14810(2021). · arXiv
- Tolga Bolukbasi , Kai-Wei Chang , James Y Zou , Venkatesh Saligrama , and Adam T Kalai . 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 ( 2016 ). Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 (2016).
- Aylin Caliskan , Joanna J Bryson , and Arvind Narayanan . 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 ( 2017 ), 183–186. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183–186.
- Paul F Christiano , Jan Leike , Tom B Brown , Miljan Martic , Shane Legg , and Dario Amodei . 2017 . Deep reinforcement learning from human preferences . In Proceedings of the 31st International Conference on Neural Information Processing Systems. 4302–4310 . Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 4302–4310.
- 10.1145/3290605.3300599 · DOI · Paper page
- Patrick Esser Robin Rombach and Björn Ommer. 2020. A note on data biases in generative models. arXiv preprint arXiv:2012.02516(2020). Patrick Esser Robin Rombach and Björn Ommer. 2020. A note on data biases in generative models. arXiv preprint arXiv:2012.02516(2020). · arXiv
- 10.1145/3325480.3326578 · DOI · Paper page
- 10.1145/3298689.3347052 · doi:10.1145/3298689.3347052
- Curtis Hawthorne Andriy Stasyuk Adam Roberts Ian Simon Cheng-Zhi Anna Huang Sander Dieleman Erich Elsen Jesse Engel and Douglas Eck. 2018. Enabling factorized piano music modeling and generation with the MAESTRO dataset. arXiv preprint arXiv:1810.12247(2018). Curtis Hawthorne Andriy Stasyuk Adam Roberts Ian Simon Cheng-Zhi Anna Huang Sander Dieleman Erich Elsen Jesse Engel and Douglas Eck. 2018. Enabling factorized piano music modeling and generation with the MAESTRO dataset. arXiv preprint arXiv:1810.12247(2018). · arXiv
- Thomas Hellström Virginia Dignum and Suna Bensch. 2020. Bias in Machine Learning–What is it Good for?arXiv preprint arXiv:2004.00686(2020). Thomas Hellström Virginia Dignum and Suna Bensch. 2020. Bias in Machine Learning–What is it Good for?arXiv preprint arXiv:2004.00686(2020). · arXiv
- Martin Heusel , Hubert Ramsauer , Thomas Unterthiner , Bernhard Nessler , and Sepp Hochreiter . 2017. GANs trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 ( 2017 ). Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017).
- Cheng- Zhi Anna Huang , Tim Cooijmans , Adam Roberts , Aaron Courville , and Doug Eck . 2017 . Counterpoint by Convolution . In Proceedings of the International Conference on Music Information Retrieval. Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron Courville, and Doug Eck. 2017. Counterpoint by Convolution. In Proceedings of the International Conference on Music Information Retrieval.
- 10.1145/2856767.2856792 · DOI · Paper page
- Cheng-Zhi Anna Huang Ashish Vaswani Jakob Uszkoreit Noam Shazeer Ian Simon Curtis Hawthorne Andrew M Dai Matthew D Hoffman Monica Dinculescu and Douglas Eck. 2018. Music Transformer. ICLR (2018). Cheng-Zhi Anna Huang Ashish Vaswani Jakob Uszkoreit Noam Shazeer Ian Simon Curtis Hawthorne Andrew M Dai Matthew D Hoffman Monica Dinculescu and Douglas Eck. 2018. Music Transformer. ICLR (2018).
- Cheng- Zhi Anna Huang , Ashish Vaswani , Jakob Uszkoreit , Ian Simon , Curtis Hawthorne , Noam Shazeer , Andrew M Dai , Matthew D Hoffman , Monica Dinculescu , and Douglas Eck . 2019 . Music Transformer. In International Conference on Learning Representations. Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. 2019. Music Transformer. In International Conference on Learning Representations.
- Yu-Siang Huang and Yi-Hsuan Yang. 2020. Pop Music Transformer: Generating Music with Rhythm and Harmony. arXiv preprint arXiv:2002.00212(2020). Yu-Siang Huang and Yi-Hsuan Yang. 2020. Pop Music Transformer: Generating Music with Rhythm and Harmony. arXiv preprint arXiv:2002.00212(2020). · arXiv
- Daphne Ippolito , Cheng- Zhi Anna Huang , Curtis Hawthorne , and Douglas Eck . 2018 . Infilling Piano Performances. In NeurIPS Workshop on Machine Learning for Creativity and Design. Daphne Ippolito, Cheng-Zhi Anna Huang, Curtis Hawthorne, and Douglas Eck. 2018. Infilling Piano Performances. In NeurIPS Workshop on Machine Learning for Creativity and Design.
- Natasha Jaques , Jennifer McCleary , Jesse Engel , David Ha , Fred Bertsch , Douglas Eck , and Rosalind Picard . 2020 . Learning via social awareness: Improving a deep generative sketching model with facial feedback . In Workshop on Artificial Intelligence in Affective Computing. PMLR, 1–9. Natasha Jaques, Jennifer McCleary, Jesse Engel, David Ha, Fred Bertsch, Douglas Eck, and Rosalind Picard. 2020. Learning via social awareness: Improving a deep generative sketching model with facial feedback. In Workshop on Artificial Intelligence in Affective Computing. PMLR, 1–9.
- Hannah Rose Kirk , Filippo Volpin , Haider Iqbal , Elias Benussi , Frederic Dreyer , Aleksandar Shtedritski , Yuki Asano , 2021. Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models. Advances in Neural Information Processing Systems 34 ( 2021 ). Hannah Rose Kirk, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, Yuki Asano, 2021. Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models. Advances in Neural Information Processing Systems 34 (2021).
- 10.1145/3290605.3300863 · DOI · Paper page
- 10.1145/3313831.3376739 · DOI · Paper page
- Mark Marrington 2017. Composing with the digital audio workstation. The Singer-Songwriter Handbook(2017) 77–89. Mark Marrington 2017. Composing with the digital audio workstation. The Singer-Songwriter Handbook(2017) 77–89. · doi:10.5040/9781501396595.ch-006
- 10.1038/nature14236 · DOI · Paper page
- 10.1007/s00521-018-3758-9 · DOI · Paper page
- Siddharth Reddy Anca D Dragan and Sergey Levine. 2021. Pragmatic Image Compression for Human-in-the-Loop Decision-Making. arXiv preprint arXiv:2108.04219(2021). Siddharth Reddy Anca D Dragan and Sergey Levine. 2021. Pragmatic Image Compression for Human-in-the-Loop Decision-Making. arXiv preprint arXiv:2108.04219(2021). · arXiv
- Adam Roberts , Jesse Engel , Colin Raffel , Curtis Hawthorne , and Douglas Eck . 2018 . A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music . In International Conference on Machine Learning (ICML). http://proceedings.mlr.press/v80/roberts18a.html Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. 2018. A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music. In International Conference on Machine Learning (ICML). http://proceedings.mlr.press/v80/roberts18a.html
- Tim Salimans , Ian Goodfellow , Wojciech Zaremba , Vicki Cheung , Alec Radford , and Xi Chen . 2016. Improved techniques for training GANs. Advances in neural information processing systems ( 2016 ). Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training GANs. Advances in neural information processing systems (2016).
- Sebastian Schelter and Julia Stoyanovich. 2020. Taming technical bias in machine learning pipelines. Bulletin of the Technical Committee on Data Engineering 43 4(2020). Sebastian Schelter and Julia Stoyanovich. 2020. Taming technical bias in machine learning pipelines. Bulletin of the Technical Committee on Data Engineering 43 4(2020).
- Ralf Schwarzer and Matthias Jerusalem . 1995 . Generalized Self-efficacy Scale. Measures in Health Psychology: A User’s Portfolio . Causal and Control Beliefs 1 , 1 (1995), 35 – 37 . Ralf Schwarzer and Matthias Jerusalem. 1995. Generalized Self-efficacy Scale. Measures in Health Psychology: A User’s Portfolio. Causal and Control Beliefs 1, 1 (1995), 35–37.
- 10.1145/2984511.2984578 · DOI · Paper page
- Ian Simon , Cheng- Zhi Anna Huang , Jesse Engel , Curtis Hawthorne , and Monica Dinculescu . 2019. Generating Piano Music with Transformer. Magenta Blog: https://magenta.tensorflow.org/piano-transformer ( 2019 ). Ian Simon, Cheng-Zhi Anna Huang, Jesse Engel, Curtis Hawthorne, and Monica Dinculescu. 2019. Generating Piano Music with Transformer. Magenta Blog: https://magenta.tensorflow.org/piano-transformer (2019).
- Ramya Srinivasan , Emily Denton , Jordan Famularo , Negar Rostamzadeh , Fernando Diaz , and Beth Coleman . 2021 . Artsheets for Art Datasets. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://openreview.net/forum?id=K7ke_GZ_6N Ramya Srinivasan, Emily Denton, Jordan Famularo, Negar Rostamzadeh, Fernando Diaz, and Beth Coleman. 2021. Artsheets for Art Datasets. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://openreview.net/forum?id=K7ke_GZ_6N
- Richard S Sutton and Andrew G Barto . 2018. Reinforcement learning: An introduction . MIT press . Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
- Li-Chia Yang and Alexander Lerch . 2020. On the evaluation of generative models in music. Neural Computing and Applications 32, 9 ( 2020 ). Li-Chia Yang and Alexander Lerch. 2020. On the evaluation of generative models in music. Neural Computing and Applications 32, 9 (2020).
- Yijun Zhou , Yuki Koyama , Masataka Goto , and Takeo Igarashi . 2021 . Interactive Exploration-Exploitation Balancing for Generative Melody Composition. In 26th International Conference on Intelligent User Interfaces. 43–47 . Yijun Zhou, Yuki Koyama, Masataka Goto, and Takeo Igarashi. 2021. Interactive Exploration-Exploitation Balancing for Generative Melody Composition. In 26th International Conference on Intelligent User Interfaces. 43–47.
Cited by
A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration
Gao, J., Gebreegziabher, S. A., Choo, K. T. W., Li, T. J.-J., Perrault, S. T., & Malone, T. W. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3650786 · Cited by 57
Kim, T. S., Lee, Y., Chang, M., & Kim, J. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606833 · Cited by 56
Kamath, P., Morreale, F., Bagaskara, P. L., Wei, Y., & Nanayakkara, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642040 · Cited by 37
Zhang, S., Wang, H., & Yi, X. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757594 · Cited by 36
Amuse: Human-AI Collaborative Songwriting with Multimodal Inspirations
Kim, Y., Lee, S.-J., & Donahue, C. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713818 · Cited by 23
““ Amuse was definitely more helpful when starting from ””
Influential
Metadata
- Authors
- Louie, R., Engel, J., & Huang, C.-Z. A.
- Venue
- 27th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2022-03-22
- Pages
- 405-417
- OpenAlex
- W4220793291
- Cited by
- 30
- References
- 20
- FWCI
- 2.12
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Expressive Communication: Evaluating Developments in Generative Models and Steering Interfaces for Music Creation. https://app.cochat.ai/papers/10.1145/3490099.3511159
- The paper
- Louie, R., Engel, J., & Huang, C.-Z. A. (2022). Expressive Communication: Evaluating Developments in Generative Models and Steering Interfaces for Music Creation. 27th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3490099.3511159
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.