CoChat

Expressive Communication: Evaluating Developments in Generative Models and Steering Interfaces for Music Creation

Open access
Louie, Ryan, Engel, Jesse, Huang, Cheng-Zhi Anna · 2022 · 27th International Conference on Intelligent User Interfaces
Cited by 30

The DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.

Abstract

There is an increasing interest from ML and HCI communities in empowering creators with better generative models and more intuitive interfaces with which to control them. In music, ML researchers have focused on training models capable of generating pieces with increasing long-range structure and musical coherence, while HCI researchers have separately focused on designing steering interfaces that support user control and ownership. In this study, we investigate how developments in both models and user interfaces are important for empowering co-creation where the goal is to create music that communicates particular imagery or ideas (e.g., as is common for other purposeful tasks in music creation like establishing mood or creating accompanying music for another media). Our study is distinguished in that it measures communication through both composer’s self-reported experiences, and how listeners evaluate this communication through the music. In an evaluation study with 26 composers creating 100+ pieces of music and listeners providing 1000+ head-to-head comparisons, we find that more expressive models and more steerable interfaces are important and complementary ways to make a difference in composers communicating through music and supporting their creative empowerment.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3490099.3511159

Cites

20 of 20 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 20 references

Research works printed but not linked: 36

  1. Alex Alemi , Ben Poole , Ian Fischer , Josh Dillon , Rif A Saurus , and Kevin Murphy . 2018. An information-theoretic analysis of deep latent-variable models. OpenReview: https://openreview.net/forum?id=H1rRWl-Cb ( 2018 ). Alex Alemi, Ben Poole, Ian Fischer, Josh Dillon, Rif A Saurus, and Kevin Murphy. 2018. An information-theoretic analysis of deep latent-variable models. OpenReview: https://openreview.net/forum?id=H1rRWl-Cb (2018).
  2. Safinah Ali and Devi Parikh. 2021. Telling Creative Stories Using Generative Visual Aids. arXiv preprint arXiv:2110.14810(2021). Safinah Ali and Devi Parikh. 2021. Telling Creative Stories Using Generative Visual Aids. arXiv preprint arXiv:2110.14810(2021). · arXiv
  3. Tolga Bolukbasi , Kai-Wei Chang , James Y Zou , Venkatesh Saligrama , and Adam T Kalai . 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 ( 2016 ). Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 (2016).
  4. Aylin Caliskan , Joanna J Bryson , and Arvind Narayanan . 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 ( 2017 ), 183–186. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183–186.
  5. Paul F Christiano , Jan Leike , Tom B Brown , Miljan Martic , Shane Legg , and Dario Amodei . 2017 . Deep reinforcement learning from human preferences . In Proceedings of the 31st International Conference on Neural Information Processing Systems. 4302–4310 . Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 4302–4310.
  6. 10.1145/3290605.3300599 · DOI · Paper page
  7. Patrick Esser Robin Rombach and Björn Ommer. 2020. A note on data biases in generative models. arXiv preprint arXiv:2012.02516(2020). Patrick Esser Robin Rombach and Björn Ommer. 2020. A note on data biases in generative models. arXiv preprint arXiv:2012.02516(2020). · arXiv
  8. 10.1145/3325480.3326578 · DOI · Paper page
  9. 10.1145/3298689.3347052 · doi:10.1145/3298689.3347052
  10. Curtis Hawthorne Andriy Stasyuk Adam Roberts Ian Simon Cheng-Zhi Anna Huang Sander Dieleman Erich Elsen Jesse Engel and Douglas Eck. 2018. Enabling factorized piano music modeling and generation with the MAESTRO dataset. arXiv preprint arXiv:1810.12247(2018). Curtis Hawthorne Andriy Stasyuk Adam Roberts Ian Simon Cheng-Zhi Anna Huang Sander Dieleman Erich Elsen Jesse Engel and Douglas Eck. 2018. Enabling factorized piano music modeling and generation with the MAESTRO dataset. arXiv preprint arXiv:1810.12247(2018). · arXiv
  11. Thomas Hellström Virginia Dignum and Suna Bensch. 2020. Bias in Machine Learning–What is it Good for?arXiv preprint arXiv:2004.00686(2020). Thomas Hellström Virginia Dignum and Suna Bensch. 2020. Bias in Machine Learning–What is it Good for?arXiv preprint arXiv:2004.00686(2020). · arXiv
  12. Martin Heusel , Hubert Ramsauer , Thomas Unterthiner , Bernhard Nessler , and Sepp Hochreiter . 2017. GANs trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 ( 2017 ). Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017).
  13. Cheng- Zhi Anna Huang , Tim Cooijmans , Adam Roberts , Aaron Courville , and Doug Eck . 2017 . Counterpoint by Convolution . In Proceedings of the International Conference on Music Information Retrieval. Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron Courville, and Doug Eck. 2017. Counterpoint by Convolution. In Proceedings of the International Conference on Music Information Retrieval.
  14. 10.1145/2856767.2856792 · DOI · Paper page
  15. Cheng-Zhi Anna Huang Ashish Vaswani Jakob Uszkoreit Noam Shazeer Ian Simon Curtis Hawthorne Andrew M Dai Matthew D Hoffman Monica Dinculescu and Douglas Eck. 2018. Music Transformer. ICLR (2018). Cheng-Zhi Anna Huang Ashish Vaswani Jakob Uszkoreit Noam Shazeer Ian Simon Curtis Hawthorne Andrew M Dai Matthew D Hoffman Monica Dinculescu and Douglas Eck. 2018. Music Transformer. ICLR (2018).
  16. Cheng- Zhi Anna Huang , Ashish Vaswani , Jakob Uszkoreit , Ian Simon , Curtis Hawthorne , Noam Shazeer , Andrew M Dai , Matthew D Hoffman , Monica Dinculescu , and Douglas Eck . 2019 . Music Transformer. In International Conference on Learning Representations. Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. 2019. Music Transformer. In International Conference on Learning Representations.
  17. Yu-Siang Huang and Yi-Hsuan Yang. 2020. Pop Music Transformer: Generating Music with Rhythm and Harmony. arXiv preprint arXiv:2002.00212(2020). Yu-Siang Huang and Yi-Hsuan Yang. 2020. Pop Music Transformer: Generating Music with Rhythm and Harmony. arXiv preprint arXiv:2002.00212(2020). · arXiv
  18. Daphne Ippolito , Cheng- Zhi Anna Huang , Curtis Hawthorne , and Douglas Eck . 2018 . Infilling Piano Performances. In NeurIPS Workshop on Machine Learning for Creativity and Design. Daphne Ippolito, Cheng-Zhi Anna Huang, Curtis Hawthorne, and Douglas Eck. 2018. Infilling Piano Performances. In NeurIPS Workshop on Machine Learning for Creativity and Design.
  19. Natasha Jaques , Jennifer McCleary , Jesse Engel , David Ha , Fred Bertsch , Douglas Eck , and Rosalind Picard . 2020 . Learning via social awareness: Improving a deep generative sketching model with facial feedback . In Workshop on Artificial Intelligence in Affective Computing. PMLR, 1–9. Natasha Jaques, Jennifer McCleary, Jesse Engel, David Ha, Fred Bertsch, Douglas Eck, and Rosalind Picard. 2020. Learning via social awareness: Improving a deep generative sketching model with facial feedback. In Workshop on Artificial Intelligence in Affective Computing. PMLR, 1–9.
  20. Hannah Rose Kirk , Filippo Volpin , Haider Iqbal , Elias Benussi , Frederic Dreyer , Aleksandar Shtedritski , Yuki Asano , 2021. Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models. Advances in Neural Information Processing Systems 34 ( 2021 ). Hannah Rose Kirk, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, Yuki Asano, 2021. Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models. Advances in Neural Information Processing Systems 34 (2021).
  21. 10.1145/3290605.3300863 · DOI · Paper page
  22. 10.1145/3313831.3376739 · DOI · Paper page
  23. Mark Marrington 2017. Composing with the digital audio workstation. The Singer-Songwriter Handbook(2017) 77–89. Mark Marrington 2017. Composing with the digital audio workstation. The Singer-Songwriter Handbook(2017) 77–89. · doi:10.5040/9781501396595.ch-006
  24. 10.1038/nature14236 · DOI · Paper page
  25. 10.1007/s00521-018-3758-9 · DOI · Paper page
  26. Siddharth Reddy Anca D Dragan and Sergey Levine. 2021. Pragmatic Image Compression for Human-in-the-Loop Decision-Making. arXiv preprint arXiv:2108.04219(2021). Siddharth Reddy Anca D Dragan and Sergey Levine. 2021. Pragmatic Image Compression for Human-in-the-Loop Decision-Making. arXiv preprint arXiv:2108.04219(2021). · arXiv
  27. Adam Roberts , Jesse Engel , Colin Raffel , Curtis Hawthorne , and Douglas Eck . 2018 . A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music . In International Conference on Machine Learning (ICML). http://proceedings.mlr.press/v80/roberts18a.html Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. 2018. A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music. In International Conference on Machine Learning (ICML). http://proceedings.mlr.press/v80/roberts18a.html
  28. Tim Salimans , Ian Goodfellow , Wojciech Zaremba , Vicki Cheung , Alec Radford , and Xi Chen . 2016. Improved techniques for training GANs. Advances in neural information processing systems ( 2016 ). Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training GANs. Advances in neural information processing systems (2016).
  29. Sebastian Schelter and Julia Stoyanovich. 2020. Taming technical bias in machine learning pipelines. Bulletin of the Technical Committee on Data Engineering 43 4(2020). Sebastian Schelter and Julia Stoyanovich. 2020. Taming technical bias in machine learning pipelines. Bulletin of the Technical Committee on Data Engineering 43 4(2020).
  30. Ralf Schwarzer and Matthias Jerusalem . 1995 . Generalized Self-efficacy Scale. Measures in Health Psychology: A User’s Portfolio . Causal and Control Beliefs 1 , 1 (1995), 35 – 37 . Ralf Schwarzer and Matthias Jerusalem. 1995. Generalized Self-efficacy Scale. Measures in Health Psychology: A User’s Portfolio. Causal and Control Beliefs 1, 1 (1995), 35–37.
  31. 10.1145/2984511.2984578 · DOI · Paper page
  32. Ian Simon , Cheng- Zhi Anna Huang , Jesse Engel , Curtis Hawthorne , and Monica Dinculescu . 2019. Generating Piano Music with Transformer. Magenta Blog: https://magenta.tensorflow.org/piano-transformer ( 2019 ). Ian Simon, Cheng-Zhi Anna Huang, Jesse Engel, Curtis Hawthorne, and Monica Dinculescu. 2019. Generating Piano Music with Transformer. Magenta Blog: https://magenta.tensorflow.org/piano-transformer (2019).
  33. Ramya Srinivasan , Emily Denton , Jordan Famularo , Negar Rostamzadeh , Fernando Diaz , and Beth Coleman . 2021 . Artsheets for Art Datasets. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://openreview.net/forum?id=K7ke_GZ_6N Ramya Srinivasan, Emily Denton, Jordan Famularo, Negar Rostamzadeh, Fernando Diaz, and Beth Coleman. 2021. Artsheets for Art Datasets. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://openreview.net/forum?id=K7ke_GZ_6N
  34. Richard S Sutton and Andrew G Barto . 2018. Reinforcement learning: An introduction . MIT press . Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
  35. Li-Chia Yang and Alexander Lerch . 2020. On the evaluation of generative models in music. Neural Computing and Applications 32, 9 ( 2020 ). Li-Chia Yang and Alexander Lerch. 2020. On the evaluation of generative models in music. Neural Computing and Applications 32, 9 (2020).
  36. Yijun Zhou , Yuki Koyama , Masataka Goto , and Takeo Igarashi . 2021 . Interactive Exploration-Exploitation Balancing for Generative Melody Composition. In 26th International Conference on Intelligent User Interfaces. 43–47 . Yijun Zhou, Yuki Koyama, Masataka Goto, and Takeo Igarashi. 2021. Interactive Exploration-Exploitation Balancing for Generative Melody Composition. In 26th International Conference on Intelligent User Interfaces. 43–47.

Cited by

See all 30

Metadata

Authors
Louie, R., Engel, J., & Huang, C.-Z. A.
Venue
27th International Conference on Intelligent User Interfaces
Publisher
ACM
Published
2022-03-22
Pages
405-417
OpenAlex
W4220793291
Cited by
30
References
20
FWCI
2.12
Type
conference paper
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Expressive Communication: Evaluating Developments in Generative Models and Steering Interfaces for Music Creation. https://app.cochat.ai/papers/10.1145/3490099.3511159
The paper
Louie, R., Engel, J., & Huang, C.-Z. A. (2022). Expressive Communication: Evaluating Developments in Generative Models and Steering Interfaces for Music Creation. 27th International Conference on Intelligent User Interfaces. https://doi.org/10.1145/3490099.3511159

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.