CoChat
Sign Up
PreprintCited by 8

AI Song Contest: Human-AI Co-Creation in Songwriting

Huang, Cheng-Zhi Anna, Koops, Hendrik Vincent, Newton-Rex, Ed, Dinculescu, Monica, Cai, Carrie J.·2020·arXiv
DOI 10.48550/arxiv.2010.05388arXiv 2010.05388
Open this paper in CoChat — free account

Abstract

Machine learning is challenging the way we make music. Although research in deep generative models has dramatically improved the capability and fluency of music models, recent work has shown that it can be challenging for humans to partner with this new class of algorithms. In this paper, we present findings on what 13 musician/developer teams, a total of 61 users, needed when co-creating a song with AI, the challenges they faced, and how they leveraged and repurposed existing characteristics of AI to overcome some of these challenges. Many teams adopted modular approaches, such as independently running multiple smaller models that align with the musical building blocks of a song, before re-combining their results. As ML models are not easily steerable, teams also generated massive numbers of samples and curated them post-hoc, or used a range of strategies to direct the generation, or algorithmically ranked the samples. Ultimately, teams not only had to manage the "flare and focus" aspects of the creative process, but also juggle them with a parallel process of exploring and curating multiple ML models and outputs. These findings reflect a need to design machine learning-powered music interfaces that are more decomposable, steerable, interpretable, and adaptive, which in return will enable artists to more effectively explore how AI can extend their personal expression.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2010.05388

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 48

  1. Virginia Braun and Victoria Clarke. Using thematic analysis in psychology. Qualitative research in psy- chology, 3(2):77-101, 2006.
  2. Virginia Braun and Victoria Clarke. What can "the- matic analysis" offer health and wellbeing researchers? International journal of qualitative studies on health and well-being, 9, 2014.
  3. Jean-Pierre Briot, Gaëtan Hadjeres, and François Pa- chet. Deep learning techniques for music generation-a survey. arXiv preprint arXiv:1709.01620, 2017. · arXiv
  4. Ching-Hua Chuan and Elaine Chew. A hybrid system for automatic generation of style-specific accompani- ment. In Proceedings of the 4th International Joint Workshop on Computational Creativity.
  5. Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A Smith. Creative writing with a machine in the loop: Case studies on slogans and stories. In Proceedings of the 23rd International Con- ference on Intelligent User Interfaces, 2018.
  6. Sebastian Deterding, Jonathan Hook, Rebecca Fiebrink, Marco Gillies, Jeremy Gow, Memo Akten, Gillian Smith, Antonios Liapis, and Kate Compton. Mixed-initiative creative interfaces. In CHI 2017 Extended Abstracts.
  7. Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020. · arXiv
  8. Sander Dieleman, Aaron van den Oord, and Karen Si- monyan. The challenge of realistic music generation: modelling raw audio at scale. In Advances in Neural Information Processing Systems.
  9. Chris Donahue, Ian Simon, and Sander Dieleman. Pi- ano Genie. In Proceedings of the International Confer- ence on Intelligent User Interfaces, 2019.
  10. Judith E Fan, Monica Dinculescu, and David Ha. col- labdraw: An environment for collaborative sketching with an artificial agent. In Proceedings of the 2019 on Creativity and Cognition.
  11. Mary Farbood and Bernd Schöner. Analysis and syn- thesis of Palestrina-style counterpoint using markov chains. In Proceedings of the International Computer Music Conference, 2001.
  12. Jose D Fernández and Francisco Vico. AI methods in algorithmic composition: A comprehensive survey. Journal of Artificial Intelligence Research, 48.
  13. Rebecca Anne Fiebrink. Real-time human interaction with supervised learning algorithms for music compo- sition and performance. PhD dissertation, Princeton University, 2011.
  14. Emma Frid, Celso Gomes, and Zeyu Jin. Music cre- ation by example. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-13, 2020.
  15. Werner Geyer, Lydia B. Chilton, Ranjitha Kumar, and Adam Tauman Kalai. Workshop on human-AI co- creation with generative models. In Proceedings of the 25th International Conference on Intelligent User In- terfaces Companion, 2020.
  16. Gaëtan Hadjeres, François Pachet, and Frank Nielsen. DeepBach: a steerable model for Bach chorales gener- ation. In International Conference on Machine Learn- ing, pages 1362-1371, 2017.
  17. Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the maestro dataset. arXiv preprint arXiv:1810.12247, 2018. · arXiv
  18. Dorien Herremans, Ching-Hua Chuan, and Elaine Chew. A functional taxonomy of music generation sys- tems. ACM Computing Surveys (CSUR), 50(5), 2017.
  19. Dorien Herremans and Kenneth Sörensen. Composing fifth species counterpoint music with a variable neigh- borhood search algorithm. Expert systems with appli- cations, 40(16):6427-6437, 2013.
  20. Eric Horvitz. Principles of mixed-initiative user inter- faces. In SIGCHI Conference on Human Factors in Computing Systems.
  21. Cheng-Zhi Anna Huang, Sherol Chen, Mark Nelson, and Doug Eck. Mixed-initiative generation of multi- channel sequential structures. In International Con- ference on Learning Representations Workshop Track, 2018.
  22. Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron Courville, and Doug Eck. Counterpoint by convolution. In Proceedings of the International Conference on Music Information Retrieval, 2017.
  23. Cheng-Zhi Anna Huang, David Duvenaud, Kenneth C. Arnold, Brenton Partridge, Josiah W. Oberholtzer, and Krzysztof Z. Gajos. Active learning of intuitive con- trol knobs for synthesizers using gaussian processes. In Proceedings of the 19th International Conference on Intelligent User Interfaces, IUI, 2014.
  24. Cheng-Zhi Anna Huang, David Duvenaud, and Krzysztof Z Gajos. Chordripple: Recommending chords to help novice composers go beyond the ordi- nary. In Proceedings of the International Conference on Intelligent User Interfaces, 2016.
  25. Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Mon- ica Dinculescu, and Douglas Eck. Music transformer. In International Conference on Learning Representa- tions, 2019.
  26. Yu-Siang Huang and Yi-Hsuan Yang. Pop music trans- former: Generating music with rhythm and harmony. arXiv preprint arXiv:2002.00212, 2020. · arXiv
  27. Daphne Ippolito, Anna Huang, Curtis Hawthorne, and Douglas Eck. Infilling piano performances. In NIPS Workshop on Machine Learning for Creativity and De- sign, 2018.
  28. Hendrik Vincent Koops, José Pedro Magalhães, and W. Bas de Haas. A functional approach to automatic melody harmonisation. In Proceedings of the First ACM SIGPLAN Workshop on Functional Art, Music, Modeling and Design. Association for Computing Ma- chinery, 2013.
  29. Antonios Liapis, Georgios N. Yannakakis, Constantine Alexopoulos, and Phil Lopes. Can computers foster hu- man users' creativity? Theory and praxis of mixed- initiative co-creativity. Digital Culture & Education, 8(2):136-153, 2016.
  30. Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J. Cai. Novice-AI music co-creation via AI-steering tools for deep generative models. In Proceedings of the CHI Conference on Human Factors in Computing Systems, 2020.
  31. Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplernn: An un- conditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.07837, 2016. · arXiv
  32. Eric Nichols, Dan Morris, and Sumit Basu. Data- driven exploration of musical chord sequences. In Pro- ceedings of the international conference on Intelligent user interfaces, 2009.
  33. Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. I lead, you help but only with enough details: Under- standing user experience of co-creation with artificial intelligence. In Proceedings of the Conference on Human Factors in Computing Systems, CHI '18, 2018.
  34. François Pachet and Pierre Roy. Musical harmoniza- tion with constraints: A survey. Constraints, 6(1):7-19, 2001.
  35. Alexandre Papadopoulos, Pierre Roy, and François Pa- chet. Assisted lead sheet composition using FlowCom- poser. In International Conference on Principles and Practice of Constraint Programming. Springer, 2016.
  36. George Papadopoulos and Geraint Wiggins. AI meth- ods for algorithmic composition: A survey, a critical view and future prospects. In AISB Symposium on Mu- sical Creativity, volume 124, 1999.
  37. Philippe Pasquier, Arne Eigenfeldt, Oliver Bown, and Shlomo Dubnov. An introduction to musical metacre- ation. Computers in Entertainment (CIE), 14(2):2, 2016.
  38. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language mod- els are unsupervised multitask learners. OpenAI Blog, 1(8):9, 2019.
  39. Adam Roberts and Jesse Engel. Hierarchical varia- tional autoencoders for music. In Proceedings NIPS machine learning for creativity and design workshop, 2017.
  40. Adam Roberts, Jesse Engel, Sageev Oore, and Douglas Eck. Learning latent representations of music to gen- erate interactive musical palettes. In IUI Workshops, 2018.
  41. Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. A hierarchical latent vector model for learning long-term structure in music. ICML, 2018.
  42. Ian Simon, Dan Morris, and Sumit Basu. MySong: au- tomatic accompaniment generation for vocal melodies. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 2008.
  43. Ian Simon, Adam Roberts, Colin Raffel, Jesse En- gel, Curtis Hawthorne, and Douglas Eck. Learning a latent space of multitrack measures. arXiv preprint arXiv:1806.00195, 2018. · arXiv
  44. Bob L Sturm, Oded Ben-Tal, Una Monaghan, Nick Collins, Dorien Herremans, Elaine Chew, Gaëtan Had- jeres, Emmanuel Deruty, and François Pachet. Ma- chine learning research that matters for music creation: A case study. Journal of New Music Research, 48(1), 2019.
  45. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016. · arXiv
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
  47. Yi Yu and Simon Canales. Conditional LSTM-GAN for melody generation from lyrics. arXiv preprint arXiv:1908.05551. · arXiv
  48. Yichao Zhou, Wei Chu, Sam Young, and Xin Chen. BandNet: A neural network-based, multi-instrument Beatles-style MIDI music composition machine. arXiv preprint arXiv:1812.07126, 2018. · arXiv

Cited by

Sorted by citations

Authors
Huang, C.-Z. A., Koops, H. V., Newton-Rex, E., Dinculescu, M., & Cai, C. J.
Venue
arXiv
Publisher
arXiv
Published
2020
DOI
10.48550/arxiv.2010.05388
arXiv
2010.05388
References
0
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review