Music Creation by Example
Open accessAbstract
Short online videos have become the dominating media on social platforms. However, finding suitable music to accompany videos can be a challenging task to some video creators, due to copyright constraints, limitations in search engines, and required audio-editing expertise. One possible solution to these problems is to use AI music generation. In this paper we present a user interface (UI) paradigm that allows users to input a song to an AI engine and then interactively regenerate and mix AI-generated music. To arrive at this design, we conducted user studies with a total of 104 video creators at several stages of our design and development process. User studies supported the effectiveness of our approach and provided valuable insights about human-AI interaction as well as the design and evaluation of mixed-initiative interfaces in creative practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Recent studies mainly focus on how generative co-creation methodologies can be used for music performance, sound design, media art, or visual communication.
“Recent studies mainly focus on issues about how generative co-creation methodologies can be used for music performance [26], sound design [23], media art [5], or visual communication [4].”
From Exploring Designers’ Perceptions and Practices of Collaborating with Generative AI as a Co-creative Agent in a Multi-stakeholder Design Process: Take the Domain of Avatar Design as an Example (2023) · cites this paper as [26]
The Conclusions (page unavailable) describe a system in which users collaborate with AI to create and mix generated music. But the Discussion identifies its application as creating music for videos, not music performance; the cited paper therefore supports the generative co-creation aspect but not the application attributed to it. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · see citing works
Allen, J. E., Guinn, C. I., & Horvtz, E. (1999) · IEEE Intelligent Systems · doi:10.1109/5254.796083 · Cited by 442 · see citing works
MelNet: a generative model for audio in the frequency domain
Vasquez, S., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1906.01083 · Cited by 107 · see citing works
Computational Creativity: Coming of Age
Colton, S., de Mántaras, R. L., & Stock, O. (2009) · AI Magazine · doi:10.1609/aimag.v30i3.2257 · Cited by 92 · cited 3× · see citing works
“The term “computational creativity” refers to the study of building software that exhibits behavior that can be deemed creative in humans [18].”
BackgroundInfluential
Flagged first, then by citations
Research works printed but not linked: 42
- Kat Agres, Jamie Forth, and Geraint A Wiggins. 2016. Evaluation of Musical Creativity and Musical Metacreation Systems. Computers in Entertainment (CIE) 14, 3 (2016), 3.
- Vanessa Theme Ament. 2014. The Foley Grail: The Art of Performing Sound for Film, Games, and Animation. Routledge, Burlingto, MA, USA.
- Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, and others. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, ACM, New York, NY, USA, 3.
- Christopher Ariza. 2009. The Interrogator as Critic: The Turing Test and the Evaluation of Generative Music Systems. Computer Music Journal 33, 2 (2009), 48-70.
- Gérard Assayag, Camilo Rueda, Mikael Laurson, Carlos Agon, and Olivier Delerue. 1999. Computer-Assisted Composition at IRCAM: From PatchWork to OpenMusic. Computer Music Journal 23, 3 (1999), 59-72.
- Francisco Bernardo, Michael Zbyszynski, Rebecca Fiebrink, and Mick Grierson. 2017. Interactive Machine Learning for End-User Innovation. In 2017 AAAI Spring Symposium Series. The AAAI Press, Palo Alto, CA, USA, 369-375.
- J Bharucha. 1988. Neural Net Modeling of Music. In Proceedings of the First Workshop on Artificial Intelligence and Music. American Association for Artifical Intelligence, Menlo Park, CA, USA, 173-182.
- Margaret A. Boden. 1998a. Creativity and Artificial Intelligence. Artificial Intelligence 103, 1 (1998), 347 - 356. http://www.sciencedirect.com/science/article/ pii/S0004370298000551 Artificial Intelligence 40 Years Later.
- Margaret A Boden. 1998b. Creativity and Artificial Intelligence. Artificial Intelligence 103, 1-2 (1998), 347-356.
- Kris Bryden. 2007. Developing Metrics for Evolving a Musically Satisfying Result from a Population of Computer Generated Music Compositions. In 2007 IEEE Symposium on Computational Intelligence in Image and Signal Processing. IEEE, Honolulu, HI, USA, 171-176.
- Elizabeth Charters. 2003. The Use of Think-Aloud Methods in Qualitative Research -An Introduction to Think-Aloud Methods. Brock Education: A Journal of Educational Research and Practice 12, 2 (2003), 68-82.
- David Cope. 1989. Experiments in Musical Intelligence (EMI): Non-Linear Linguistic-Based Composition. Journal of New Music Research 18, 1-2 (1989), 117-139.
- David Cope. 2000. The Algorithmic Composer. Vol. 16
- Ramon Lopez De Mantaras and Josep Lluis Arcos. 2002. AI and Music: From Composition to Expressive Performance. AI magazine 23, 3 (2002), 43-43.
- Steven P Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L Schwartz, and Scott R Klemmer. 2010. Parallel Prototyping Leads to Better Design Results, More Divergence, and Increased Self-Efficacy. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 4 (2010), 18.
- Jose D Fernández and Francisco Vico. 2013. AI Methods in Algorithmic Composition: A Comprehensive Survey. Journal of Artificial Intelligence Research 48 (2013), 513-582.
- Rebecca Fiebrink, Baptiste Caramiaux, R Dean, and A McLean. 2018. The Machine Learning Algorithm as Creative Musical Tool. Oxford University Press, New York, NY, USA, Chapter 12, 181-208.
- Dorien Herremans and Elaine Chew. 2017. MorpheuS: Generating Structured Music with Constrained Patterns and Tension. IEEE Transactions on Affective Computing (2017), 510-523.
- Bruce L. Jacob. 1996. Algorithmic Composition as a Model of Creativity. Organised Sound 1, 3 (1996), 157-165.
- Lilong Jiang, Protiva Rahman, and Arnab Nandi. 2018. Evaluating Interactive Data Systems: Workloads, Metrics, and Guidelines. In Proceedings of the 2018 International Conference on Management of Data. ACM, ACM, New York, NY, USA, 1637-1644.
- Simon C Jones and Thomas G Schumacher. 1992. Muzak: On Functional Music and Power. Critical Studies in Media Communication 9, 2 (1992), 156-169.
- Yuli Levtov. 2018. Algorithmic Music for Mass Consumption and Universal Production. In The Oxford Handbook of Algorithmic Music. Oxford University Press, New York, NY, USA.
- Jen-Chun Lin, Wen-Li Wei, and Hsin-Min Wang. 2015. EMV-Matchmaker: Emotional Temporal course Modeling and Matching for Automatic Music Video Generation. In Proceedings of the 23rd ACM International conference on Multimedia. ACM, ACM, New York, NY, USA, 899-902.
- Jen-Chun Lin, Wen-Li Wei, and Hsin-Min Wang. 2016. DEMV-matchmaker: Emotional Temporal Course Representation and Deep Similarity Matching for Automatic Music Video Generation. In Proceedings of ICASSP. IEEE, Shanghai, China, 2772-2776.
- Jen-Chun Lin, Wen-Li Wei, James Yang, Hsin-Min Wang, and Hong-Yuan Mark Liao. 2017. Automatic Music Video Generation Based on Simultaneous Soundtrack Recommendation and Video Editing. In Proceedings of the 25th ACM international conference on Multimedia. ACM, ACM, New York, NY, USA, 519-527.
- L Manovich. 2018. AI Aesthetics. Strelka Press, Moscow, Russia.
- Jon McCormack and Mark d'Inverno. 2014. On the Future of Computers and Creativity. In AISB 2014 Symposium on Computational Creativity. Society for the Study of Artificial Intelligence and the Simulation of Behaviour, London, UK, 9.
- Alex McLean and Roger T Dean. 2018a. Musical Algorithms as Tools, Languages, and Partners: A Perspective. Oxford University Press, New Yor, NY, USA, 3-15.
- Alex McLean and Roger T Dean. 2018b. The Oxford Handbook of Algorithmic Music. Oxford University Press, New York, NY, USA.
- Eduardo Reck Miranda. 2013. Readings in Music and Artificial Intelligence. Routledge, New York and London.
- Gerhard Nierhaus. 2009. Algorithmic Composition: Paradigms of Automated Music Generation. Springer Science & Business Media, New York, NY, USA.
- NIME. 2019. The International Conference on New Interfaces for Musical Expression. (2019). https://www.nime.org/.
- Alexandre Papadopoulos, Pierre Roy, and François Pachet. 2016. Assisted Lead Sheet Composition Using FlowComposer. In Proceedings of International Conference on Principles and Practice of Constraint Programming. Springer, Toulouse, France, 769-785.
- George Papadopoulos and Geraint Wiggins. 1999. AI Methods for Algorithmic Composition: A Survey, a Critical View and Future Prospects. In AISB Symposium on Musical Creativity, Vol. 124. Edinburgh, UK, 110-117.
- Richard Scheines. 1988. Automating Creativity. In Aspects of Artificial Intelligence. Springer, 339-365.
- Adriana Schulz, Ariel Shamir, David IW Levin, Pitchaya Sitthi-Amorn, and Wojciech Matusik. 2014. Design and Fabrication by Example. ACM Transactions on Graphics (TOG) 33, 4 (2014), 62.
- Rajiv Ratn Shah, Yi Yu, and Roger Zimmermann. 2014. Advisor: Personalized Video Soundtrack Recommendation by Late Fusion with Heuristic Rankings. In Proceedings of the 22nd ACM international conference on Multimedia. ACM, ACM, New York, NY, USA, 607-616.
- Kristina Shea, Robert Aish, and Marina Gourtovaia. 2005. Towards Integrated Performance-Driven Generative Design Tools. Automation in Construction 14, 2 (2005), 253-264.
- Thomas Strothotte and Stefan Schlechtweg. 2002. Non-Photorealistic Computer Graphics: Modeling, Rendering, and Animation. Morgan Kaufmann, San Francisco, CA, USA.
- Ju-Chiang Wang, Yi-Hsuan Yang, I-Hong Jhuo, Yen-Yu Lin, Hsin-Min Wang, and others. 2012. The Acousticvisual Emotion Guassians Model for Automatic Generation of Music Video. In Proceedings of the 20th ACM international conference on Multimedia. ACM, ACM, New York, NY, USA, 1379-1380.
- Ariel Weingarten, Ben Lafreniere, George Fitzmaurice, and Tovi Grossman. 2019. DreamRooms: Prototyping Rooms in Collaboration with a Generative Process. In Proceedings of Graphics Interface 2019 (GI 2019). Canadian Information Processing Society, Mississauga, Ontario, Canada, 9.
- Iannis Xenakis. 1992. Formalized Music: Thought and Mathematics in Composition. Pendragon Press, Stuvesant, NY, USA.
Cited by
Sorted by citations
Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189 · see citing works
Kim, T. S., Lee, Y., Chang, M., & Kim, J. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606833 · Cited by 56 · see citing works
Kamath, P., Morreale, F., Bagaskara, P. L., Wei, Y., & Nanayakkara, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642040 · Cited by 37 · see citing works
Understanding Human-AI Collaboration in Music Therapy Through Co-Design with Therapists
Sun, J., Yang, J., Zhou, G., Jin, Y., & Gong, J. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642764 · Cited by 33 · see citing works
He, Q., Zheng, W., Bao, H., Chen, R., & Tong, X. (2023) · Proceedings of the Eleventh International Symposium of Chinese CHI · doi:10.1145/3629606.3629675 · Cited by 15 · see citing works
Novice-Centered Application Design for Music Creation
Kobayashi, A., Sato, T., & Tateno, K. (2024) · Designing Interactive Systems Conference · doi:10.1145/3656156.3663695 · Cited by 5 · see citing works
Fotaris, P. (2026) · Cogent Arts & Humanities · doi:10.1080/23311983.2026.2722084
- Authors
- Frid, E., Gomes, C., & Jin, Z.
- Affiliations
- KTH Royal Institute of Technology (SE)
- Adobe Systems (United States) (US)
- ORCID
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-13
- OpenAlex
- W3005487936
- Cited by
- 58
- References
- 4
- Citations by year
- 2026: 6 · 2025: 11 · 2024: 12 · 2023: 6 · 2022: 13 · 2021: 8 · 2020: 2
- FWCI
- 2.21
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, public-domain)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Music Technology and Sound Studies
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Keywords
- AI music generation, music mixing, video creators, human-AI interaction, user interface design, creative practice, user study
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
