Learning Embodied Sound-Motion Mappings: Evaluating AI-Generated Dance Improvisation
The DOI matches its Crossref record; 26 of the 27 references it cites were found in a registry or an index.
Abstract
Through dance, a wide range of emotions can be expressed. As virtual agents and robots continue to become part of our daily lives, the need for them to efficiently convey emotion and intent increases. When trained to dance, to what extent can AI learn to model the tacit mappings between sound and motion? Here, we explore the creative capacity of a generative model trained on 3D motion capture recordings of improvised dance. We perform a perceptual judgment experiment wherein respondents rate movement generated by our model as well as human performances. While the sound-motion mappings remain somewhat elusive, particularly when compared to examples of human dance, our study shows that in certain aspects related to perceived dance-likeness and expressivity, the model successfully mimics human dance movement. By employing a perceptual study to evaluate our generative model, we aim to further our ability to understand the affordances and limitations of creative AI.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Although recent work has begun to probe at design considerations for generative AI, this work has been limited to specific application domains or technologies. For example, guidelines of various maturity levels exist for GAN-based interfaces [57, 195], image creation [102, 104, 173], prompt engineering [102, 104], virtual reality [169], collaborative storytelling [146], and workflows with co-crea”
From Design Principles for Generative AI Applications (2024) · cites this paper as [173]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Social psychology of creativity: A consensual assessment technique.
Amabile, T. M. (1982) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.43.5.997 · Cited by 2,278
Perceiving affect from arm movement
Pollick, F. E., Paterson, H. M., Bruderlin, A., & Sanford, A. J. (2001) · Cognition · doi:10.1016/s0010-0277(01)00147-0 · Cited by 616
Point-Light Biological Motion Perception Activates Human Premotor Cortex
Saygin, A. P., Wilson, S. M., Hagler, D. J., Bates, E., & Sereno, M. I. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.0504-04.2004 · Cited by 413
A Comparison of Expert and Nonexpert Raters Using the Consensual Assessment Technique
Kaufman, J. C., Baer, J., Cole, J. C., & Sexton∗, J. D. (2008) · Creativity Research Journal · doi:10.1080/10400410802059929 · Cited by 374 · cited 1×
“Gathering qualitative feedback, particularly from individuals with domainspecific knowledge, can shape the direction of research and inform better algorithm design [2, 23].”
BackgroundSome Empirical Criteria for Attributing Creativity to a Computer Program
Ritchie, G. (2007) · Minds and Machines · doi:10.1007/s11023-007-9066-2 · Cited by 286
Harvey, F. G., Yurick, M., Nowrouzezahrai, D., & Pal, C. (2020) · ACM Transactions on Graphics · doi:10.1145/3386569.3392480 · Cited by 266
Nonverbal Behavior Generator for Embodied Conversational Agents
Lee, J., & Marsella, S. (2006) · Lecture Notes in Computer Science · doi:10.1007/11821830_20 · Cited by 238
Burger, B., Thompson, M. R., Luck, G., Saarikallio, S., & Toiviainen, P. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00183 · Cited by 196
Dance with Melody: An LSTM-autoencoder Approach to Music-oriented Dance Synthesis
Tang, T., Jia, J., & Mao, H. (2018) · Proceedings of the 26th ACM international conference on Multimedia · doi:10.1145/3240508.3240526 · Cited by 158
The Impact of the Bass Drum on Human Dance Movement
Van Dyck, E., Moelants, D., Demey, M., Deweppe, A., Coussement, P., & Leman, M. (2013) · Music Perception · doi:10.1525/mp.2013.30.4.349 · Cited by 147
Exploring Perceptual and Acoustical Correlates of Polyphonic Timbre
Alluri, V., & Toiviainen, P. (2010) · Music Perception An Interdisciplinary Journal · doi:10.1525/mp.2010.27.3.223 · Cited by 134
What Is Embodied Music Cognition?
Leman, M., Maes, P., Nijs, L., & Dyck, E. V. (2018) · Springer handbooks · doi:10.1007/978-3-662-55004-5_34 · Cited by 117
MULTI-FEATURE MODELING OF PULSE CLARITY: DESIGN, VALIDATION AND OPTIMIZATION
Lartillot, O., Eerola, T., Toiviainen, P., & Fornari, J. (2008) · doi:10.5072/zenodo.243404 · Cited by 106
Learn to Dance with AIST++: Music Conditioned 3D Dance GenerationPreprint
Li, R., Yang, S., Ross, D. A., & Kanazawa, A. (2021) · arXiv (Cornell University) · Cited by 88
(2010) · Lecture notes in computer science · doi:10.1007/978-3-642-12553-9 · Cited by 79
Evidence of Big Five and Aggressive Personalities in Gait Biomechanics
Satchell, L. P., Morris, P., Mills, C., O’Reilly, L., Marshman, P., & Akehurst, L. (2016) · Journal of Nonverbal Behavior · doi:10.1007/s10919-016-0240-1 · Cited by 58
Sturm, B. L. T., & Ben‐Tal, O. (2017) · Journal of Creative Music Systems · doi:10.5920/jcms.2017.09 · Cited by 56
Analyzing correspondence between sound objects and body motion
Nymoen, K., Godøy, R. I., Jensenius, A. R., & Tørresen, J. (2013) · ACM Transactions on Applied Perception · doi:10.1145/2465780.2465783 · Cited by 54
Self-citationProceedings of the Seventh International Conference on Computational Creativity
Llano, M. T., Christian Guckelsberger, B., Hepworth, R., Gow, J., Corneli, J., & Colton, S. (2016) · Cited by 46
Dance Notations and Robot Motion
Laumond, J., & Abe, N. (2015) · Springer tracts in advanced robotics · doi:10.1007/978-3-319-25739-6 · Cited by 44
Recognizing Induced Emotions of Happiness and Sadness from Dance Movement
Dyck, E. V., Vansteenkiste, P., Lenoir, M., Lesaffre, M., & Leman, M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0089773 · Cited by 41
Bias in perception of art produced by artificial intelligence
Hong, J.-W. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-91244-8_24 · Cited by 39 · cited 1×
“Common methods include Turing test approaches, rating schemes and focus groups, revealing interesting insights into our potentially embodied responses to art [11], possible biases against AI [21] and how best to”
MethodCarlson, E. J., Saari, P., Burger, B., & Toiviainen, P. (2020) · Journal of New Music Research · doi:10.1080/09298215.2020.1711778 · Cited by 35
RoboJam: A Musical Mixture Density Network for Collaborative Touchscreen Interaction
Martin, C. P., & Torresen, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-77583-8_11 · Cited by 17
Self-citationUnderstanding Musical Predictions With an Embodied Interface for Musical Machine Learning
Martin, C. P., Glette, K., Nygaard, T. F., & Tørresen, J. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00006 · Cited by 3
Self-citation
Research works printed but not linked: 19
- Birgitta Burger and Petri Toiviainen . 2013 . MoCap Toolbox-A Matlab toolbox for computational analysis of movement data . In 10th Sound and Music Computing Conference, SMC 2013, Stockholm, Sweden. Logos Verlag Berlin. Birgitta Burger and Petri Toiviainen. 2013. MoCap Toolbox-A Matlab toolbox for computational analysis of movement data. In 10th Sound and Music Computing Conference, SMC 2013, Stockholm, Sweden. Logos Verlag Berlin.
- Birgitta Burger and Petri Toiviainen . 2020. See How It Feels to Move: Relationships Between Movement Characteristics and Perception of Emotions in Dance. Human Technology 16, 3 ( 2020 ). Birgitta Burger and Petri Toiviainen. 2020. See How It Feels to Move: Relationships Between Movement Characteristics and Perception of Emotions in Dance. Human Technology 16, 3 (2020).
- Baptiste Caramiaux , Frédéric Bevilacqua , and Norbert Schnell . 2010. Towards a Gesture-Sound Cross-Modal Analysis . In Gesture in Embodied Communication and Human-Computer Interaction , Stefan Koppand Ipke Wachsmuth (Eds.). Springer , Berlin Heidelberg , 158–170. Baptiste Caramiaux, Frédéric Bevilacqua, and Norbert Schnell. 2010. Towards a Gesture-Sound Cross-Modal Analysis. In Gesture in Embodied Communication and Human-Computer Interaction, Stefan Koppand Ipke Wachsmuth (Eds.). Springer, Berlin Heidelberg, 158–170.
- Antonio Daniele , Caroline Di Bernardi Luft , and Nick Bryan-Kinns . 2021. “ What Is Human? ” A Turing Test for Artistic Creativity. In Artificial Intelligence in Music, Sound, Art and Design, Juan Romero, Tiago Martins, and Nereida Rodríguez-Fernández (Eds.) . Springer International Publishing , Cham , 396–411. Antonio Daniele, Caroline Di Bernardi Luft, and Nick Bryan-Kinns. 2021. “What Is Human?” A Turing Test for Artistic Creativity. In Artificial Intelligence in Music, Sound, Art and Design, Juan Romero, Tiago Martins, and Nereida Rodríguez-Fernández (Eds.). Springer International Publishing, Cham, 396–411.
- Ronald Fischer and Taciano L Milfont . 2010. Standardization in psychological research.International Journal of Psychological Research 3, 1 ( 2010 ), 88–96. Ronald Fischer and Taciano L Milfont. 2010. Standardization in psychological research.International Journal of Psychological Research 3, 1 (2010), 88–96.
- J Françoise , P Pasquier , O Alemi , J Françoise , and P Pasquier . 2017 . GrooveNet: real-time music-driven dance movement generation using artificial neural networks . In Proceedings of SIGKDD 2017 Workshop on Machine Learning for Creativity (Nova Scotia), Vol. 6. J Françoise, P Pasquier, O Alemi, J Françoise, and P Pasquier. 2017. GrooveNet: real-time music-driven dance movement generation using artificial neural networks. In Proceedings of SIGKDD 2017 Workshop on Machine Learning for Creativity (Nova Scotia), Vol. 6.
- Rolf Inge Godøy , Egil Haga , and Alexander Refsum Jensenius . 2006 . Exploring music-related gestures by sound-tracing: A preliminary study . In Proceedings of the COST287-ConGAS 2nd International Symposium on Gesture Interfaces for Multimedia Systems (GIMS2006) . 27–33. Rolf Inge Godøy, Egil Haga, and Alexander Refsum Jensenius. 2006. Exploring music-related gestures by sound-tracing: A preliminary study. In Proceedings of the COST287-ConGAS 2nd International Symposium on Gesture Interfaces for Multimedia Systems (GIMS2006). 27–33.
- Victor E Gonzalez-Sanchez , Agata Zelechowska , and Alexander Refsum Jensenius . 2018. Correspondences between music and involuntary human micromotion during standstill. Frontiers in psychology 9 ( 2018 ), 1382. Victor E Gonzalez-Sanchez, Agata Zelechowska, and Alexander Refsum Jensenius. 2018. Correspondences between music and involuntary human micromotion during standstill. Frontiers in psychology 9 (2018), 1382.
- Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850(2013). Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850(2013). · arXiv
- David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017). David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017). · arXiv
- Lou E Hicks . 1970. Some properties of ipsative, normative, and forced-choice normative measures.Psychological bulletin 74, 3 ( 1970 ), 167. Lou E Hicks. 1970. Some properties of ipsative, normative, and forced-choice normative measures.Psychological bulletin 74, 3 (1970), 167.
- Alexander Refsum Jensenius , Agata Zelechowska , and Victor Evaristo Gonzalez Sanchez . 2017 . The musical influence on people’s micromotion when standing still in groups . In Proceedings of the SMC Conferences . Aalto University, 195–200. Alexander Refsum Jensenius, Agata Zelechowska, and Victor Evaristo Gonzalez Sanchez. 2017. The musical influence on people’s micromotion when standing still in groups. In Proceedings of the SMC Conferences. Aalto University, 195–200.
- Tejaswinee Kelkar and Alexander Refsum Jensenius . 2018. Analyzing Free-Hand Sound-Tracings of Melodic Phrases. Applied Sciences 8, 1 ( 2018 ). Tejaswinee Kelkar and Alexander Refsum Jensenius. 2018. Analyzing Free-Hand Sound-Tracings of Melodic Phrases. Applied Sciences 8, 1 (2018).
- Diederik P Kingma and Jimmy Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014). Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014). · arXiv
- Meha Kumar , Duri Long , and Brian Magerko . 2020 . Creativity Metrics for a Lead-and-Follow Dynamic in an Improvisational Dance Agent . In Proceedings of the Eleventh International Conference on Computational Creativity. Meha Kumar, Duri Long, and Brian Magerko. 2020. Creativity Metrics for a Lead-and-Follow Dynamic in an Improvisational Dance Agent. In Proceedings of the Eleventh International Conference on Computational Creativity.
- Harry McGurk and John MacDonald . 1976. Hearing lips and seeing voices. Nature 264, 5588 ( 1976 ), 746–748. Harry McGurk and John MacDonald. 1976. Hearing lips and seeing voices. Nature 264, 5588 (1976), 746–748.
- Johannes Michalak , Nikolaus F Troje , Julia Fischer , Patrick Vollmar , Thomas Heidenreich , and Dietmar Schulte . 2009. Embodiment of sadness and depression—gait patterns associated with dysphoric mood. Psychosomatic medicine 71, 5 ( 2009 ), 580–587. Johannes Michalak, Nikolaus F Troje, Julia Fischer, Patrick Vollmar, Thomas Heidenreich, and Dietmar Schulte. 2009. Embodiment of sadness and depression—gait patterns associated with dysphoric mood. Psychosomatic medicine 71, 5 (2009), 580–587.
- Nick Neave , Kristofor McCarty , Jeanette Freynik , Nicholas Caplan , Johannes Hönekopp , and Bernhard Fink . 2011. Male dance moves that catch a woman’s eye. Biology letters 7, 2 ( 2011 ), 221–224. Nick Neave, Kristofor McCarty, Jeanette Freynik, Nicholas Caplan, Johannes Hönekopp, and Bernhard Fink. 2011. Male dance moves that catch a woman’s eye. Biology letters 7, 2 (2011), 221–224.
- Benedikte Wallace , Charles P. Martin , Jim Torresen , and Kristian Nymoen . 2020 . Towards Movement Generation with Audio Features . In Proceedings of the Eleventh International Conference on Computational Creativity. Benedikte Wallace, Charles P. Martin, Jim Torresen, and Kristian Nymoen. 2020. Towards Movement Generation with Audio Features. In Proceedings of the Eleventh International Conference on Computational Creativity.
Cited by
Design Principles for Generative AI Applications
Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R., & Geyer, W. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642466 · Cited by 232
“…to generative AI systems allow users to specify their intent via sketches and gestures [32], UI controls [103, 105], video-scanned body movements [174], and even various forms of improv performance [15, 68]. use cases in which false positives and false negatives are important outcome metrics; in…”
BackgroundLarge-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189
When creators meet the metaverse: a survey on computational arts in user-centric perspectives
Lee, L., Li, S., Lin, Z., Hu, R., Gong, Z., Li, T., Kumar, A., & Hui, P. (2026) · Foundations and Trends® in Human–Computer Interaction · doi:10.13140/rg.2.2.36609.17761/2 · Cited by 161
When and How to Use AI in the Design Process? Implications for Human-AI Design Collaboration
Lee, S.-Y., Law, M., & Hoffman, G. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2353451 · Cited by 97
Breaking from realism: exploring the potential of glitch in AI-generated dance
Wallace, B., Nymoen, K., Tørresen, J., & Martin, C. P. (2024) · Digital Creativity · doi:10.1080/14626268.2024.2327006 · Cited by 32
Self-citation
Metadata
- Authors
- Wallace, B., Martin, C. P., Tørresen, J., & Nymoen, K.
- Affiliations
- University of Oslo (NO)
- Australian National University (AU)
- Venue
- Creativity and Cognition
- Publisher
- ACM
- Published
- 2021-06-22
- Pages
- 1-9
- OpenAlex
- W3176390735
- Cited by
- 13
- References
- 27
- Citations by year
- 2026: 2 · 2024: 6 · 2023: 3 · 2022: 1 · 2021: 1
- FWCI
- 4.10
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Creativity and Cognition (publishedVersion)
- ANU Open Research (Australian National University) (submittedVersion)
- Funders
- Research Council of Norway (262762)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- dance improvisation, embodied AI, generative models, 3D motion capture, virtual agents, robot emotion expression, expressivity, perceptual evaluation, creative AI, tacit knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.