Learning Embodied Sound-Motion Mappings: Evaluating AI-Generated Dance Improvisation
The DOI matches its Crossref record; 26 of the 27 references it cites were found in a registry or an index.
Abstract
Through dance, a wide range of emotions can be expressed. As virtual agents and robots continue to become part of our daily lives, the need for them to efficiently convey emotion and intent increases. When trained to dance, to what extent can AI learn to model the tacit mappings between sound and motion? Here, we explore the creative capacity of a generative model trained on 3D motion capture recordings of improvised dance. We perform a perceptual judgment experiment wherein respondents rate movement generated by our model as well as human performances. While the sound-motion mappings remain somewhat elusive, particularly when compared to examples of human dance, our study shows that in certain aspects related to perceived dance-likeness and expressivity, the model successfully mimics human dance movement. By employing a perceptual study to evaluate our generative model, we aim to further our ability to understand the affordances and limitations of creative AI.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Although recent work has begun to probe at design considerations for generative AI, this work has been limited to specific application domains or technologies. For example, guidelines of various maturity levels exist for GAN-based interfaces [57, 195], image creation [102, 104, 173], prompt engineering [102, 104], virtual reality [169], collaborative storytelling [146], and workflows with co-crea”
From Design Principles for Generative AI Applications (2024) · cites this paper as [173]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Social psychology of creativity: A consensual assessment technique.
Amabile, T. M. (1982) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.43.5.997 · Cited by 2,278
Perceiving affect from arm movement
Pollick, F. E., Paterson, H. M., Bruderlin, A., & Sanford, A. J. (2001) · Cognition · doi:10.1016/s0010-0277(01)00147-0 · Cited by 616
Point-Light Biological Motion Perception Activates Human Premotor Cortex
Saygin, A. P., Wilson, S. M., Hagler, D. J., Bates, E., & Sereno, M. I. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.0504-04.2004 · Cited by 413
A Comparison of Expert and Nonexpert Raters Using the Consensual Assessment Technique
Kaufman, J. C., Baer, J., Cole, J. C., & Sexton∗, J. D. (2008) · Creativity Research Journal · doi:10.1080/10400410802059929 · Cited by 374
Some Empirical Criteria for Attributing Creativity to a Computer Program
Ritchie, G. (2007) · Minds and Machines · doi:10.1007/s11023-007-9066-2 · Cited by 286
Harvey, F. G., Yurick, M., Nowrouzezahrai, D., & Pal, C. (2020) · ACM Transactions on Graphics · doi:10.1145/3386569.3392480 · Cited by 266
Nonverbal Behavior Generator for Embodied Conversational Agents
Lee, J., & Marsella, S. (2006) · Lecture Notes in Computer Science · doi:10.1007/11821830_20 · Cited by 238
Burger, B., Thompson, M. R., Luck, G., Saarikallio, S., & Toiviainen, P. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00183 · Cited by 196
Dance with Melody: An LSTM-autoencoder Approach to Music-oriented Dance Synthesis
Tang, T., Jia, J., & Mao, H. (2018) · Proceedings of the 26th ACM international conference on Multimedia · doi:10.1145/3240508.3240526 · Cited by 158
Research works printed but not linked: 19
- Birgitta Burger and Petri Toiviainen . 2013 . MoCap Toolbox-A Matlab toolbox for computational analysis of movement data . In 10th Sound and Music Computing Conference, SMC 2013, Stockholm, Sweden. Logos Verlag Berlin. Birgitta Burger and Petri Toiviainen. 2013. MoCap Toolbox-A Matlab toolbox for computational analysis of movement data. In 10th Sound and Music Computing Conference, SMC 2013, Stockholm, Sweden. Logos Verlag Berlin.
- Birgitta Burger and Petri Toiviainen . 2020. See How It Feels to Move: Relationships Between Movement Characteristics and Perception of Emotions in Dance. Human Technology 16, 3 ( 2020 ). Birgitta Burger and Petri Toiviainen. 2020. See How It Feels to Move: Relationships Between Movement Characteristics and Perception of Emotions in Dance. Human Technology 16, 3 (2020).
- Baptiste Caramiaux , Frédéric Bevilacqua , and Norbert Schnell . 2010. Towards a Gesture-Sound Cross-Modal Analysis . In Gesture in Embodied Communication and Human-Computer Interaction , Stefan Koppand Ipke Wachsmuth (Eds.). Springer , Berlin Heidelberg , 158–170. Baptiste Caramiaux, Frédéric Bevilacqua, and Norbert Schnell. 2010. Towards a Gesture-Sound Cross-Modal Analysis. In Gesture in Embodied Communication and Human-Computer Interaction, Stefan Koppand Ipke Wachsmuth (Eds.). Springer, Berlin Heidelberg, 158–170.
- Antonio Daniele , Caroline Di Bernardi Luft , and Nick Bryan-Kinns . 2021. “ What Is Human? ” A Turing Test for Artistic Creativity. In Artificial Intelligence in Music, Sound, Art and Design, Juan Romero, Tiago Martins, and Nereida Rodríguez-Fernández (Eds.) . Springer International Publishing , Cham , 396–411. Antonio Daniele, Caroline Di Bernardi Luft, and Nick Bryan-Kinns. 2021. “What Is Human?” A Turing Test for Artistic Creativity. In Artificial Intelligence in Music, Sound, Art and Design, Juan Romero, Tiago Martins, and Nereida Rodríguez-Fernández (Eds.). Springer International Publishing, Cham, 396–411.
- Ronald Fischer and Taciano L Milfont . 2010. Standardization in psychological research.International Journal of Psychological Research 3, 1 ( 2010 ), 88–96. Ronald Fischer and Taciano L Milfont. 2010. Standardization in psychological research.International Journal of Psychological Research 3, 1 (2010), 88–96.
- J Françoise , P Pasquier , O Alemi , J Françoise , and P Pasquier . 2017 . GrooveNet: real-time music-driven dance movement generation using artificial neural networks . In Proceedings of SIGKDD 2017 Workshop on Machine Learning for Creativity (Nova Scotia), Vol. 6. J Françoise, P Pasquier, O Alemi, J Françoise, and P Pasquier. 2017. GrooveNet: real-time music-driven dance movement generation using artificial neural networks. In Proceedings of SIGKDD 2017 Workshop on Machine Learning for Creativity (Nova Scotia), Vol. 6.
- Rolf Inge Godøy , Egil Haga , and Alexander Refsum Jensenius . 2006 . Exploring music-related gestures by sound-tracing: A preliminary study . In Proceedings of the COST287-ConGAS 2nd International Symposium on Gesture Interfaces for Multimedia Systems (GIMS2006) . 27–33. Rolf Inge Godøy, Egil Haga, and Alexander Refsum Jensenius. 2006. Exploring music-related gestures by sound-tracing: A preliminary study. In Proceedings of the COST287-ConGAS 2nd International Symposium on Gesture Interfaces for Multimedia Systems (GIMS2006). 27–33.
- Victor E Gonzalez-Sanchez , Agata Zelechowska , and Alexander Refsum Jensenius . 2018. Correspondences between music and involuntary human micromotion during standstill. Frontiers in psychology 9 ( 2018 ), 1382. Victor E Gonzalez-Sanchez, Agata Zelechowska, and Alexander Refsum Jensenius. 2018. Correspondences between music and involuntary human micromotion during standstill. Frontiers in psychology 9 (2018), 1382.
- Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850(2013). Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850(2013). · arXiv
- David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017). David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017). · arXiv
- Lou E Hicks . 1970. Some properties of ipsative, normative, and forced-choice normative measures.Psychological bulletin 74, 3 ( 1970 ), 167. Lou E Hicks. 1970. Some properties of ipsative, normative, and forced-choice normative measures.Psychological bulletin 74, 3 (1970), 167.
- Alexander Refsum Jensenius , Agata Zelechowska , and Victor Evaristo Gonzalez Sanchez . 2017 . The musical influence on people’s micromotion when standing still in groups . In Proceedings of the SMC Conferences . Aalto University, 195–200. Alexander Refsum Jensenius, Agata Zelechowska, and Victor Evaristo Gonzalez Sanchez. 2017. The musical influence on people’s micromotion when standing still in groups. In Proceedings of the SMC Conferences. Aalto University, 195–200.
- Tejaswinee Kelkar and Alexander Refsum Jensenius . 2018. Analyzing Free-Hand Sound-Tracings of Melodic Phrases. Applied Sciences 8, 1 ( 2018 ). Tejaswinee Kelkar and Alexander Refsum Jensenius. 2018. Analyzing Free-Hand Sound-Tracings of Melodic Phrases. Applied Sciences 8, 1 (2018).
- Diederik P Kingma and Jimmy Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014). Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014). · arXiv
- Meha Kumar , Duri Long , and Brian Magerko . 2020 . Creativity Metrics for a Lead-and-Follow Dynamic in an Improvisational Dance Agent . In Proceedings of the Eleventh International Conference on Computational Creativity. Meha Kumar, Duri Long, and Brian Magerko. 2020. Creativity Metrics for a Lead-and-Follow Dynamic in an Improvisational Dance Agent. In Proceedings of the Eleventh International Conference on Computational Creativity.
- Harry McGurk and John MacDonald . 1976. Hearing lips and seeing voices. Nature 264, 5588 ( 1976 ), 746–748. Harry McGurk and John MacDonald. 1976. Hearing lips and seeing voices. Nature 264, 5588 (1976), 746–748.
- Johannes Michalak , Nikolaus F Troje , Julia Fischer , Patrick Vollmar , Thomas Heidenreich , and Dietmar Schulte . 2009. Embodiment of sadness and depression—gait patterns associated with dysphoric mood. Psychosomatic medicine 71, 5 ( 2009 ), 580–587. Johannes Michalak, Nikolaus F Troje, Julia Fischer, Patrick Vollmar, Thomas Heidenreich, and Dietmar Schulte. 2009. Embodiment of sadness and depression—gait patterns associated with dysphoric mood. Psychosomatic medicine 71, 5 (2009), 580–587.
- Nick Neave , Kristofor McCarty , Jeanette Freynik , Nicholas Caplan , Johannes Hönekopp , and Bernhard Fink . 2011. Male dance moves that catch a woman’s eye. Biology letters 7, 2 ( 2011 ), 221–224. Nick Neave, Kristofor McCarty, Jeanette Freynik, Nicholas Caplan, Johannes Hönekopp, and Bernhard Fink. 2011. Male dance moves that catch a woman’s eye. Biology letters 7, 2 (2011), 221–224.
- Benedikte Wallace , Charles P. Martin , Jim Torresen , and Kristian Nymoen . 2020 . Towards Movement Generation with Audio Features . In Proceedings of the Eleventh International Conference on Computational Creativity. Benedikte Wallace, Charles P. Martin, Jim Torresen, and Kristian Nymoen. 2020. Towards Movement Generation with Audio Features. In Proceedings of the Eleventh International Conference on Computational Creativity.
Cited by
Design Principles for Generative AI Applications
Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R., & Geyer, W. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642466 · Cited by 232
“…to generative AI systems allow users to specify their intent via sketches and gestures [32], UI controls [103, 105], video-scanned body movements [174], and even various forms of improv performance [15, 68]. use cases in which false positives and false negatives are important outcome metrics; in…”
BackgroundLarge-scale Text-to-Image Generation Models for Visual Artists’ Creative Works
Ko, H.-K., Park, G., Jeon, H., Jo, J., Kim, J., & Seo, J. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584078 · Cited by 189
When Creators Meet the Metaverse: A Survey on Computational ArtsPreprint
Lee, L., Lin, Z., Hu, R., Gong, Z., Kumar, A., Li, T., Li, S., & Hui, P. (2021) · arXiv (Cornell University) · doi:10.13140/rg.2.2.36609.17761/2 · Cited by 160
When and How to Use AI in the Design Process? Implications for Human-AI Design Collaboration
Lee, S.-Y., Law, M., & Hoffman, G. (2025) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2353451 · Cited by 97
Breaking from realism: exploring the potential of glitch in AI-generated dance
Wallace, B., Nymoen, K., Tørresen, J., & Martin, C. P. (2024) · Digital Creativity · doi:10.1080/14626268.2024.2327006 · Cited by 32
Self-citation
Metadata
- Authors
- Wallace, B., Martin, C. P., Tørresen, J., & Nymoen, K.
- Affiliations
- University of Oslo (NO)
- Australian National University (AU)
- Venue
- Creativity and Cognition
- Publisher
- ACM
- Published
- 2021-06-22
- Pages
- 1-9
- OpenAlex
- W3176390735
- Cited by
- 13
- References
- 27
- Citations by year
- 2026: 2 · 2024: 6 · 2023: 3 · 2022: 1 · 2021: 1
- FWCI
- 4.10
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Creativity and Cognition (publishedVersion)
- ANU Open Research (Australian National University) (submittedVersion)
- Funders
- Research Council of Norway (262762)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- dance improvisation, embodied AI, generative models, 3D motion capture, virtual agents, robot emotion expression, expressivity, perceptual evaluation, creative AI, tacit knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Learning Embodied Sound-Motion Mappings: Evaluating AI-Generated Dance Improvisation. https://app.cochat.ai/papers/10.1145/3450741.3465245
- The paper
- Wallace, B., Martin, C. P., Tørresen, J., & Nymoen, K. (2021). Learning Embodied Sound-Motion Mappings: Evaluating AI-Generated Dance Improvisation. Creativity and Cognition. https://doi.org/10.1145/3450741.3465245
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.