CoChat

Learning Embodied Sound-Motion Mappings: Evaluating AI-Generated Dance Improvisation

The DOI matches its Crossref record; 26 of the 27 references it cites were found in a registry or an index.

Abstract

Through dance, a wide range of emotions can be expressed. As virtual agents and robots continue to become part of our daily lives, the need for them to efficiently convey emotion and intent increases. When trained to dance, to what extent can AI learn to model the tacit mappings between sound and motion? Here, we explore the creative capacity of a generative model trained on 3D motion capture recordings of improvised dance. We perform a perceptual judgment experiment wherein respondents rate movement generated by our model as well as human performances. While the sound-motion mappings remain somewhat elusive, particularly when compared to examples of human dance, our study shows that in certain aspects related to perceived dance-likeness and expressivity, the model successfully mimics human dance movement. By employing a perceptual study to evaluate our generative model, we aim to further our ability to understand the affordances and limitations of creative AI.

Overview

Integrity signals

Registry

Crossref · 10.1145/3450741.3465245

Cites

26 of 26 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 26 references

Research works printed but not linked: 19

  1. Birgitta Burger and Petri Toiviainen . 2013 . MoCap Toolbox-A Matlab toolbox for computational analysis of movement data . In 10th Sound and Music Computing Conference, SMC 2013, Stockholm, Sweden. Logos Verlag Berlin. Birgitta Burger and Petri Toiviainen. 2013. MoCap Toolbox-A Matlab toolbox for computational analysis of movement data. In 10th Sound and Music Computing Conference, SMC 2013, Stockholm, Sweden. Logos Verlag Berlin.
  2. Birgitta Burger and Petri Toiviainen . 2020. See How It Feels to Move: Relationships Between Movement Characteristics and Perception of Emotions in Dance. Human Technology 16, 3 ( 2020 ). Birgitta Burger and Petri Toiviainen. 2020. See How It Feels to Move: Relationships Between Movement Characteristics and Perception of Emotions in Dance. Human Technology 16, 3 (2020).
  3. Baptiste Caramiaux , Frédéric Bevilacqua , and Norbert Schnell . 2010. Towards a Gesture-Sound Cross-Modal Analysis . In Gesture in Embodied Communication and Human-Computer Interaction , Stefan Koppand Ipke Wachsmuth (Eds.). Springer , Berlin Heidelberg , 158–170. Baptiste Caramiaux, Frédéric Bevilacqua, and Norbert Schnell. 2010. Towards a Gesture-Sound Cross-Modal Analysis. In Gesture in Embodied Communication and Human-Computer Interaction, Stefan Koppand Ipke Wachsmuth (Eds.). Springer, Berlin Heidelberg, 158–170.
  4. Antonio Daniele , Caroline Di Bernardi Luft , and Nick Bryan-Kinns . 2021. “ What Is Human? ” A Turing Test for Artistic Creativity. In Artificial Intelligence in Music, Sound, Art and Design, Juan Romero, Tiago Martins, and Nereida Rodríguez-Fernández (Eds.) . Springer International Publishing , Cham , 396–411. Antonio Daniele, Caroline Di Bernardi Luft, and Nick Bryan-Kinns. 2021. “What Is Human?” A Turing Test for Artistic Creativity. In Artificial Intelligence in Music, Sound, Art and Design, Juan Romero, Tiago Martins, and Nereida Rodríguez-Fernández (Eds.). Springer International Publishing, Cham, 396–411.
  5. Ronald Fischer and Taciano L Milfont . 2010. Standardization in psychological research.International Journal of Psychological Research 3, 1 ( 2010 ), 88–96. Ronald Fischer and Taciano L Milfont. 2010. Standardization in psychological research.International Journal of Psychological Research 3, 1 (2010), 88–96.
  6. J Françoise , P Pasquier , O Alemi , J Françoise , and P Pasquier . 2017 . GrooveNet: real-time music-driven dance movement generation using artificial neural networks . In Proceedings of SIGKDD 2017 Workshop on Machine Learning for Creativity (Nova Scotia), Vol. 6. J Françoise, P Pasquier, O Alemi, J Françoise, and P Pasquier. 2017. GrooveNet: real-time music-driven dance movement generation using artificial neural networks. In Proceedings of SIGKDD 2017 Workshop on Machine Learning for Creativity (Nova Scotia), Vol. 6.
  7. Rolf Inge Godøy , Egil Haga , and Alexander Refsum Jensenius . 2006 . Exploring music-related gestures by sound-tracing: A preliminary study . In Proceedings of the COST287-ConGAS 2nd International Symposium on Gesture Interfaces for Multimedia Systems (GIMS2006) . 27–33. Rolf Inge Godøy, Egil Haga, and Alexander Refsum Jensenius. 2006. Exploring music-related gestures by sound-tracing: A preliminary study. In Proceedings of the COST287-ConGAS 2nd International Symposium on Gesture Interfaces for Multimedia Systems (GIMS2006). 27–33.
  8. Victor E Gonzalez-Sanchez , Agata Zelechowska , and Alexander Refsum Jensenius . 2018. Correspondences between music and involuntary human micromotion during standstill. Frontiers in psychology 9 ( 2018 ), 1382. Victor E Gonzalez-Sanchez, Agata Zelechowska, and Alexander Refsum Jensenius. 2018. Correspondences between music and involuntary human micromotion during standstill. Frontiers in psychology 9 (2018), 1382.
  9. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850(2013). Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850(2013). · arXiv
  10. David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017). David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017). · arXiv
  11. Lou E Hicks . 1970. Some properties of ipsative, normative, and forced-choice normative measures.Psychological bulletin 74, 3 ( 1970 ), 167. Lou E Hicks. 1970. Some properties of ipsative, normative, and forced-choice normative measures.Psychological bulletin 74, 3 (1970), 167.
  12. Alexander Refsum Jensenius , Agata Zelechowska , and Victor Evaristo Gonzalez Sanchez . 2017 . The musical influence on people’s micromotion when standing still in groups . In Proceedings of the SMC Conferences . Aalto University, 195–200. Alexander Refsum Jensenius, Agata Zelechowska, and Victor Evaristo Gonzalez Sanchez. 2017. The musical influence on people’s micromotion when standing still in groups. In Proceedings of the SMC Conferences. Aalto University, 195–200.
  13. Tejaswinee Kelkar and Alexander Refsum Jensenius . 2018. Analyzing Free-Hand Sound-Tracings of Melodic Phrases. Applied Sciences 8, 1 ( 2018 ). Tejaswinee Kelkar and Alexander Refsum Jensenius. 2018. Analyzing Free-Hand Sound-Tracings of Melodic Phrases. Applied Sciences 8, 1 (2018).
  14. Diederik P Kingma and Jimmy Ba . 2014 . Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014). Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014). · arXiv
  15. Meha Kumar , Duri Long , and Brian Magerko . 2020 . Creativity Metrics for a Lead-and-Follow Dynamic in an Improvisational Dance Agent . In Proceedings of the Eleventh International Conference on Computational Creativity. Meha Kumar, Duri Long, and Brian Magerko. 2020. Creativity Metrics for a Lead-and-Follow Dynamic in an Improvisational Dance Agent. In Proceedings of the Eleventh International Conference on Computational Creativity.
  16. Harry McGurk and John MacDonald . 1976. Hearing lips and seeing voices. Nature 264, 5588 ( 1976 ), 746–748. Harry McGurk and John MacDonald. 1976. Hearing lips and seeing voices. Nature 264, 5588 (1976), 746–748.
  17. Johannes Michalak , Nikolaus F Troje , Julia Fischer , Patrick Vollmar , Thomas Heidenreich , and Dietmar Schulte . 2009. Embodiment of sadness and depression—gait patterns associated with dysphoric mood. Psychosomatic medicine 71, 5 ( 2009 ), 580–587. Johannes Michalak, Nikolaus F Troje, Julia Fischer, Patrick Vollmar, Thomas Heidenreich, and Dietmar Schulte. 2009. Embodiment of sadness and depression—gait patterns associated with dysphoric mood. Psychosomatic medicine 71, 5 (2009), 580–587.
  18. Nick Neave , Kristofor McCarty , Jeanette Freynik , Nicholas Caplan , Johannes Hönekopp , and Bernhard Fink . 2011. Male dance moves that catch a woman’s eye. Biology letters 7, 2 ( 2011 ), 221–224. Nick Neave, Kristofor McCarty, Jeanette Freynik, Nicholas Caplan, Johannes Hönekopp, and Bernhard Fink. 2011. Male dance moves that catch a woman’s eye. Biology letters 7, 2 (2011), 221–224.
  19. Benedikte Wallace , Charles P. Martin , Jim Torresen , and Kristian Nymoen . 2020 . Towards Movement Generation with Audio Features . In Proceedings of the Eleventh International Conference on Computational Creativity. Benedikte Wallace, Charles P. Martin, Jim Torresen, and Kristian Nymoen. 2020. Towards Movement Generation with Audio Features. In Proceedings of the Eleventh International Conference on Computational Creativity.

Cited by

See all 15

Metadata

Authors
Wallace, B., Martin, C. P., Tørresen, J., & Nymoen, K.
Affiliations
  • University of Oslo (NO)
  • Australian National University (AU)
Venue
Creativity and Cognition
Publisher
ACM
Published
2021-06-22
Pages
1-9
OpenAlex
W3176390735
Cited by
13
References
27
Citations by year
2026: 2 · 2024: 6 · 2023: 3 · 2022: 1 · 2021: 1
FWCI
4.10
Type
conference paper
Language
en
Open access
closed
Locations
  • Creativity and Cognition (publishedVersion)
  • ANU Open Research (Australian National University) (submittedVersion)
Funders
Research Council of Norway (262762)
Topics
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
  • Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
Keywords
dance improvisation, embodied AI, generative models, 3D motion capture, virtual agents, robot emotion expression, expressivity, perceptual evaluation, creative AI, tacit knowledge

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Learning Embodied Sound-Motion Mappings: Evaluating AI-Generated Dance Improvisation. https://app.cochat.ai/papers/10.1145/3450741.3465245
The paper
Wallace, B., Martin, C. P., Tørresen, J., & Nymoen, K. (2021). Learning Embodied Sound-Motion Mappings: Evaluating AI-Generated Dance Improvisation. Creativity and Cognition. https://doi.org/10.1145/3450741.3465245

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.