Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
Diffusion models have experienced a surge of interest as highly expressive yet efficiently trainable probabilistic models. We show that these models are an excellent fit for synthesising human motion that co-occurs with audio, e.g., dancing and co-speech gesticulation, since motion is complex and highly ambiguous given audio, calling for a probabilistic description. Specifically, we adapt the DiffWave architecture to model 3D pose sequences, putting Conformers in place of dilated convolutions for improved modelling power. We also demonstrate control over motion style, using classifier-free guidance to adjust the strength of the stylistic expression. Experiments on gesture and dance generation confirm that the proposed method achieves top-of-the-line motion quality, with distinctive styles whose expression can be made more or less pronounced. We also synthesise path-driven locomotion using the same model architecture. Finally, we generalise the guidance procedure to obtain product-of-expert ensembles of diffusion models and demonstrate how these may be used for, e.g., style interpolation, a contribution we believe is of independent interest.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Davis, S., & Mermelstein, P. (1980) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1980.1163420 · Cited by 5,465
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1×
“…the gradient operation and taking the exponent 7 , we can see that the proposed ensemble corresponds to performing denoising steps based on a product of multiple different denoising distributions, This is a product of several probability density functions, i.e., a product of experts [Hinton 2002].”
BackgroundConformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3015 · Cited by 3,150
Conformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.08100 · Cited by 3,150 · cited 16×
“Deep-learning techniques used for directly generating dance pose sequences include recurrent neural networks and autoencoders [Crnkovic-Friis and Crnkovic-Friis 2016; Papillon et al. 2023; Tang et al. 2018], GANs [Hu et al. 2022], combinations of GANs and VAEs [Lee et al. 2019b], dilated convolutions and gated activat…”
BackgroundMethodInfluentialOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Film: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490
Long Short-Term Memory-Networks for Machine Reading
Cheng, J., Dong, L., & Lapata, M. (2016) · arXiv · doi:10.48550/arxiv.1601.06733 · Cited by 1,198
A note on the evaluation of generative models
Theis, L., Oord, A. V. D., & Bethge, M. (2015) · arXiv · doi:10.48550/arxiv.1511.01844 · Cited by 1,026
(2005) · doi:10.5555/1046920.1088696 · Cited by 756
BEAT: the Behavior Expression Animation Toolkit
Cassell, J., Vilhjálmsson, H. H., & Bickmore, T. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383315 · Cited by 739 · cited 2×
“Early work in gesture generation focussed on rule-based approaches [Cassell et al. 2001; Kopp et al. 2003; Lee and Marsella 2006; Lhommet et al. 2015] that typically would play pre-recorded gesture clips (or “lexemes”), at timings selected by hand-crafted rules; see Wagner et al. [2014] for a…”
BackgroundHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
Li, R., Yang, S., Ross, D. A., & Kanazawa, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01315 · Cited by 553 · cited 1×
“…combinations of GANs and VAEs [Lee et al. 2019b], dilated convolutions and gated activation units [Zhuang et al. 2022], Transformer architectures [Li et al. 2022, 2020, 2021; Valle-Pérez et al. 2021], VQ-VAEs and Transformers with re-inforcement learning [Siyao et al. 2022], and Conformers…”
BackgroundCompositional Visual Generation with Composable Diffusion Models
Liu, N., Li, S., Du, Y., Torralba, A., & Tenenbaum, J. B. (2022) · arXiv · doi:10.48550/arxiv.2206.01714 · Cited by 295
(2020) · doi:10.1111/cgf.13946 · Cited by 176
Self-citationEvaluation of Speech-to-Gesture Generation Using Bi-Directional LSTM Network
Hasegawa, D., Kaneko, N., Shirakawa, S., Sakuta, H., & Sumi, K. (2018) · Proceedings of the 18th International Conference on Intelligent Virtual Agents · doi:10.1145/3267851.3267878 · Cited by 131
Cited by
ImitationNet: Unsupervised Human-to-Robot Motion Retargeting via Shared Latent Space
Yan, Y., Mascaro, E. V., & Lee, D. (2023) · 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids57100.2023.10375150 · Cited by 27
Mehta, S., Deichler, A., O’Regan, J., Moëll, B., Beskow, J., Henter, G. E., & Alexanderson, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00201 · Cited by 7
Self-citation
Metadata
- Authors
- Alexanderson, S., Nagy, R., Beskow, J., & Henter, G. E.
- Venue
- ACM Transactions on Graphics
- Published
- 2023-07-26
- Volume
- 42
- Issue
- 4
- Pages
- 1-20
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W4377010269
- Cited by
- 194
- References
- 18
- FWCI
- 21.09
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.