The DOI matches its Crossref record; 90 of the 94 references it cites were found in a registry or an index.
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
Open accessAbstract
We present AIST++, a new multi-modal dataset of 3D dance motion and music, along with FACT, a Full-Attention Cross-modal Transformer network for generating 3D dance motion conditioned on music. The proposed AIST++ dataset contains 5.2 hours of 3D dance motion in 1408 sequences, covering 10 dance genres with multi-view videos with known camera poses—the largest dataset of this kind to our knowledge. We show that naively applying sequence models such as transformers to this dataset for the task of music conditioned 3D motion generation does not produce satisfactory 3D motion that is well correlated with the input music. We overcome these shortcomings by introducing key changes in its architecture design and supervision: FACT model involves a deep cross-modal transformer block with full-attention that is trained to predict N future motions. We empirically show that these changes are key factors in generating long sequences of realistic dance motion that are well-attuned to the input music. We conduct extensive experiments on AIST++ with user studies, where our method outperforms recent state-of-the-art methods both qualitatively and quantitatively. The code and the dataset can be found at: https://google.github.io/aichoreographer.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We note that the simple prediction-based late fusion [247], [248] is less adopted in MML Transformers.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [248]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 8×
“We propose Full Attention Cross-Modal Transformer (FACT) model for the task of 3D dance motion generation.”
BackgroundInfluentialAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“All our experiments are trained with 16 batch size using Adam [45] optimizer.”
MethodJakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 2×
“On the other hand, models like BERT [17] employ full-attention for feature learning, but rarely are these models employed in an auto-regressive manner, due to its inefficiency at test time.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“On the other hand, models like BERT [17] employ full-attention for feature learning, but rarely are these models employed in an auto-regressive manner, due to its inefficiency at test time.”
MethodGANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., & Sheikh, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2929257 · Cited by 5,536 · cited 1×
“Training data for 2D pose and audio is abundant thanks to the high reliability of 2D pose detectors [14].”
BackgroundDeep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734
Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653 · cited 2×
“While there are many large scale 3D motion capture datasets [39, 59, 1, 37], mocap dataset of 3D dance is quite limited as it requires heavy instrumentation and expert dancers for capture.”
BackgroundMethodlibrosa: Audio and Music Signal Analysis in Python
McFee, B., Raffel, C., Liang, D., Ellis, D. P. W., McVicar, M., Battenberg, E., & Nieto, O. (2015) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-7b98e3ed-003 · Cited by 3,099 · cited 3×
“We use the publicly available audio processing toolbox Librosa [60] to extract the music features including: 1-dim envelope , 20-dim MFCC , 12-dim chroma , 1-dim one-hot peaks and 1-dim one-hot beats , resulting in a 35-dim music feature.”
MethodInfluential2D Human Pose Estimation: New Benchmark and State of the Art Analysis
Andriluka, M., Pishchulin, L., Gehler, P., & Schiele, B. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.471 · Cited by 2,699 · cited 1×
“Besides, we also calculate the PCKh metric introduced in [6] on our AIST++.”
MethodScheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621 · cited 1×
“Some works [8, 56, 49] propose to ease this problem by periodically using the network’s own outputs as inputs during training.”
BackgroundSequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R. J., Darrell, T., & Saenko, K. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.515 · Cited by 1,448 · cited 3×
“Auto-regressive models like RNNs and vanilla Transformers are capable of generating unbounded motion in theory, but in practice suffer from regression to the mean where motion “freezes” after several iterations, or drift to unnatural motions [4, 3].”
BackgroundAMASS: Archive of Motion Capture As Surface Shapes
Mahmood, N., Ghorbani, N., Troje, N. F., Pons-Moll, G., & Black, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00554 · Cited by 1,323 · cited 3×
“While there are many large scale 3D motion capture datasets [39, 59, 1, 37], mocap dataset of 3D dance is quite limited as it requires heavy instrumentation and expert dancers for capture.”
BackgroundDense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,279
VideoBERT: A Joint Model for Video and Language Representation Learning
Sun, C., Myers, A., Vondrick, C., Murphy, K. P., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,255 · cited 1×
“More recently, with the development of attention mechanism [84], transformer based networks achieve top performance for visual-text [95, 77, 19, 54, 38, 76, 76], visual-audio [26, 89] cross-modal sequence to sequence generation task.”
BackgroundVIBE: Video Inference for Human Body Pose and Shape Estimation
Kocabas, M., Athanasiou, N., & Black, M. J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00530 · Cited by 1,134 · cited 1×
“We use the 2D mean per joint position error MPJPE2D, commonly used for 3D reconstruction quality measurement [46, 39, 65]) to evaluate the consistency between the predicted 2D keypoints and the reconstructed 3D keypoints along with the estimated camera parameters.”
MethodStructural-RNN: Deep Learning on Spatio-Temporal Graphs
Jain, A., Zamir, A. R., Savarese, S., & Saxena, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.573 · Cited by 1,133 · cited 1×
“…many approaches explore the applicability of neural networks to generate 3D motion by training on a large-scale motion capture dataset, where network architectures such as CNNs [35, 34], GANs [31], RBMs [80], RNNs [24, 4, 40, 27, 16, 18, 88, 12, 87] and Transform-ers [3, 9] have been explored.”
BackgroundKovar, L., Gleicher, M. L., & Pighin, F. P. (2008) · ACM SIGGRAPH Courses · doi:10.1145/1401132.1401202 · Cited by 1,073
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R. J., & Saenko, K. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1173 · Cited by 1,043 · cited 3×
“Auto-regressive models like RNNs and vanilla Transformers are capable of generating unbounded motion in theory, but in practice suffer from regression to the mean where motion “freezes” after several iterations, or drift to unnatural motions [4, 3].”
BackgroundDescribing Videos by Exploiting Temporal Structure
Yao, L., Torabi, A., Cho, K., Ballas, N., Pal, C., Larochelle, H., & Courville, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.512 · Cited by 1,035 · cited 1×
“Initially, combination of CNNs and RNNs [86, 85, 91, 93] were prominent in approaching this problem.”
BackgroundRecurrent Network Models for Human Dynamics
Fragkiadaki, K., Levine, S., Felsen, P., & Malik, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.494 · Cited by 933
A Comparative Study on Transformer vs RNN in Speech Applications
Karita, S., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., Someki, M., Yalta, N., Yamamoto, R., Wang, X., et al. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003750 · Cited by 766
FastSpeech: Fast, Robust and Controllable Text to SpeechPreprint
Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09263 · Cited by 724
Brand, M., & Hertzmann, A. (2000) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/344779.344865 · Cited by 704
A deep learning framework for character motion synthesis and editing
Holden, D., Saito, J., & Komura, T. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925975 · Cited by 656
End-to-End Dense Video Captioning with Masked Transformer
Zhou, L., Zhou, Y., Corso, J. J., Socher, R., & Xiong, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00911 · Cited by 605
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
Yu, H., Wang, J., Huang, Z., Yang, Y., & Xu, W. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.496 · Cited by 575 · cited 1×
“Initially, combination of CNNs and RNNs [86, 85, 91, 93] were prominent in approaching this problem.”
BackgroundA hierarchical approach to interactive motion editing for human-like figures
Lee, J., & Shin, S. Y. (1999) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/311535.311539 · Cited by 569 · cited 1×
“Motion graphs [53, 7, 47] address this problem by generating motions in a non-parametric manner.”
BackgroundPhase-functioned neural networks for character control
Holden, D., Komura, T., & Saito, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073663 · Cited by 556 · cited 1×
“Phase-functioned neural networks and it’s variations [94, 33, 73, 74] address this issue via conditioning the network weights on phase, however, they do not scale well to represent a wide variety of motion.”
BackgroundDeep Representation Learning for Human Motion Prediction and Classification
Bütepage, J., Black, M. J., Kragić, D., & Kjellström, H. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.173 · Cited by 435 · cited 1×
“…many approaches explore the applicability of neural networks to generate 3D motion by training on a large-scale motion capture dataset, where network architectures such as CNNs [35, 34], GANs [31], RBMs [80], RNNs [24, 4, 40, 27, 16, 18, 88, 12, 87] and Transform-ers [3, 9] have been explored.”
BackgroundLPCNET: Improving Neural Speech Synthesis through Linear Prediction
Valin, J.-M., & Skoglund, J. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682804 · Cited by 427
Lu, J., Yang, J., Batra, D., & Parikh, D. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00754 · Cited by 422
Entangled Transformer for Image Captioning
Li, G., Zhu, L., Liu, P., & Yang, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00902 · Cited by 411 · cited 1×
“More recently, with the development of attention mechanism [84], transformer based networks achieve top performance for visual-text [95, 77, 19, 54, 38, 76, 76], visual-audio [26, 89] cross-modal sequence to sequence generation task.”
BackgroundInteractive motion generation from examples
Arıkan, O., & Forsyth, D. A. (2002) · ACM Transactions on Graphics · doi:10.1145/566654.566606 · Cited by 405
Factored conditional restricted Boltzmann Machines for modeling motion style
Taylor, G. W., & Hinton, G. E. (2009) · International Conference on Machine Learning · doi:10.1145/1553374.1553505 · Cited by 362 · cited 2×
“With the development in deep learning, many approaches explore the applicability of neural networks to generate 3D motion by training on a large-scale motion capture dataset, where network architectures such as CNNs [35, 34], GANs [31], RBMs [80], RNNs [24, 4, 40, 27, 16, 18, 88, 12, 87] and Transform-ers [3, 9] have…”
BackgroundLearning Individual Styles of Conversational Gesture
Ginosar, S., Bar, A., Kohavi, G., Chan, C., Owens, A., & Malik, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00361 · Cited by 362 · cited 5×
“More recent approach employ LSTMs [5, 79, 90, 97, 42], GANs [51, 78, 28], transformer encoder with RNN de-coder [36] or convolutional [2, 92] sequence-to-sequence models.”
BackgroundMethodInfluentialHuang, Y., Kaufmann, M., Aksan, E., Black, M. J., Hilliges, O., & Pons-Moll, G. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275108 · Cited by 342
Learning Video Representations using Contrastive Bidirectional TransformerPreprint
Sun, C., Baradel, F., Murphy, K. P., & Schmid, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05743 · Cited by 307
Neural state machine for character-scene interactions
Starke, S., Zhang, H., Komura, T., & Saito, J. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356505 · Cited by 291 · cited 1×
“Phase-functioned neural networks and it’s variations [94, 33, 73, 74] address this issue via conditioning the network weights on phase, however, they do not scale well to represent a wide variety of motion.”
BackgroundMode-adaptive neural networks for quadruped motion control
Zhang, H., Starke, S., Komura, T., & Saito, J. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201366 · Cited by 276 · cited 1×
“Phase-functioned neural networks and it’s variations [94, 33, 73, 74] address this issue via conditioning the network weights on phase, however, they do not scale well to represent a wide variety of motion.”
BackgroundLearning motion manifolds with convolutional autoencoders
Holden, D., Saito, J., Komura, T., & Joyce, T. (2015) · SIGGRAPH Asia Technical Briefs · doi:10.1145/2820903.2820918 · Cited by 268 · cited 1×
“…many approaches explore the applicability of neural networks to generate 3D motion by training on a large-scale motion capture dataset, where network architectures such as CNNs [35, 34], GANs [31], RBMs [80], RNNs [24, 4, 40, 27, 16, 18, 88, 12, 87] and Transform-ers [3, 9] have been explored.”
BackgroundLearning Variable-Length Markov Models of Behavior
Galata, A., Johnson, N. J., & Hogg, D. (2001) · Computer Vision and Image Understanding · doi:10.1006/cviu.2000.0894 · Cited by 267 · cited 1×
“Earlier works employ statistical models such as kernel-based probability distribution [64, 10, 25, 11] to synthesize motion, but abstract away motion details.”
MethodCross View Fusion for 3D Human Pose Estimation
Qiu, H., Wang, C., Wang, J., Wang, N., & Zeng, W. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00444 · Cited by 241 · cited 1×
“We use the 2D mean per joint position error MPJPE-2D, commonly used for 3D reconstruction quality measurement [46, 39, 65]) to evaluate the consistency between the predicted 2D keypoints and the reconstructed 3D keypoints along with the estimated camera parameters.”
MethodNeural Kinematic Networks for Unsupervised Motion Retargetting
Villegas, R., Yang, J., Ceylan, D., & Lee, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00901 · Cited by 240
Learning Human Motion Models for Long-Term Predictions
Ghosh, P. S., Song, J., Aksan, E., & Hilliges, O. (2017) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00059 · Cited by 227
Heck, R., & Gleicher, M. L. (2007) · ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D) · doi:10.1145/1230100.1230123 · Cited by 211 · cited 1×
“A challenge in motion graph is in generating plausible transition that some approaches address via parameterizing the transition [30].”
BackgroundMulti-modal Dense Video Captioning
lashin, V., & Rahtu, E. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00487 · Cited by 205 · cited 1×
“More recently, with the development of attention mechanism [84], transformer based networks achieve top performance for visual-text [95, 77, 19, 54, 38, 76, 76], visual-audio [26, 89] cross-modal sequence to sequence generation task.”
BackgroundHuman Motion Prediction via Spatio-Temporal Inpainting
Hernandez, A., Gall, J., & Moreno, F. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00723 · Cited by 196 · cited 1×
“…many approaches explore the applicability of neural networks to generate 3D motion by training on a large-scale motion capture dataset, where network architectures such as CNNs [35, 34], GANs [31], RBMs [80], RNNs [24, 4, 40, 27, 16, 18, 88, 12, 87] and Transform-ers [3, 9] have been explored.”
BackgroundStructured Prediction Helps 3D Human Motion Modelling
Aksan, E., Kaufmann, M., & Hilliges, O. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00724 · Cited by 194 · cited 3×
“Auto-regressive models like RNNs and vanilla Transformers are capable of generating unbounded motion in theory, but in practice suffer from regression to the mean where motion “freezes” after several iterations, or drift to unnatural motions [4, 3].”
BackgroundMethodBenchmarking and Error Diagnosis in Multi-instance Pose Estimation
Ronchi, M. R., & Perona, P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.48 · Cited by 187 · cited 1×
“• 17 COCO-format[70] human joint locations in both 2D and 3D;”
BackgroundLocal motion phases for learning multi-contact character movements
Starke, S., Zhao, Y., Komura, T., & Zaman, K. A. (2020) · ACM Transactions on Graphics · doi:10.1145/3386569.3392450 · Cited by 183 · cited 1×
“Phase-functioned neural networks and it’s variations [94, 33, 73, 74] address this issue via conditioning the network weights on phase, however, they do not scale well to represent a wide variety of motion.”
BackgroundTang, T., Jia, J., & Mao, H. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240526 · Cited by 168 · cited 2×
“As such, many of these previous works operate on either small-scale or private motion capture datasets [79, 5, 96].”
BackgroundMethodAction-Agnostic Human Pose Forecasting
Chiu, H.-K., Adeli, E., Wang, B., Huang, D.-A., & Niebles, J. C. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00156 · Cited by 163
Shlizerman, E., Dery, L., Schoen, H., & Kemelmacher-Shlizerman, I. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00790 · Cited by 154 · cited 2×
“Many prior works [72, 68, 42, 28, 92] solve this problem by predicting future motion deterministically from audio without seed motion.”
BackgroundLeveraging Weakly Supervised Data to Improve End-to-end Speech-to-text Translation
Ye, J., Johnson, M., Macherey, W., Weiss, R. J., Cao, Y. J., Chiu, C., Ari, N., Laurenzo, S., & Wu, Y. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683343 · Cited by 153
Auto-Conditioned Recurrent Networks for Extended Complex Human Motion SynthesisPreprint
Li, Z., Zhou, Y., Xiao, S., He, C., Huang, Z., & Li, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05363 · Cited by 150
Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents
Bhattacharya, U., Rewkowski, N., Banerjee, A., Guhan, P., Bera, A., & Manocha, D. (2021) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr50410.2021.00037 · Cited by 144 · cited 2×
“Auto-regressive models like RNNs and vanilla Transformers are capable of generating unbounded motion in theory, but in practice suffer from regression to the mean where motion “freezes” after several iterations, or drift to unnatural motions [4, 3].”
BackgroundDeepDance: Music-to-Dance Motion Choreography With Adversarial Learning
Sun, G., Wong, Y., Cheng, Z., Kankanhalli, M., Geng, W., & Li, X. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.2981989 · Cited by 140 · cited 1×
“More recent approach employ LSTMs [5, 79, 90, 97, 42], GANs [51, 78, 28], transformer encoder with RNN de-coder [36] or convolutional [2, 92] sequence-to-sequence models.”
MethodFoley Music: Learning to Generate Music from Videos
Gan, C., Huang, D. Y., Chen, P., Tenenbaum, J. B., & Torralba, A. B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58621-8_44 · Cited by 139
Investigating the use of recurrent motion modelling for speech gesture generation
Ferstl, Y., & McDonnell, R. (2018) · ACM International Conference on Intelligent Virtual Agents (IVA) · doi:10.1145/3267851.3267898 · Cited by 138 · cited 1×
“In natural language processing and computer vision, tasks like text to speech (TTS) [69, 41, 43, 83] and speech to gesture [22, 28, 23], image/video captioning (pixels to text) [13, 44, 58, 48] involve solving the cross-modal sequence to sequence generation problem.”
BackgroundWeakly Supervised Dense Event Captioning in VideosPreprint
Duan, X., Huang, W., Gan, C., Wang, J., Zhu, W., & Huang, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.03849 · Cited by 131 · cited 1×
“More recently, with the development of attention mechanism [84], transformer based networks achieve top performance for visual-text [95, 77, 19, 54, 38, 76, 76], visual-audio [26, 89] cross-modal sequence to sequence generation task.”
BackgroundDancing‐to‐Music Character Animation
Shiratori, T., Nakazawa, A., & Ikeuchi, K. (2006) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2006.00964.x · Cited by 126 · cited 1×
“For 3D dance generation, earlier approaches explore matching existing 3D motion to music [71] using motion graph based approach [20].”
Imitation Learning for Human Pose Prediction
Wang, B., Adeli, E., Chiu, H.-K., Huang, D.-A., & Niebles, J. C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00722 · Cited by 104 · cited 1×
“…many approaches explore the applicability of neural networks to generate 3D motion by training on a large-scale motion capture dataset, where network architectures such as CNNs [35, 34], GANs [31], RBMs [80], RNNs [24, 4, 40, 27, 16, 18, 88, 12, 87] and Transform-ers [3, 9] have been explored.”
BackgroundCross-Modal Relation-Aware Networks for Audio-Visual Event Localization
Xu, H., Zeng, R., Wu, Q., Tan, M., & Gan, C. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413581 · Cited by 103 · cited 1×
“More recently, with the development of attention mechanism [84], transformer based networks achieve top performance for visual-text [95, 77, 19, 54, 38, 76, 76], visual-audio [26, 89] cross-modal sequence to sequence generation task.”
BackgroundFerreira, J., Coutinho, T., Gomes, T. L., NETO, J. F. L., Azevedo, R., Martins, R., & Nascimento, E. R. (2020) · Computers & Graphics · doi:10.1016/j.cag.2020.09.009 · Cited by 101 · cited 1×
“Audio To Human Motion Generation Audio to motion generation has been studied in 2D pose context either in optimization based approach [81], or learning based approaches [52, 72, 51, 67, 68, 21] where 2D pose skele-tons are generated from a conditioning audio.”
BackgroundLearning Statistical Models of Human Motion
Bowden, R. (2000) · Cited by 93
Dance Revolution: Long-Term Dance Generation with Music via Curriculum LearningPreprint
Huang, R., Huang, H., Wu, W., Sawada, K. B., Zhang, M., & Jiang, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06119 · Cited by 92 · cited 7×
“Similar to the prior work [36], we calculate the average Euclidean distance in the feature space across 40 generated motions on the AIST++ test set to measure the diversity.”
BackgroundMethodResultInfluentialExample-Based Automatic Music-Driven Conventional Dance Motion Synthesis
Fan, R., Xu, S., & Geng, W. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.73 · Cited by 90 · cited 1×
“For 3D dance generation, earlier approaches explore matching existing 3D motion to music [71] using motion graph based approach [20].”
Efficient content-based retrieval of motion capture data
Müller, M., Röder, T., & Clausen, M. V. (2005) · ACM SIGGRAPH Conference Papers (SIGGRAPH) · doi:10.1145/1186822.1073247 · Cited by 87 · cited 1×
“As prior work used motionencoders that are not public, we measure FID with two welldesigned motion feature extractors [62, 63] implemented in fairmotion [29]: (1) a geometric feature extractor that produces a boolean vector zg ∈ R(33) expressing geometric relations between certain body points in the motion sequence X…”
MethodDu, X., Vasudevan, R., & Johnson‐Roberson, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2895266 · Cited by 85 · cited 1×
“…many approaches explore the applicability of neural networks to generate 3D motion by training on a large-scale motion capture dataset, where network architectures such as CNNs [35, 34], GANs [31], RBMs [80], RNNs [24, 4, 40, 27, 16, 18, 88, 12, 87] and Transform-ers [3, 9] have been explored.”
BackgroundTsuchida, S., Fukayama, S., Hamasaki, M., & Goto, M. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3527853 · Cited by 84
Learning to Generate Diverse Dance Motions with TransformerPreprint
Li, J., Yin, Y., Chu, H., Zhou, Y., Wang, T., Fidler, S., & Li, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.08171 · Cited by 75 · cited 10×
“[55], which discretizes the motion, leading to discontinuous outputs that results in high Distk.”
BackgroundMethodResultInfluentialSelf-supervised Dance Video Synthesis Conditioned on Music
Ren, X., Li, H., Huang, Z., & Chen, Q. (2020) · ACM Multimedia · doi:10.1145/3394171.3413932 · Cited by 74
ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit
Ye, Z., Wu, H., Jia, J., Bu, Y., Chen, W., Meng, F., & Wang, Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3414005 · Cited by 68 · cited 2×
“More recent approach employ LSTMs [5, 79, 90, 97, 42], GANs [51, 78, 28], transformer encoder with RNN de-coder [36] or convolutional [2, 92] sequence-to-sequence models.”
BackgroundMethodWeakly-Supervised Deep Recurrent Neural Networks for Basic Dance Step Generation
Yalta, N., Watanabe, S., Nakadai, K., & Ogata, T. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8851872 · Cited by 57 · cited 2×
“More recent approach employ LSTMs [5, 79, 90, 97, 42], GANs [51, 78, 28], transformer encoder with RNN decoder [36] or convolutional [2, 92] sequence-to-sequence models.”
MethodGenerative Autoregressive Networks for 3D Dancing Move Synthesis From Music
Ahn, H., Kim, J., Kim, K., & Oh, S. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2977333 · Cited by 52 · cited 1×
“More recent approach employ LSTMs [5, 79, 90, 97, 42], GANs [51, 78, 28], transformer encoder with RNN de-coder [36] or convolutional [2, 92] sequence-to-sequence models.”
MethodAnimating by multi-level sampling
Pullen, K., & Bregler, C. (2002) · Computer Animation (CA) · doi:10.1109/ca.2000.889031 · Cited by 51 · cited 1×
“Earlier works employ statistical models such as kernel-based probability distribution [64, 10, 25, 11] to synthesize motion, but abstract away motion details.”
MethodAdversarial gesture generation with realistic gesture phasing
Ferstl, Y., Neff, M., & McDonnell, R. (2020) · Computers & Graphics · doi:10.1016/j.cag.2020.04.007 · Cited by 51 · cited 1×
“In natural language processing and computer vision, tasks like text to speech (TTS) [69, 41, 43, 83] and speech to gesture [22, 28, 23], image/video captioning (pixels to text) [13, 44, 58, 48] involve solving the cross-modal sequence to sequence generation problem.”
BackgroundTemporally Guided Music-to-Body-Movement Generation
Kao, H.-K., & Su, L. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413848 · Cited by 43 · cited 2×
“More recent approach employ LSTMs [5, 79, 90, 97, 42], GANs [51, 78, 28], transformer encoder with RNN de-coder [36] or convolutional [2, 92] sequence-to-sequence models.”
BackgroundMethodFMDistance: A Fast and Effective Distance Function for Motion Capture Data
Onuma, K., Faloutsos, C., & Hodgins, J. K. (2008) · doi:10.2312/egs.20081027 · Cited by 37 · cited 1×
“As prior work used motionencoders that are not public, we measure FID with two welldesigned motion feature extractors [62, 63] implemented in fairmotion [29]: (1) a geometric feature extractor that produces a boolean vector zg ∈ R(33) expressing geometric relations between certain body points in the motion sequence X…”
MethodListen to Dance: Music-driven choreography generation using Autoregressive Encoder-Decoder NetworkPreprint
Lee, J., Kim, S., & Lee, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00818 · Cited by 35 · cited 1×
“Audio To Human Motion Generation Audio to motion generation has been studied in 2D pose context either in optimization based approach [81], or learning based approaches [52, 72, 51, 67, 68, 21] where 2D pose skele-tons are generated from a conditioning audio.”
BackgroundDance music, movement and tempo preferences
Moelants, D. (2003) · Ghent University Academic Bibliography (Ghent University) · Cited by 26
Kundu, J. N., Buckchash, H., Mandikal, P., Rahul, M., Jamkhandi, A., & Babu, R. V. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093627 · Cited by 26 · cited 1×
“Some works [8, 56, 49] propose to ease this problem by periodically using the network’s own outputs as inputs during training.”
BackgroundMusic2Dance: Music-driven Dance Generation using WaveNet.Preprint
Zhuang, W., Wang, C., Xia, S., Chai, J., & Wang, Y. (2020) · arXiv (Cornell University) · Cited by 19
Attention, please: A Spatio-temporal Transformer for 3D Human Motion PredictionPreprint
Aksan, E., Cao, P., Kaufmann, M., & Hilliges, O. (2020) · arXiv (Cornell University) · Cited by 18
Music-oriented Dance Video Synthesis with Pose Perceptual LossPreprint
Ren, X., Li, H., Huang, Z., & Chen, Q. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.06606 · Cited by 16 · cited 1×
“Audio To Human Motion Generation Audio to motion generation has been studied in 2D pose context either in optimization based approach [81], or learning based approaches [52, 72, 51, 67, 68, 21] where 2D pose skele-tons are generated from a conditioning audio.”
BackgroundTowards 3D Dance Motion Synthesis and ControlPreprint
Zhuang, W., Wang, Y., Robinson, J. P., Wang, C., Shao, M., Fu, Y., & Xia, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05743 · Cited by 15 · cited 1×
“More recent approach employ LSTMs [5, 79, 90, 97, 42], GANs [51, 78, 28], transformer encoder with RNN de-coder [36] or convolutional [2, 92] sequence-to-sequence models.”
MethodFeel The Music: Automatically Generating A Dance For An Input SongPreprint
Tendulkar, P., Das, A., Kembhavi, A., & Parikh, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11905 · Cited by 5 · cited 1×
“Audio To Human Motion Generation Audio to motion generation has been studied in 2D pose context either in optimization based approach [81], or learning based approaches [52, 72, 51, 67, 68, 21] where 2D pose skele-tons are generated from a conditioning audio.”
Background
Research works printed but not linked: 3
- lamothe, 2019, The dancing species: how moving together in time helps make us human, Aeon
- alemi, 2017, Groovenet: Real-time music-driven dance movement generation using artificial neural networks, Networks, 8, 26
- chen, 2021, Choreomaster: choreography-oriented music-driven dance synthesis, ACM Transactions on Graphics (TOG), 40, 1
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
“Depth to Binaural Audio [14], music piece & seed 3D motion to long-range future 3D motions [146], X-raying image & question to answer [244], video & text & audio to text [245]), and”
BackgroundAction-Conditioned 3D Human Motion Synthesis with Transformer VAE
Petrovich, M., Black, M. J., & Varol, G. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01080 · Cited by 571
“Very recently, [39] presents a Transformer-based method for dance generation conditioned on music and past motion.”
MethodMotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3355414 · Cited by 890
“, 2018), 3D keypoints (Ionescu et al., 2013; Joo et al., 2015; Mehta et al., 2017; Trumble et al., 2017; Li et al., 2021) and statistical model parameters (Yu et al.”
BackgroundMotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
Zhang, M., Cai, Z., Pan, L., Hong, F., Guo, X., Yang, L., & Liu, Z. (2022) · arXiv · doi:10.48550/arxiv.2208.15001 · Cited by 437
“, 2018), 3D keypoints (Ionescu et al., 2013; Joo et al., 2015; Mehta et al., 2017; Trumble et al., 2017; Li et al., 2021) and statistical model parameters (Yu et al.”
BackgroundTEMOS: Generating Diverse Human Motions from Textual Descriptions
Petrovich, M., Black, M. J., & Varol, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20047-2_28 · Cited by 429
Metadata
- Authors
- Li, R., Yang, S., Ross, D. A., & Kanazawa, A.
- Affiliations
- University of Southern California (US)
- Google (United States) (US)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 13381-13392
- OpenAlex
- W3204221554
- Cited by
- 553
- References
- 94
- Citations by year
- 2026: 92 · 2025: 146 · 2024: 154 · 2023: 113 · 2022: 46 · 2021: 2
- FWCI
- 117.49
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Keywords
- cross-modal transformer, full attention, AIST, dance genres, choreography
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
