The DOI matches its Crossref record; 90 of the 94 references it cites were found in a registry or an index.
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
Open accessAbstract
We present AIST++, a new multi-modal dataset of 3D dance motion and music, along with FACT, a Full-Attention Cross-modal Transformer network for generating 3D dance motion conditioned on music. The proposed AIST++ dataset contains 5.2 hours of 3D dance motion in 1408 sequences, covering 10 dance genres with multi-view videos with known camera poses—the largest dataset of this kind to our knowledge. We show that naively applying sequence models such as transformers to this dataset for the task of music conditioned 3D motion generation does not produce satisfactory 3D motion that is well correlated with the input music. We overcome these shortcomings by introducing key changes in its architecture design and supervision: FACT model involves a deep cross-modal transformer block with full-attention that is trained to predict N future motions. We empirically show that these changes are key factors in generating long sequences of realistic dance motion that are well-attuned to the input music. We conduct extensive experiments on AIST++ with user studies, where our method outperforms recent state-of-the-art methods both qualitatively and quantitatively. The code and the dataset can be found at: https://google.github.io/aichoreographer.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We note that the simple prediction-based late fusion [247], [248] is less adopted in MML Transformers.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [248]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 8×
“We propose Full Attention Cross-Modal Transformer (FACT) model for the task of 3D dance motion generation.”
BackgroundInfluentialAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“All our experiments are trained with 16 batch size using Adam [45] optimizer.”
MethodJakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 2×
“On the other hand, models like BERT [17] employ full-attention for feature learning, but rarely are these models employed in an auto-regressive manner, due to its inefficiency at test time.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“On the other hand, models like BERT [17] employ full-attention for feature learning, but rarely are these models employed in an auto-regressive manner, due to its inefficiency at test time.”
MethodGANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., & Sheikh, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2929257 · Cited by 5,536 · cited 1×
“Training data for 2D pose and audio is abundant thanks to the high reliability of 2D pose detectors [14].”
BackgroundDeep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734
Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653 · cited 2×
“While there are many large scale 3D motion capture datasets [39, 59, 1, 37], mocap dataset of 3D dance is quite limited as it requires heavy instrumentation and expert dancers for capture.”
BackgroundMethod
Research works printed but not linked: 3
- lamothe, 2019, The dancing species: how moving together in time helps make us human, Aeon
- alemi, 2017, Groovenet: Real-time music-driven dance movement generation using artificial neural networks, Networks, 8, 26
- chen, 2021, Choreomaster: choreography-oriented music-driven dance synthesis, ACM Transactions on Graphics (TOG), 40, 1
Cited by
Sorted by citations
Items 101–200 of 553
GUESS: GradUally Enriching SyntheSis for Text-Driven Human Motion Generation
Gao, X., Yang, Y., Xie, Z., Du, S., Sun, Z., & Wu, Y. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3352002 · Cited by 32
Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generation
Chen, B., Li, Y., Ding, Y.-X., Shao, T., & Zhou, K. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680847 · Cited by 38
“As shown in Table 2, our method significantly outperforms baselines in terms of FGD [53], BC [26], and diversity [25].”
MethodHe, X., Fang, Y., Xiang, N., Li, Z., Wang, Q., Yang, C., Wang, H. H., & Pan, Y. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129532 · Cited by 31
“Recent research applying transformer and attention mechanisms has advanced fields like humor detection[52], product recognition[53], time series alignment in human behavior modeling[24], and dance choreography [54].”
Zhou, B., Geißler, D., Faulhaber, M., Gleiß, C., Zahn, E. F., Ray, L. S. S., Gamarra, D. F., Rey, V. F., Suh, S., Bian, S., et al. (2023) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3580883 · Cited by 34
“…sensing requires more sophisticated resonant excitation signals and frequency counting, modern integrated circuit (IC) technologies have already packaged most necessary electronic front-end in miniaturized packages such as the recent chip models FDC1004 and FDC2214 (Texas Instruments, USA) [89].”
OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers
Liang, H., Bao, J., Zhang, R., Ren, S., Xu, Y., Yang, S., Chen, X., Jingyi Yu, A., & Xu, L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00053 · Cited by 33
“At the pre-training stage, we utilize various publicly available human motion datasets, such as artist-created datasets [21, 48], marker-based [7, 28, 44, 47, 75], IMU-based [39, 78] and multi-view marker-less [9, 38, 40, 90] motion capture datasets, totaling over 20 million frames.”
BackgroundMethodSelf-citationMVHumanNet: A Large-Scale Dataset of Multi-View Daily Dressing Human Captures
Xiong, Z., Li, C., Liu, K., Liao, H., Hu, J., Zhu, J., Ning, S., Qiu, L., Wang, C., Wang, S., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01872 · Cited by 26
“Previous multi-view human datasets [31, 41, 55] involve only a few dozen human subjects.”
BackgroundLocomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environments
Lee, J., & Joo, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00886 · Cited by 25
“• Naturalness: We measure the naturalness of the synthesized motions by measuring the Fréchet distance, as reported in [15, 35, 40] between the synthesized motion and motions from motion capture data.”
MethodGenerative Proxemics: A Prior for 3D Social Interaction from Images
Müller, L., Ye, V., Pavlakos, G., Black, M. J., & Kanazawa, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00925 · Cited by 41
Self-citationYeung, C., Suzuki, T., Tanaka, R., Yin, Z., & Fujii, K. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00592 · Cited by 41
“Additionally, alternative approaches exist, including inferring 3D poses from 2D annotations (e.g., 3DSP [80]), estimating 3D poses via triangulation from multiple 2D pose detections (e.g., ASPset-510 [49]), and reconstructing 3D poses from real-world 3D human pose data (e.g., AIST++ [31]).”
Animation from Blur: Multi-modal Blur Decomposition with Motion Guidance
Zhong, Z., Sun, X. N., Wu, Z., Zheng, Y., Lin, S. C.-F., & Sato, I. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19800-7_35 · Cited by 24
RGBT Tracking via Progressive Fusion Transformer With Dynamically Guided Learning
Zhu, Y., Li, C., Wang, X., Tang, J., & Huang, Z. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3391802 · Cited by 24
“Transformer plays an important role in the field of multimodal learning because of its strong feature representation abilities and flexible network architecture [33], [34], [35], [36], [37], [38], [39].”
BackgroundCase Study of China’s Compulsory Education System: AI Apps and Extracurricular Dance Learning
Cao, X. (2023) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2023.2188539 · Cited by 27
“Dance requires training and proficiency in a rich repertoire of dance moves to create expressive choreography (Li et al., 2021).”
BackgroundLet's all dance: Enhancing amateur dance motions
Zhou, Q., Li, M., Zeng, Q., Aristidou, A., Zhang, X., Chen, L., & Tu, C. (2023) · Computational Visual Media · doi:10.1007/s41095-022-0292-6 · Cited by 23
“Instead, we propose a data augmentation scheme that artificially synthesizes random non-professional dances, by altering professional ones taken from the AIST++ dataset [3], both on their spatial and temporal domains.”
BackgroundMethodInfluentialTask-Oriented Human-Object Interactions Generation with Implicit Neural Representations
Li, Q., Wang, J. B., Loy, C. C., & Dai, B. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00301 · Cited by 38
“Our method generates continuous motions that are parameterized only by the temporal coordinate, which allows for upsampling of the sequence to arbitrary frames and adjusting the motion speeds by designing the temporal coordinate vector.”
BackgroundMethodDiff-BGM: A Diffusion Model for Video Background Music Generation
Li, S., Qin, Y., Zheng, M., Jin, X., & Liu, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02582 · Cited by 31
POPDG: Popular 3D Dance Generation with PopDanceSet
Luo, Z., Ren, M., Hu, X., Huang, Y., & Li, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02548 · Cited by 26
“With FACT’s introduction [24], Transformers have gained prominence for their superior temporal feature modeling [16, 17, 43].”
BackgroundMethodInfluentialMotion In-Betweening via Deep -Interpolator
Oreshkin, B. N., Valkanas, A., Harvey, F. G., Ménard, L.-S., Bocquelet, F., & Coates, M. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3309107 · Cited by 41
“These range from audio conditioned human motion generation such as creating dances for a musical piece [38], [39], to semantically conditioned motion [40], [41].”
BackgroundSMPLer: Taming Transformers for Monocular 3D Human Shape and Pose Estimation
Xu, X., Liu, L., & Yan, S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3341630 · Cited by 22
“It has been widely used in many applications including visual tracking [8], [9], virtual/augmented reality [10], [11], [12], [13], [14], [15], [16], [17], [18], motion generation [19], [20], image manipulation [21], [22], and neural radiance field [23], [24], [25].”
MethodParCo: Part-Coordinating Text-to-Motion Synthesis
Zou, Q., Yuan, S., Du, S., Wang, Y., Liu, C., Xu, Y., Chen, J., & Ji, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72992-8_8 · Cited by 42
Relightable Neural Human Assets from Multi-view Gradient Illuminations
Zhou, T., He, K., Wu, D., Xu, T. G., Zhang, Q., Shao, K., Chen, W., Xu, L., & Jingyi Yu, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00420 · Cited by 20
“Several valuable datasets [5, 49, 84] have been made available, fostering algorithm development.”
BackgroundNPC: Neural Point Characters from Video
Su, S.-Y., Bagautdinov, T., & Rhodin, H. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01359 · Cited by 20
“We additionally evaluate NPC on one subject with loose clothing from ZJU-Mocap [8, 35], and use challenging motion sequences, including dancing and gymnastic from poses AIST++ [20] and SURREAL+CMU-Mocap [6, 48] dataset, for animating our learned characters.”
MethodLiveHPS: LiDAR-Based Scene-Level Human Pose and Shape Estimation in Free Environment
Ren, Y., Han, X., Zhao, C., Wang, J., Xu, L., Jingyi Yu, A., & Ma, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00128 · Cited by 24
“To enrich the dataset with various poses and shapes for pretraining, we follow LIP [45] to create synthetic point clouds from SURREAL [51], AIST++ [34], and portions of AMASS [37], including ACCAD and BMLMovi.”
BackgroundMethodBidirectional Autoregressive Diffusion Model for Dance Generation
Zhang, C., Tang, Y., Zhang, N., Lin, R.-S., Han, M., Xiao, J., & Wang, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00072 · Cited by 21
“These methods span a diverse spectrum, encompassing CNN [16], GANs [26, 47], and Transformer models [28, 29].”
BackgroundMethodPoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling
Baradel, F., Brégier, R., Groueix, T., Weinzaepfel, P., Kalantidis, Y., & Rogez, G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3216899 · Cited by 21
“cross-modal transformer architecture has been proposed in [59] to generate human pose sequences conditioned on music.”
BackgroundDance Style Transfer with Cross-modal Transformer
Yin, W., Yin, H., Baraka, K., Kragić, D., & Björkman, M. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00503 · Cited by 19
“We generate 3D dance motion samples with paired music from an existing database called AIST++ Dance Database [27].”
V2Meow: Meowing to the Visual Beat via Video-to-Music Generation
Su, K., Li, J. Y., Huang, Q., Kuzmin, D., Lee, J., Donahue, C., Sha, F., Jansen, A., Wang, Y., Verzetti, M., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28299 · Cited by 36
“The zero-shot evaluation on dance videos in the AIST++ test split (Li et al. 2021) reveals that V2Meow can attain performances comparable to those of specialized dance-to-music generation baselines (Zhu et al. 2022a,b), measured by beat coverage and beat hit.”
Background4D Facial Expression Diffusion Model
Zou, K., Faisan, S., Yu, B., Valette, S., & Seo, H. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3653455 · Cited by 28
Large Motion Model for Unified Multi-modal Motion Generation
Zhang, M., Jin, D., Gu, C., Hong, F., Cai, Z., Huang, J., Zhang, C., Guo, X., Yang, L., He, Y., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72624-8_23 · Cited by 49
Genre-Conditioned Long-Term 3D Dance Generation Driven by Music
Huang, Y., Zhang, J., Liu, S., Bao, Q., Zeng, D., Chen, Z., & Liu, W. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747838 · Cited by 18
GroupDancer: Music to Multi-People Dance Synthesis with Style Collaboration
Wang, Z., Jia, J., Wu, H., Xing, J., Cai, J., Meng, F., Chen, G., & Wang, Y. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548090 · Cited by 18
“AI Choreographer [18] introduced the AIST++ dataset of 3D dance motions accompanied by music and multi-view images based on the 2D dance video database AIST [23].”
BackgroundMethodInfluentialKinematic-aware Hierarchical Attention Network for Human Pose Estimation in Videos
Jin, K.-M., Lim, B.-S., Lee, G., Kang, T.-K., & Lee, S. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00568 · Cited by 18
“Fully-annotated benchmark datasets [17, 15, 48, 27] for videos are suitable for learning temporal features because they provide supervision for all frames and contain a few people with less occlusion.”
BackgroundMethodMultimodal dance style transfer
Yin, W., Yin, H., Baraka, K., Kragić, D., & Björkman, M. (2023) · Machine Vision and Applications · doi:10.1007/s00138-023-01399-x · Cited by 18
“We utilize the AIST+ Dance Database [42] as our source of data to generate 3D dance motion samples with paired music.”
MethodMNET++: Music-Driven Pluralistic Dancing Toward Multiple Dance Genre Synthesis
Kim, J., Kwon, B., Kim, J., & Lee, S. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3312092 · Cited by 20
“We experiment on the AIST++ [10] dataset and present in-depth ablation studies on our overall frameworks.”
BackgroundMethodResultInfluentialEmotional Listener Portrait: Realistic Listener Motion Simulation in Conversation
Song, L., Yin, G., Jin, Z., Dong, X., & Xu, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01905 · Cited by 20
“humancomputer interaction [57, 25, 60], virtual reality [22, 24], metaverse [8, 7, 46] and media forensics [42, 37, 43, 17] etc.”
BackgroundYu, X., Elazab, A., Ge, R., Zhu, J., Zhang, L., Jia, G., Wu, Q., Wan, X., Li, L., & Wang, C. (2025) · Neural Networks · doi:10.1016/j.neunet.2024.107096 · Cited by 18
Music-to-Dance Generation with Multiple Conformer
Zhang, M., Liu, C., Chen, Y., Lei, Z., & Wang, M. (2022) · International Conference on Multimedia Retrieval · doi:10.1145/3512527.3531430 · Cited by 17
“As in [13], we use Fréchet Inception Distance (FID) [6], diversity, and beat alignment score as the evaluation metrics to measure the quality of generated dances.”
BackgroundMethodInfluentialMultimodal false information detection method based on Text-CNN and SE module
Liang, Y., Tohti, T., & Hamdulla, A. (2022) · PLoS ONE · doi:10.1371/journal.pone.0277463 · Cited by 17
Multi-Condition Latent Diffusion Network for Scene-Aware Neural Human Motion Prediction
Gao, X., Yang, Y., Wu, Y., Du, S., & Qi, G.-J. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3414935 · Cited by 17
“Considering that real-world human motions are compatible with their surrounding scene contexts, future human motions are joint conditional distributions of body joints within the scene context [46], [47], [48], [49], [50], [51].”
BackgroundFreeMan: Towards Benchmarking 3D Human Pose Estimation Under Real-World Conditions
Wang, J., Yang, F., Li, B., Gou, W., Yan, D., Zeng, A., Gao, Y., Wang, J., Jing, Y., & Zhang, R. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02075 · Cited by 17
MusicFace: Music-driven expressive singing face synthesis
Liu, P., Deng, W., Li, H., Wang, J., Zheng, Y., Ding, Y., Gu, X. D., & Zeng, M. (2023) · Computational Visual Media · doi:10.1007/s41095-023-0343-7 · Cited by 16
ASMNet: Action and Style-Conditioned Motion Generative Network for 3D Human Motion Generation
Li, Z., Wang, Y., Du, X., Wang, C., Koch, R., & Liu, M. (2024) · Cyborg and Bionic Systems · doi:10.34133/cbsystems.0090 · Cited by 16
“Conditioned motion generation tends to generate motion based on given control conditions, which can take various forms such as music [22,23], audio [24], speech [25,26], motion clips [27,28], and text information [17,29].”
BackgroundPROSE: Predicting Multiple Operators and Symbolic Expressions using multimodal transformers
Liu, Y., Zhang, Z., & Schaeffer, H. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106707 · Cited by 34
Vamos: Versatile Action Models for Video Understanding
Wang, S., Zhao, Q., Do, M. Q., Agarwal, N., Lee, K., & Sun, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73254-6_9 · Cited by 34
SAMR: Symmetric masked multimodal modeling for general multi-modal 3D motion retrieval
Li, Y., Wu, S., Zhu, Y., Sun, W., Zhang, Z., Song, S., & Zhai, G. (2025) · Displays · doi:10.1016/j.displa.2025.102987 · Cited by 17
Kim, D., Kim, T., Lee, I., & Lee, S. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3360452 · Cited by 15
“To demonstrate the proposed transformers, GCNs [39], [41] and multimodal transformers [35], [36], [45] are tested on OptiTrack and Real-world datasets for DanceQA.”
BackgroundMethodInfluentialHybridCap: Inertia-Aid Monocular Capture of Challenging Human Motions
Liang, H., He, Y., Zhao, C., Li, M., Wang, J., Jingyi Yu, A., & Xu, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25240 · Cited by 9
Loi, I., Zacharaki, E. I., & Μουστάκας, Κ. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3308753 · Cited by 16
“MDM is a transformer-based model that produces motion sequences conditioned on either natural language, audio clips [45], [59] or actions [60], while it can also generate motion without relying on conditions.”
BackgroundCheng, Q., Li, X., & Fu, X. (2024) · SIGGRAPH Asia Conference Papers · doi:10.1145/3680528.3687677 · Cited by 17
“Beat Consistency (BC) [30] calculates the average distance between every audio beat and its nearest motion beat.”
It Takes Two: Learning to Plan for Human-Robot Cooperative Carrying
Ng, E., Liu, Z., & Kennedy, M. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161386 · Cited by 15
“• Fretchet distance (FD): FD can be used to capture motion realism by measuring the distribution distance between generated and ground-truth motion sequences as used in [14], [15].”
MethodQEAN: quaternion-enhanced attention network for visual dance generation
Zhou, Z., Huo, Y., Huang, G., Zeng, A., Chen, X., Huang, L., & Li, Z. (2024) · The Visual Computer · doi:10.1007/s00371-024-03376-5 · Cited by 18
“Compared to the work of Li [19], the proposed SPE better increases the model’s representation and utilisation of positional information.”
BackgroundMethodInfluentialComputational kinematics of dance: distinguishing hip hop genres
Baker, B., Liu, T., Matelsky, J., Parodi, F., Mensh, B. D., Krakauer, J. W., & Kording, K. P. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1295308 · Cited by 14
Tanaka, R., Suzuki, T., & Fujii, K. (2024) · Proceedings of the 7th ACM International Workshop on Multimedia Content Analysis in Sports · doi:10.1145/3689061.3689077 · Cited by 17
“6M [31] (daily motions), MPI-INF-3DHP [45] (daily motions and outdoor activities), and AIST++ [38] (dance).”
BackgroundMethodInfluentialA Diffusion Modeling-Based System for Teaching Dance to Digital Human
Zhou, L., Zhao, J., & He, J. (2024) · Applied Sciences · doi:10.3390/app14199084 · Cited by 14
Zhuang, H., Lei, S., Xiao, L., Li, W., Chen, L., Yang, S., Wu, Z., Kang, S., & Meng, H. M. L. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095203 · Cited by 14
“We compare our proposed method with FACT [9] and Bai-lando [18], which are among the state-of-the-art dance generation methods.”
BackgroundMethodInfluentialSelf-citationDuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modeling
Ghosh, A., Zhou, B., Dabral, R., Wang, J., Golyanik, V., Theobalt, C., Slusallek, P., & Guo, C. (2025) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/3721238.3730741 · Cited by 21
AMD: Autoregressive Motion Diffusion
Han, B., Peng, H., Dong, M., Ren, Y., Shen, Y. L., & Xu, C. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.27973 · Cited by 23
BeatDance: A Beat-Based Model-Agnostic Contrastive Learning Framework for Music-Dance Retrieval
Yang, K., Zhou, X., Tang, X., Diao, R., Liu, H., He, J., & Fan, Z. (2024) · International Conference on Multimedia Retrieval · doi:10.1145/3652583.3658045 · Cited by 13
“However, these methods still face several challenges.”
Esaki, K., & Nagao, K. (2024) · Applied Sciences · doi:10.3390/app14145981 · Cited by 13
Dance2MIDI: Dance-driven multi-instrument music generation
Han, B., Li, Y., Shen, Y., Ren, Y., & Han, F. (2024) · Computational Visual Media · doi:10.1007/s41095-024-0417-1 · Cited by 19
Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-Speech Gesture Generation
Qi, X., Pan, J., Li, P., Yuan, R., Chi, X., Li, M., Luo, W., Xue, W., Zhang, S., Liu, Q., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00992 · Cited by 13
“Literally, the task most closely related to ours is AI choreographer [24, 26, 42] which generates the motion from music signals.”
BackgroundFürElise: Capturing and Physically Synthesizing Hand Motion of Piano Performance
Wang, R., Xu, P., Shi, H., Schumann, E., & Liu, C. K. (2024) · SIGGRAPH Asia Conference Papers · doi:10.1145/3680528.3687703 · Cited by 13
Keyframe Control of Music-Driven 3D Dance Generation
Yang, Z., Wen, Y., Chen, S., Liu, X., Gao, Y., Liu, Y., Gao, L., & Fu, H. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3235538 · Cited by 12
Breaking The Limits of Text-conditioned 3D Motion Synthesis with Elaborative Descriptions
Qian, Y., Urbanek, J., Hauptmann, A. G., & Won, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00219 · Cited by 14
“The first types of methods aim to generate motion sequences based on a specific kind of modality like music [19, 21, 22], speech [11, 5, 3], and text [23, 25, 1, 2] etc.”
BackgroundFu, C., Wang, Y., Zhang, J., Jiang, Z., Mao, X., Wu, J., Cao, W., Wang, C., Ge, Y., & Liu, Y. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680625 · Cited by 13
“We also incorporate Semantic-Relevant Gesture Recall (SRGR) [27] and BeatAlign [24] to evaluate the semantic relevance and synchrony of the generated gestures.”
MethodMasked Kinematic Continuity-aware Hierarchical Attention Network for pose estimation in videos
Jin, K.-M., Lee, G., Nam, W.-J., Kang, T.-K., Kim, H., & Lee, S. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.10.038 · Cited by 12
Chiquier, M., & Vondrick, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02019 · Cited by 12
“Others condition based on other modalities such as spatial trajectory [19], action class [13, 43, 54], natural language text [3, 44, 54], as well as audio [29, 5].”
BackgroundSun, Y., Dougherty, A. W., Zhang, Z., Choi, Y. K., & Wu, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01367 · Cited by 11
DanceComposer: Dance-to-Music Generation Using a Progressive Conditional Music Generator
Liang, X., Li, W., Huang, L., & Gao, C. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3405734 · Cited by 15
“Thus we introduce the metric Beat Align Score (BeatAlign) from the music-to-dance task [8] and define it as the average distance between every beat in generated music and its nearest ground truth music beat as: (10) where B g and B v record the time of generated beats and ground truth beats,…”
BackgroundMethodInfluentialZhao, Y., Wu, C., Huang, B., Zhi, Y., Zhao, C., Wang, J., & Gao, S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3599415 · Cited by 11
Unimotion: Unifying 3D Human Motion Synthesis and Understanding
Li, C., Chibane, J., He, Y., Pearl, N., Geiger, A., & Pons‐Moll, G. (2025) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv66043.2025.00027 · Cited by 11
“Effective controllability requires that an avatar be capable of executing detailed local sub-tasks according to a timeline while simultaneously understanding the overall global ob-jective.”
Finite Scalar Quantization as Facial Tokenizer for Dyadic Reaction Generation
Dam, Q. T., Nguyen, T. T. N., Tran, D. T., & Lee, J. (2024) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg59268.2024.10581929 · Cited by 11
“For generative predictors, the mainstream approach relies on either Trans-former [10], [8], [7], Diffusion [13], [3], or Normalizing Flow-Recurrent Neural Net [6].”
MethodWang, Z., Yang, J., & Fowlkes, C. C. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00258 · Cited by 13
“Mocap dataset We use [26] AMASS, AIST++ [18] and SPIN [15] dataset to train our occlusion aware GIKNet.”
InfluentialDance-Conditioned Artistic Music Generation by Creative-GAN
HUANG, J., HUANG, X., Yang, L., & Tao, Z. (2023) · IEICE Transactions on Fundamentals of Electronics Communications and Computer Sciences · doi:10.1587/transfun.2023eap1059 · Cited by 10
MOSA: Music Motion With Semantic Annotation Dataset for Cross-Modal Music Processing
Huang, Y.-F., Moran, N., Coleman, S., Kelly, J., Wei, S.-H., Chen, P.-Y., Huang, Y., Chen, T.-P., Kuo, Y. S., Wei, Y.-C., et al. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3407529 · Cited by 10
“Since human body motion is highly time-dependent and space-dependent, for motion generation, we design time-wise loss to enhance the continuity of the generated motion along the timeline, and space-wise loss to regulate the relative association among x-, y, and z-dimensions in the space.”
BackgroundDiffusion-Based Graph Generative Methods
Chen, H., Xu, C., Zheng, L., Zhang, Q., & Lin, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3466301 · Cited by 11
“AIST++ [176] is a dataset with high-quality dance motions with 10 different dance motion genres.”
BackgroundTowards Emotion-enriched Text-to-Motion Generation via LLM-guided Limb-level Emotion Manipulating
Yu, T., Wang, J.-J., Wang, J.-W., Luo, J.-M., & Zhou, G.-D. (2024) · ACM Multimedia · doi:10.1145/3664647.3681487 · Cited by 17
“Human motion generation leverages various control signals (e.g., music [13, 14], action categories [11, 21] and text descriptions [10, 35, 37]) to generate vivid and realistic motions, which has numerous practical applications in fields such as game production, film and virtual reality.”
Dance-to-Music Generation with Encoder-based Textual Inversion
Li, S., Dong, W., Zhang, Y., Tang, F., Ma, C., Deußen, O., Lee, T., & Xu, C. (2024) · SIGGRAPH Asia Conference Papers · doi:10.1145/3680528.3687562 · Cited by 11
Beat-It: Beat-Synchronized Multi-condition 3D Dance Generation
Huang, Z., Xu, X., Xu, C., Zhang, H., Zheng, C., Qin, J., & He, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72655-2_16 · Cited by 10
Cross-Modality Object Detection Based on DETR
Huang, X. L. D., & Ma, G. (2025) · IEEE Access · doi:10.1109/access.2025.3551947 · Cited by 12
“Multimodal Transformer has proven effective in tasks such as image segmentation [18], [19], cross-modality sequence generation [20], [21], and multispectral object detection [12], [13].”
ChoreoCraft: In-situ Crafting of Choreography in Virtual Reality through Creativity Support Tool
Han, H., Jung, K., & Yoon, S. H. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714220 · Cited by 10
“…of algorithmic assistance in the choreographic process has been a topic of recent research, with a focus on generating dance motions through various input modalities such as music [2, 48, 78, 88, 90, 94], music & motion [30, 45, 71, 72, 74, 80], music & text [32], video [13], motion & text [46].”
MMoFusion: Multi-modal co-speech motion generation with diffusion model
Wang, S., Zhang, J., Tan, X., Xie, Z., Wang, C., & Ma, L. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111774 · Cited by 10
The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion
Chen, C., Zhang, J., Lakshmikanth, S. K., Fang, Y., Shao, R., Wetzstein, G., Fei-Fei, L., & Adeli, E. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00581 · Cited by 25
BRACE: The Breakdancing Competition Dataset for Dance Motion Synthesis
Moltisanti, D., Wu, J., Dai, B., & Loy, C. C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20074-8_19 · Cited by 10
VR Dance Training System Capable of Human Motion Tracking and Automatic Dance Evaluation
Esaki, K., & Nagao, K. (2022) · PRESENCE Virtual and Augmented Reality · doi:10.1162/pres_a_00383 · Cited by 9
“To achieve efficient 3D pose estimation suitable for dancing, we first trained a convolutional neural network (CNN)-based model for 2D pose estimation by using the AIST dataset (Tsuchida et al., 2019) and keypoint annotation in AIST ++ (Li et al., 2021).”
BackgroundMethodMAMMOS: MApping Multiple human MOtion with Scene understanding and natural interactions
Lim, D., Jeong, C., & Kim, Y. M. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00462 · Cited by 13
“Recent studies even synthe-size motion without any explicit pose information, either from languages [1, 2, 10, 27, 28, 4, 9, 34, 38] or music [35, 19, 20, 21].”
BackgroundCoDancers: Music-Driven Coherent Group Dance Generation with Choreographic Unit
Yang, K., Tang, X., Diao, R., Liu, H., He, J., & Fan, Z. (2024) · International Conference on Multimedia Retrieval · doi:10.1145/3652583.3657998 · Cited by 11
“In Solo Music2Dance tasks, with the recent progress in deep learning, various structures have been designed, including CNNs[9], RNNs[1, 10, 31], GCNs[6, 27, 34], GANs[16, 30], Transformers[17, 18] and Diffusions[25, 35].”
MethodInfluentialA Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals
Tang, J., Wang, J., Ji, K., Xu, L., Jingyi Yu, A., & Ye, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02008 · Cited by 12
“Human motion can be generated by any signal that describes the motion, including text [4, 9, 20, 31, 44, 51, 52, 61, 68, 72], audio [35, 54], action labels [9, 19, 42, 51], or unconditioned [44, 74, 75].”
BackgroundDanceCamera3D: 3D Camera Movement Synthesis with Music and Dance
Wang, Z., Jia, J., Sun, S., Wu, H., Han, R. D., Li, Z., Tang, D., Zhou, J., & Luo, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00754 · Cited by 11
“Since this feature extractor calculates average velocity and acceleration, we compute kinetic features on split 2.5-second data to ensure the density of feature distribution which is similar to settings in AIST++ [27].”
BackgroundMethodInfluentialHUMOS: Human Motion Model Conditioned on Body Shape
Tripathi, S., Taheri, O., Lassner, C., Black, M. J., Holden, D., & Stoll, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72640-8_8 · Cited by 9
Revisit Human-Scene Interaction via Space Occupancy
Liu, X., Hou, H., Yang, Y., Li, Y., & Lu, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72973-7_1 · Cited by 17
ReinDiffuse: Crafting Physically Plausible Motions with Reinforced Diffusion Model
Han, G., Liang, M., Tang, J., Cheng, Y., Liu, W., & Huang, S. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00222 · Cited by 25
“Conditioned inputs include text [2,5,9,11,12,29], music [20,21,23,34], audio [1,44], images [6, 31], objects [22], and trajectories [17, 36, 39].”
Evaluation of generative models for emotional 3D animation generation in VR
Chhatre, K., Guarese, R., Matviienko, A., & Peters, C. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2025.1598099 · Cited by 9
“…metrics —for instance, Frechet Gesture Distance [64, 125] (comparing latent features between generated and ground-truth motion), beat alignment [56, 112] (assessing motion-speech correlation via kinematic and audio beats), semantic-relevance gesture recall [60], and gesture diversity [55, 59]…”
Retrieving Semantics from the Deep: an RAG Solution for Gesture Synthesis
Mughal, M. H., Dabral, R., Scholman, M., Demberg, V., & Theobalt, C. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01545 · Cited by 11
Multi-Granularity Interaction for Multi-Person 3D Motion Prediction
Liu, C., & Mu, Y. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3298755 · Cited by 10
“T HE technique of human 3D motion prediction has a wide range of real-world applications, ranging from 3D character animations [1], [2], [3], decision-making systems for The authors are with the Wangxuan Institute of Computer Technology, Peking University, Beijing 100871, China (e-mail:…”
MethodTaming Diffusion Models for Music-Driven Conducting Motion Generation
Zhao, Z., Bai, J., Chen, D., Wang, D., & Pan, Y. (2023) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v1i1.27474 · Cited by 12
Bridging the Gap Between Human Motion and Action Semantics via Kinematic Phrases
Liu, X., Li, Y., Zeng, A., Zhou, Z., You, Y., & Lu, C. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73242-3_13 · Cited by 14
PoseExaminer: Automated Testing of Out-of-Distribution Robustness in Human Pose and Shape Estimation
Liu, Q., Kortylewski, A., & Yuille, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00072 · Cited by 9
Co-speech Gesture Synthesis by Reinforcement Learning with Contrastive Pretrained Rewards
Sun, M., Zhao, M., Hou, Y., Li, M., Huang, X., Xu, S., & Hao, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00231 · Cited by 9
“Existing works focus on exploring the effectiveness of different network architectures, including the multi-layer perceptron (MLP) [18], convolutional neural networks (CNNs) [11], recurrent neural networks (RNNs) [4] and Transformers [18, 29].”
MethodVeRi3D: Generative Vertex-based Radiance Fields for 3D Controllable Human Image Synthesis
Chen, X., Huang, J., Bin, Y., Yu, L., & Liao, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00825 · Cited by 8
Exploring AI-assisted Ideation and Prototyping for Choreography
Liu, Y., & Sra, M. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640544.3645227 · Cited by 8
“We further fine-tuned this model using the AIST++ dance dataset [26].”
Method
Metadata
- Authors
- Li, R., Yang, S., Ross, D. A., & Kanazawa, A.
- Affiliations
- University of Southern California (US)
- Google (United States) (US)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 13381-13392
- OpenAlex
- W3204221554
- Cited by
- 553
- References
- 94
- Citations by year
- 2026: 92 · 2025: 146 · 2024: 154 · 2023: 113 · 2022: 46 · 2021: 2
- FWCI
- 117.49
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Keywords
- cross-modal transformer, full attention, AIST, dance genres, choreography
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
