The DOI matches its Crossref record; 90 of the 94 references it cites were found in a registry or an index.
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
Open accessAbstract
We present AIST++, a new multi-modal dataset of 3D dance motion and music, along with FACT, a Full-Attention Cross-modal Transformer network for generating 3D dance motion conditioned on music. The proposed AIST++ dataset contains 5.2 hours of 3D dance motion in 1408 sequences, covering 10 dance genres with multi-view videos with known camera poses—the largest dataset of this kind to our knowledge. We show that naively applying sequence models such as transformers to this dataset for the task of music conditioned 3D motion generation does not produce satisfactory 3D motion that is well correlated with the input music. We overcome these shortcomings by introducing key changes in its architecture design and supervision: FACT model involves a deep cross-modal transformer block with full-attention that is trained to predict N future motions. We empirically show that these changes are key factors in generating long sequences of realistic dance motion that are well-attuned to the input music. We conduct extensive experiments on AIST++ with user studies, where our method outperforms recent state-of-the-art methods both qualitatively and quantitatively. The code and the dataset can be found at: https://google.github.io/aichoreographer.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We note that the simple prediction-based late fusion [247], [248] is less adopted in MML Transformers.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [248]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 8×
“We propose Full Attention Cross-Modal Transformer (FACT) model for the task of 3D dance motion generation.”
BackgroundInfluentialAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“All our experiments are trained with 16 batch size using Adam [45] optimizer.”
MethodJakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 2×
“On the other hand, models like BERT [17] employ full-attention for feature learning, but rarely are these models employed in an auto-regressive manner, due to its inefficiency at test time.”
MethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 2×
“On the other hand, models like BERT [17] employ full-attention for feature learning, but rarely are these models employed in an auto-regressive manner, due to its inefficiency at test time.”
MethodGANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., & Sheikh, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2929257 · Cited by 5,536 · cited 1×
“Training data for 2D pose and audio is abundant thanks to the high reliability of 2D pose detectors [14].”
BackgroundDeep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., & J, B. M. (2015) · ACM Transactions on Graphics · doi:10.1145/2816795.2818013 · Cited by 3,734
Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653 · cited 2×
“While there are many large scale 3D motion capture datasets [39, 59, 1, 37], mocap dataset of 3D dance is quite limited as it requires heavy instrumentation and expert dancers for capture.”
BackgroundMethod
Research works printed but not linked: 3
- lamothe, 2019, The dancing species: how moving together in time helps make us human, Aeon
- alemi, 2017, Groovenet: Real-time music-driven dance movement generation using artificial neural networks, Networks, 8, 26
- chen, 2021, Choreomaster: choreography-oriented music-driven dance synthesis, ACM Transactions on Graphics (TOG), 40, 1
Cited by
Sorted by citations
Items 501–553 of 553
Dance Motion-Guided Music Generation via Residual Vector Quantization
Lin, S., Moshe Zukerman, D., & Yan, H. (2026) · Electronics · doi:10.3390/electronics15102098
Hierarchical Cross-Modal Fusion Model for Field Vehicle Classification
Zhang, J., Li, B., Liu, H., Tang, H., Xia, L., & Huang, J. (2026) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2026.3693447
“To address the modality information imbalance in multimodal fusion, we adopt a multilevel fusion strategy from the perspective of abstract semantic information [24], aiming to mitigate intermodality discrepancies and achieve e ff ective integration.”
Hierarchical Diffusion for Sparse-to-Full Human Motion Reconstruction
Du, F., Yang, Y., Gao, X., Hou, H., & Sun, C. (2026) · IEEE Signal Processing Letters · doi:10.1109/lsp.2026.3694693
“Be-yondimagegeneration[15],theyhavealsobeenadoptedforhu-manmotionsynthesisundervariousconditions[16],[17],[18], [19],[20].”
洪浩偉, & Zheng, Z. (2025) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait67930.2025.11522126
“Currently, several types of conditional methods have emerged, using modalities such as action labels [11-13], [19], music [2], [20-22], speech [23], [24], control signals [25], and text [18], [26-28].”
Li, J. W., Zhang, Z., Wang, B., Liao, H., & Han, J. (2026) · IEEE Signal Processing Letters · doi:10.1109/lsp.2026.3695826
“We employ the AIST++ dataset [9], containing 1,408 high-fidelity dance sequences paired with music across diverse gen-res.”
InfluentialSynSinger: Audio-Driven Singing Avatar Motion Synthesis
Wang, Y., & Xu, Z. S. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-9507-5_24
LaMP: Learning Robust Latent Motion Prior for Optimization-Based Human Motion Generation
Lyu, X., Karunratanakul, K., Zhao, K., & Tang, S. (2026) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv69130.2026.00090
“Those works generate human motion both conditionally and unconditionally [29, 37, 45, 47, 48] such as text [1, 2, 7, 9, 32], audio [17, 18, 22], and actions [3, 11, 30].”
Co-speech holistic 3D motion generation with style from video
Zhang, Y., Wen, Y., Li, C., Jing, L., & Yu, J. (2026) · Graphical Models · doi:10.1016/j.gmod.2026.101332
Koukopoulos, Z., Koukopoulos, D. K., & Fidas, C. A. (2026) · Digital Applications in Archaeology and Cultural Heritage · doi:10.1016/j.daach.2026.e00565
Text Dictates, Music Decorates: Energy-Based Attention for Editable Dance Motion Generation
Yoo, S. J., Peng, S., Gu, F., Aloimonos, S., & Fermuller, C. M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37369-4_14
Odoriko: A Shape-Aware Multimodal Diffusion Framework for Human Motion
Shim, D., Tanke, J., Uchida, K., Simon, C., Saito, K., Shibuya, T., Takahashi, S., & Mitsufuji, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37435-6_26
Audio-Related Multimodal Learning
Chen, W., Sun, Y., & Xu, K. (2026) · Springer eBooks · doi:10.1007/978-981-95-0836-5_21
Fine-Grained Human Action Recognition Via Laban Movement Analysis and Attentive Bi-LSTM
Wang, M., Xun, M., & Yun, Q. (2026) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001426550141
BeatDance: Generating beat-consistent 3D dance with hierarchical spatial–temporal modeling
Shen, X., Shi, D., Zhang, J. L., Li, H., Zhao, H., Zhang, D., Zhuge, Y., Wu, Z., Yue, G., & Zhou, W. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.114344
OmniDance: Multimodal Driven Dance Video Generation with Large-Scale Internet Data
Yang, K., Zhu, J., Tang, X., Peng, Z., Zhang, X., Chen, C., Wang, P., Wu, J., Chu, X., Liu, H., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37369-4_31 · Cited by 1
Legato: Articulation and Dynamic-Aware Dance Generation via VQ-VAE
Park, S. Y., Park, J. H., & Shon, J. G. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-7896-2_39
ARMS: Anchor–Relational Motion Streaming for Seamless Solo-Social Motion Transitions
Liu, H., Yu, Q., Fujiwara, K., Uchiyama, H., & Kiyokawa, K. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37232-1_28
Fieraru, M. (2026) · Journal of Imaging · doi:10.3390/jimaging12070311
“…from confounding factors such as differences in architecture and other training data, we run a controlled fine-tuning experiment, evaluating the resulting checkpoints both on the Fit3D test set (Table 5) and on AIST++ [54] as an out-of-domain generalization test on street dance motions (Table 6).”
DMP: Directable Motion Retargeting through Motion Paraphrasing
Lee, S., Rempe, D., Jiang, Y., Zhang, H., Wang, T., Won, J., & Peng, X. B. (2026) · ACM Transactions on Graphics · doi:10.1145/3829359
“In line with prior works [Dabral et al. 2023b; Li et al. 2021; Zhang et al. 2025], we compute FID with well-established kinetic features that captures dynamic aspects of motions [Onuma et al. 2008], using the fairmotion implementation [Gopinath and Won 2020].”
Music-to-Dance Generation via Atomic Movements
Cai, X., Sun, Y., Zheng, M., Chen, Q., Jin, X., Zhu, S. C., & Liu, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37577-3_13
Rehabilitation movement simulation via joint angle-based generative AI
Santangelo, G., Alessi, C., Nicora, G., Sacchi, N., Pe, S., Ferrara, A., Bellazzi, R., & Dagliati, A. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103488
Xiao, Q., Lu, K., Zhang, M., Li, Q., & Yang, W. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02539-9
“The explicit models mainly refer to parameterized mesh techniques [5, 6 ].”
TECMO: Efficient Te mporal C ross-Interaction Mo dule for Long Synchronized Audio-Video Generation
Ergasti, A., Tarollo, G., Botti, F., Fontanini, T., Ferrari, C., Bertozzi, M., & Prati, A. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3717470
“AIST++ [32] is a subset of AIST [33] that provides 5.2 hours of video featuring paired audio and dancer movements.”
InfluentialGeneration in Generation: Fluid Co-Speech Gesture Synthesis With Generative Continuous Quantization
Li, J. Q., Zhao, Y., Guo, X., & Li, J. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3715321
“3) Beat Alignment Score (BC): The Beat Consistency score, introduced by Li et al. [91], measures the alignment between motion beats and audio beats.”
Creative Strategy for Choreographer’s Interaction with Artificial Intelligence
Gryzunova, O. V. (2026) · Observatory of Culture · doi:10.25281/2072-3156-2026-23-3-267-277
Mocap Denoising via Sparsity Optimization on High‐Order Motion Feature
Li, C., Liu, L., & Zhang, Y. (2026) · Computer Graphics Forum · doi:10.1111/cgf.70585
ChoreDiffusion: Beat-Aware Diffusion for Music-to-Dance Generation
Gao, Y., Wu, Q., Zhu, S., He, K., Weng, W., & Zhou, J. (2026) · Information · doi:10.3390/info17080808
“The blue curve is the kinematic velocity of the generated motion; green dashed lines mark kinematic beats, extracted as local minima of the kinematic velocity following the AIST++ protocol [5]; orange dashed lines are the music beats detected from the audio.”
InfluentialL-dance: LLM-guided motion intensity aware natural dance generation
Shin, W., Jung, Y.-H., Lee, M. Y., & Seo, S. (2026) · The Visual Computer · doi:10.1007/s00371-026-04706-5
Yeh, H.-H., Lin, F.-C., & Jiang, C.-T. (2026) · IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan) · doi:10.1109/icce-taiwan71481.2026.11652385
“We evaluate on AIST++ [6] (5.2 h of motion data).”
InfluentialRevised Research on Dance Movement Structure and Teaching Reconstruction Driven by Generative AI
Fang, X. (2026) · SHS Web of Conferences · doi:10.1051/shsconf/202623601029
“…of retaining the original dance artistic conception; when choreographers input lyrical piano music, the model preferentially outputs slow, soft and smooth body trajectory sequences, while dynamic electronic music corresponds to high-frequency, high-amplitude jumping and rotating movements[10].”
HardMo++: A Large-Scale Hard-Case Dataset for Motion Capture
Peng, J., Shen, S., Li, Z., Liu, Y., & Zhu, Z. (2026) · Electronics · doi:10.3390/electronics15173870
StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring
Li, J., Zhang, X., Zhang, J., Yang, K., & Hoi, S. C. H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37595-7_32
“Beat Consistency (BC) [18] measures temporal alignment between motion and speech beats.”
ShapeRhythm: Personalized and Harmonic Dance Generation Bridging Diverse Bodies and Melodies
Sun, M., Tian, Z., Qi, X., Chen, S., 胡旭睿, Liu, Q., Zhang, M., Sun, Z., & Shan, C. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-03017-5
ChoreoMamba: Long-Sequence Dance Generation with Bidirectional Mamba
Tan, V., Kim, H., Noh, J., & Choi, S. (2026) · Journal of the Korea Computer Graphics Society · doi:10.15701/kcgs.2026.32.4.1
DanceCrafter: A two-stage framework for high-quality music-driven image animation
Wang, Y., Wang, R., Cheng, Z., & Song, L. (2026) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2026.104965
OmniPoser: Flexible Human Motion Recovery in the Wild with Masked Flow Matching
Liu, M., & Tang, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37032-7_1
“We evaluate on AIST++ [18], TotalCapture [34], and Nymeria, comparing with Stereo-Inertial Poser (SIP) [33] and RobustCap [26].”
Huang, M., Luo, B., & Ayi, S. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.08.069
Yu, R., Liu, Y., Wang, J., Xu, C., Shao, Y., Sun, B., Liu, Y., & Luo, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37174-4_27
Esaki, K., & Nagao, K. (2026) · Virtual Reality · doi:10.1007/s10055-026-01499-9
Towards music-aware robot dance generation: bridging rhythm, motion and control
Yang, Q., Li, M., Wang, J., Lu, Y., Zhao, Q., Shen, Q., & Cao, X. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02679-y
“The advent of cross-modal Transformers [7– 9 Recently, diffusion models [ 10–13 , 27 ] Although motion refinement and inpainting methods have been explored to complete missing regions and interpolate between constrained poses [28–30], they primarily focus on motion completion or transition…”
HiChor: Hierarchical Choreography Generation from Pop Music with Choreographic Primitives
Kim, J. J., Huh, J., Choi, J., Cheng, W.-H., Jiang-Lin, J.-Y., Lin, W., & Lee, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37520-9_7
Evaluating Cross-Cultural Generative Embodied Social Agents
Funk, M., Okada, S., & André, E. (2026) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3776591.3836651
“(4 ) Complementing FGD, different metrics based on the beat finding approach of Li et al. [23] have emerged to add synchronization to speech rhythm.”
InfiniteDance: Scalable Dance Generation Towards in-the-wild Generalization
Li, R., Hu, Z., Li, S., Zhang, Y., Xie, H., Zhang, M., Guo, J., Li, X., & Liu, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37083-9_15 · Cited by 3
Self-citationOmniMotion-X: Versatile Multimodal Whole-Body Motion Generation
Xu, G., Bian, Y., Zeng, A., Chen, Z., Shi, M., Huang, S., Li, W., Duan, L., & Xu, Q. (2026) · doi:10.1109/cvprf71909.2026.00353
霍宇飞, Li, A., Dai, W., Wu, S., & Tang, Y. (2026) · doi:10.1109/cvprf71909.2026.00335
“FACT[32] takes both music and random motion as input to a Trans-former network, generating new dance frames autoregres-sively per frame basis.”
InfluentialContact Matrix: Enhancing Dance Motion Synthesis with Precise Interaction Modeling
Chen, X., Cen, Z., Pi, H., Peng, S., Zhou, X., & Liu, Y. (2026) · doi:10.1109/cvprf71909.2026.00349
FUSION: Full-body Unified Motion Prior for Body and Hands via Diffusion
Duran, E., Athanasiou, N., Kocabas, M., Black, M. J., & Taheri, O. (2026) · doi:10.1109/cvprf71909.2026.00334
“These meth-ods use various conditioning signals such as text descriptions [2, 3, 20, 34, 40, 42, 48, 49, 57], audio [22, 51], trajectories [18, 38], action labels [33], and object-related information [5, 9, 21, 35, 46, 47, 56, 60].”
Nagy, R., Voß, H., Hoang-Minh, T., Tsakov, M., Nikolov, T., Zhang, Z., Ao, T., Yang, S., Huang, S., Cheng, Y., et al. (2026) · doi:10.1109/cvprf71909.2026.00211
Self-citationDanceLongCam: High-speed stable long-sequence 3D dance camera movement synthesis
Wang, X., Huang, X., Hu, S., Cai, J., & Zhang, S. (2026) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2026.104979
SportRecast: Bringing New Identities to Dynamic Sports Scenes
Tang, H., Qian, J., Zhe, S., Jia, C., Zhang, J., Pan, X., Guo, P., Zhao, L., & 马思伟 (2026) · IEEE International Conference on Multimedia and Expo workshops · doi:10.1109/icmew69720.2026.11720708
“The experimental data consist of three parts: AIST++ [16], NeuMan [17], and sports videos collected from the Internet.”
Cyclic phase and reference recovery from event-window LDP motion descriptors
Wang, Y., Wen, Y., Du, R., Fu, Y., & Liang, E. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.117209
TempoPose: Co-Speech Gesture Generation With Dynamic Adaptation for Efficiency and Quality
Zhang, M., & Wang, S. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729692
“Diversity reflects motion variation via average L1 distance across gesture samples [29].”
Metadata
- Authors
- Li, R., Yang, S., Ross, D. A., & Kanazawa, A.
- Affiliations
- University of Southern California (US)
- Google (United States) (US)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 13381-13392
- OpenAlex
- W3204221554
- Cited by
- 553
- References
- 94
- Citations by year
- 2026: 92 · 2025: 146 · 2024: 154 · 2023: 113 · 2022: 46 · 2021: 2
- FWCI
- 117.49
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Human Motion and Animation
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Keywords
- cross-modal transformer, full attention, AIST, dance genres, choreography
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
