CoChat
Sign Up
PreprintCited by 1Generative Adversarial Networks and Image SynthesisHuman Motion and AnimationSpeech and Audio Processing

EMO2: End-Effector Guided Audio-Driven Avatar Video Generation

Open access
Tian, Linrui, Hu, Siqi, Wang, Qi, Zhang, Bang, Bo, Liefeng·2025·arXiv
DOI 10.48550/arxiv.2501.10687arXiv 2501.10687
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we propose a novel audio-driven talking head method capable of simultaneously generating highly expressive facial expressions and hand gestures. Unlike existing methods that focus on generating full-body or half-body poses, we investigate the challenges of co-speech gesture generation and identify the weak correspondence between audio features and full-body gestures as a key limitation. To address this, we redefine the task as a two-stage process. In the first stage, we generate hand poses directly from audio input, leveraging the strong correlation between audio signals and hand movements. In the second stage, we employ a diffusion model to synthesize video frames, incorporating the hand poses generated in the first stage to produce realistic facial expressions and body movements. Our experimental results demonstrate that the proposed method outperforms state-of-the-art approaches, such as CyberHost and Vlogger, in terms of both visual quality and synchronization accuracy. This work provides a new perspective on audio-driven gesture generation and a robust framework for creating expressive and natural talking head animations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2501.10687

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent speech-driven video generation methods such as Loopy, Hallo3, and EMO predominantly focus on synthesizing talking-head videos confined to facial regions, neglecting full-body movements and multi-character interactions.

    “On the other hand, recent speech-driven video generation methods, such as Loopy, Hallo3, and EMO [18, 24, 33-35, 40], predominantly focus on synthesizing talking-head videos confined to facial regions. These approaches neglect essential fullbody movements and multi-character interactions critical for expressive storytelling, thus significantly limiting their applicability in realistic and inter”

    From MoCha: Towards Movie-Grade Talking Character Synthesis (2025) · cites this paper as [35]

    The “Audio-driven human video generation” section (p. 2) says the field often focuses on talking heads and discusses EMO, Loopy, and Hallo in that context. Section 3.3 (p. 3) contrasts EMO’s facial-expression approach with EMO2’s body-motion generation. The copy does not establish that all three named methods are confined to facial regions or that they neglect multi-character interactions. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Unlike prior works that rely on external control signals, MoCha is trained directly on text and speech without auxiliary conditioning, which simplifies the model architecture and improves motion diversity and generalization.

    “Unlike prior works such as EMO [33, 35], SONIC [17], Echomimicv2 [24], Loopy [18], and Hallo3 [40], which rely heavily on external control signals (e.g., reference images, skeletons, keypoints), MoCha is trained directly on text and speech without any auxiliary conditioning. This simplifies the model architecture and improves motion diversity and generalization.”

    From MoCha: Towards Movie-Grade Talking Character Synthesis (2025) · cites this paper as [35]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 44

  1. Bie, X., Guo, W., Leglaive, S., Girin, L., Moreno-Noguer, F., and Alameda-Pineda, X. Hit-dvae: Human motion generation via hierarchical transformer dynamical vae. arXiv preprint arXiv:2204.01565 , 2022. · arXiv
  2. Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., and Li, Z. Pixartα : Fast training of diffusion transformer for photorealistic text-to- Ours image synthesis, 2023. URL https://arxiv.org/ abs/2310.00426 . · arXiv
  3. Chen, J., Liu, Y., Wang, J., Zeng, A., Li, Y., and Chen, Q. Diffsheg: A diffusion-based approach for real-time speech-driven holistic 3d expression and gesture generation, 2024. URL https://arxiv.org/abs/2401. 04747 .
  4. Chung, J. S. and Zisserman, A. Out of time: automated lip sync in the wild. In Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13 , pp. 251-263. Springer, 2017.
  5. Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., and Rubinstein, M. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. arXiv preprint arXiv:1804.03619 , 2018. · arXiv
  6. Fan, Y., Lin, Z., Saito, J., Wang, W., and Komura, T. Faceformer: Speech-driven 3d facial animation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.
  7. Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 , 2023. · arXiv
  8. Habibie, I., Xu, W., Mehta, D., Liu, L., Seidel, H.-P., PonsMoll, G., Elgharib, M., and Theobalt, C. Learning speechdriven 3d conversational gestures from video, 2021. URL https://arxiv.org/abs/2102.06837 . · arXiv
  9. HaCohen, Y., Chiprut, N., Brazowski, B., Shalem, D., Moshe, D., Richardson, E., Levin, E., Shiran, G., Zabari, N., Gordon, O., Panet, P., Weissbuch, S., Kulikov, V., Bitterman, Y., Melumian, Z., and Bibi, O. Ltxvideo: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103 , 2024. · arXiv
  10. He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CoRR , abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385 . · arXiv
  11. He, X., Huang, Q., Zhang, Z., Lin, Z., Wu, Z., Yang, S., Li, M., Chen, Z., Xu, S., and Wu, X. Co-speech gesture video generation via motion-decoupled diffusion model, 2024. URL https://arxiv.org/abs/2404.01862 . · arXiv
  12. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.
  13. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.
  14. Hogue, S., Zhang, C., Daruger, H., Tian, Y., and Guo, X. Diffted: One-shot audio-driven ted talk video generation with diffusion-based co-speech gestures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , pp. 1922-1931, June 2024.
  15. Hor´ e, A. and Ziou, D. Image quality metrics: Psnr vs. ssim. In 2010 20th International Conference on Pattern Recognition , pp. 2366-2369, 2010. doi: 10.1109/ICPR. 2010.579. · DOI
  16. Hu, L., Gao, X., Zhang, P., Sun, K., Zhang, B., and Bo, L. Animate anyone: Consistent and controllable image-tovideo synthesis for character animation. arXiv preprint arXiv:2311.17117 , 2023. · arXiv
  17. Hu, S., Babaians, E., Karimi, M., and Steinbach, E. Nmpcmp: Real-time nonlinear model predictive control for safe motion planning in manipulator teleoperation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pp. 8309-8316, 2021. doi: 10.1109/IROS51168.2021.9636802. · DOI
  18. Idrees, S., Choi, J., and Sohn, S. Advmt: Adversarial motion transformer for long-term human motion prediction. arXiv preprint arXiv:2401.05018 , 2024. · arXiv
  19. Jiang, J., Liang, C., Yang, J., Lin, G., Zhong, T., and Zheng, Y. Loopy: Taming audio-driven portrait avatar with long-term motion dependency, 2024. URL https: //arxiv.org/abs/2409.02634 .
  20. Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014.
  21. Kipp, M. Gesture generation by imitation: from human behavior to computer character animation. 2005. URL https://api.semanticscholar. org/CorpusID:26271318 .
  22. Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., Zhang, J., Wu, K., Lin, Q., Yuan, J., Long, Y., Wang, A., Wang, A., Li, C., Huang, D., Yang, F., Tan, H., Wang, H., Song, J., Bai, J., Wu, J., Xue, J., Wang, J., Wang, K., Liu, M., Li, P., Li, S., Wang, W., Yu, W., Deng, X., Li, Y., Chen, Y., Cui, Y., Peng, Y., Yu, Z., He, Z., Xu, Z., Zhou, Z., Xu, Z., Tao, Y., Lu, Q., Liu, S., Zhou, D., Wang, H., Yang, Y., Wang, D., Liu, Y., Jiang, J., and Zhong, C. Hunyuanvideo: A systematic framework for large video generative models, 2025. URL https://arxiv.org/abs/2412.03603 . · arXiv
  23. Lin, G., Jiang, J., Liang, C., Zhong, T., Yang, J., and Zheng, Y. Cyberhost: Taming audio-driven avatar diffusion model with region codebook attention, 2024. URL https://arxiv.org/abs/2409.01876 . · arXiv
  24. Meng, R., Zhang, X., Li, Y., and Ma, C. Echomimicv2: Towards striking, simplified, and semi-body human animation, 2024. URL https://arxiv.org/abs/2411. 10061 .
  25. Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A. A. A., Tzionas, D., and Black, M. J. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , pp. 10975-10985, 2019.
  26. Peebles, W. and Xie, S. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748 , 2022. · arXiv
  27. Qi, X., Zhang, H., Wang, Y., Pan, J., Liu, C., Li, P., Chi, X., Li, M., Xue, W., Zhang, S., Luo, W., Liu, Q., and Guo, Y. Cocogesture: Toward coherent co-speech 3d gesture generation in the wild, 2024. URL https://arxiv. org/abs/2405.16874 .
  28. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pp. 8748-8763. PMLR, 2021.
  29. Rakita, D., Mutlu, B., and Gleicher, M. Relaxedik: Realtime synthesis of accurate and feasible robot arm motion. In Robotics: Science and Systems , volume 14, pp. 26-30. Pittsburgh, PA, 2018.
  30. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 10684-10695, 2022.
  31. Romero, J., Tzionas, D., and Black, M. J. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) , 36 (6), November 2017.
  32. Schneider, S., Baevski, A., Collobert, R., and Auli, M. wav2vec: Unsupervised pre-training for speech recognition. pp. 3465-3469, 09 2019. doi: 10.21437/Interspeech. 2019-1873. · DOI
  33. Tian, L., Wu, J., Xiong, Z., and Zhu, X. Vote for grasp poses from noisy point sets by learning from human. In 2021 27th International Conference on Mechatronics and Machine Vision in Practice (M2VIP) , pp. 349-356. IEEE, 2021.
  34. Tian, L., Wang, Q., Zhang, B., and Bo, L. Emo: Emote portrait alive generating expressive portrait videos with audio2video diffusion model under weak conditions. In European Conference on Computer Vision , pp. 244-260. Springer, 2025.
  35. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips. cc/paper\_files/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper. pdf .
  36. Wagner, P., Malisz, Z., and Kopp, S. Gesture and speech in interaction: An overview. Speech Communication , 57:209-232, 2014. ISSN 01676393. doi: https://doi.org/10.1016/j.specom.2013.09. 008. URL https://www.sciencedirect.com/ science/article/pii/S0167639313001295 . · DOI
  37. Xing, J., Xia, M., Zhang, Y., Cun, X., Wang, J., and Wong, T.-T. Codetalker: Speech-driven 3d facial animation with discrete motion prior, 2023. URL https://arxiv. org/abs/2301.02379 .
  38. Xu, M., Li, H., Su, Q., Shang, H., Zhang, L., and Liu, C. Hallo: Hierarchical audio-driven visual synthesis for portrait image animation, 2024. URL https:// arxiv.org/abs/2406.08801 . · arXiv
  39. Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072 , 2024. · arXiv
  40. Yi, H., Liang, H., Liu, Y., Cao, Q., Wen, Y., Bolkart, T., Tao, D., and Black, M. J. Generating holistic 3d human motion from speech. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 469-480, June 2023.
  41. Zadeh, A. B., Liang, P. P., Poria, S., Cambria, E., and Morency, L.-P. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 2236-2246, 2018.
  42. Zhang, Y., Gu, J., Wang, L.-W., Wang, H., Cheng, J., Zhu, Y., and Zou, F. Mimicmotion: High-quality human motion video generation with confidence-aware pose guidance. arXiv preprint arXiv:2406.19680 , 2024. · arXiv
  43. Zhu, L., Liu, X., Liu, X., Qian, R., Liu, Z., and Yu, L. Taming diffusion models for audio-driven co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10544-10553, 2023a.
  44. Zhu, L., Yang, D., Zhu, T., Reda, F., Chan, W., Saharia, C., Norouzi, M., and Kemelmacher-Shlizerman, I. Tryondiffusion: A tale of two unets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 4606-4615, 2023b.

Cited by

Sorted by citations

Related

Authors
Tian, L., Hu, S., Wang, Q., Zhang, B., & Bo, L.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2501.10687
arXiv
2501.10687
OpenAlex
W4406699492
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Engineering › Control and Systems Engineering › Human Motion and Animation
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Keywords
co-speech gesture generation, facial expression synthesis, diffusion models, audio-video synchronization, two-step synthesis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review