The DOI matches its DataCite record; 27 of the 29 references it cites were found in a registry or an index, and 1 are not checked yet.
EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions
Open accessAbstract
In this work, we tackle the challenge of enhancing the realism and expressiveness in talking head video generation by focusing on the dynamic and nuanced relationship between audio cues and facial movements. We identify the limitations of traditional techniques that often fail to capture the full spectrum of human expressions and the uniqueness of individual facial styles. To address these issues, we propose EMO, a novel framework that utilizes a direct audio-to-video synthesis approach, bypassing the need for intermediate 3D models or facial landmarks. Our method ensures seamless frame transitions and consistent identity preservation throughout the video, resulting in highly expressive and lifelike animations. Experimental results demonsrate that EMO is able to produce not only convincing speaking videos but also singing videos in various styles, significantly outperforming existing state-of-the-art methodologies in terms of expressiveness and realism.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On the other hand, recent speech-driven video generation methods, such as Loopy, Hallo3, and EMO [18, 24, 33-35, 40], predominantly focus on synthesizing talking-head videos confined to facial regions. These approaches neglect essential fullbody movements and multi-character interactions critical for expressive storytelling, thus significantly limiting their applicability in realistic and inter”
From MoCha: Towards Movie-Grade Talking Character Synthesis (2025) · cites this paper as [33]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unlike prior works such as EMO [33, 35], SONIC [17], Echomimicv2 [24], Loopy [18], and Hallo3 [40], which rely heavily on external control signals (e.g., reference images, skeletons, keypoints), MoCha is trained directly on text and speech without any auxiliary conditioning. This simplifies the model architecture and improves motion diversity and generalization.”
From MoCha: Towards Movie-Grade Talking Character Synthesis (2025) · cites this paper as [33]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It is well known that the features in the U-Net's selfattention layers can reflect the consistency of image appearance.”
From DiffSim: Taming Diffusion Models for Evaluating Visual Similarity (2024) · cites this paper as [47]
The cited source says what the claim attributes to it.
In the citing paper
“Recently works, such as EMO [34] and Hallo [40], generate audio-driven portrait videos end-to-end using diffusion models, which eliminate intermediate facial representations and learn natural motion from data [10, 18, 34, 37, 40].”
From MoCha: Towards Movie-Grade Talking Character Synthesis (2025) · cites this paper as [34]
The cited source says what the claim attributes to it.
In the citing paper
“On the other hand, recent speech-driven video generation methods, such as Loopy, Hallo3, and EMO [18, 24, 33-35, 40], predominantly focus on synthesizing talking-head videos confined to facial regions. These approaches neglect essential fullbody movements and multi-character interactions critical for expressive storytelling, thus significantly limiting their applicability in realistic and inter”
From MoCha: Towards Movie-Grade Talking Character Synthesis (2025) · cites this paper as [34]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Scalable Diffusion Models with Transformers
Peebles, W., & Xie, S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00387 · Cited by 2,284
wav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
Magic3D: High-Resolution Text-to-3D Content Creation
Lin, C.-H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.-Y., & Lin, T.-Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00037 · Cited by 1,100
MediaPipe: A framework for building perception pipelines
Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M. G., Lee, J., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.08172 · Cited by 1,043
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022
Imagic: Text-Based Real Image Editing with Diffusion Models
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., & Irani, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00582 · Cited by 991
CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2336244 · Cited by 954
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897
Research works printed but not linked: 16
- [9]9. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems 30 (2017)
- [8]8. Guo, Y., Yang, C., Rao, A., Wang, Y., Qiao, Y., Lin, D., Dai, B.: Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 (2023) · arXiv
- [10]10. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in neural information processing systems 33 , 6840-6851 (2020)
- [12]12. Hu, L., Gao, X., Zhang, P., Sun, K., Zhang, B., Bo, L.: Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117 (2023) · arXiv
- [14]14. Kingma, D.P., Welling, M.: Auto-Encoding Variational Bayes. In: 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 1416, 2014, Conference Track Proceedings (2014)
- [18]18. Ma, X., Wang, Y., Jia, G., Chen, X., Liu, Z., Li, Y.F., Chen, C., Qiao, Y.: Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048 (2024) · arXiv
- [19]19. Ma, Y., Zhang, S., Wang, J., Wang, X., Zhang, Y., Deng, Z.: Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiv preprint arXiv:2312.09767 (2023) · arXiv
- [22]22. Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 (2022) · arXiv
- [27]27. Schneider, S., Baevski, A., Collobert, R., Auli, M.: wav2vec: Unsupervised pretraining for speech recognition. pp. 3465-3469 (09 2019). https://doi.org/10. 21437/Interspeech.2019-1873
- [26]26. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015. pp. 234-241. Springer International Publishing, Cham (2015)
- [31]31. Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: International conference on machine learning. pp. 2256-2265. PMLR (2015)
- [33]33. Stypułkowski, M., Vougioukas, K., He, S., Zięba, M., Petridis, S., Pantic, M.: Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation. In: https://arxiv.org/abs/2301.03396 (2023) · arXiv
- [32]32. Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: International Conference on Learning Representations (2021), https://openreview.net/forum? id=St1giarCHLP
- [34]34. Sun, X., Zhang, L., Zhu, H., Zhang, P., Zhang, B., Ji, X., Zhou, K., Gao, D., Bo, L., Cao, X.: Vividtalk: One-shot audio-driven talking head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.01841 (2023) · arXiv
- [38]38. Ye, Z., Zhong, T., Ren, Y., Yang, J., Li, W., Huang, J., Jiang, Z., He, J., Huang, R., Liu, J., et al.: Real3d-portrait: One-shot realistic 3d talking portrait synthesis. arXiv preprint arXiv:2401.08503 (2024) · arXiv
- [42]42. Zhou, Y., Han, X., Shechtman, E., Echevarria, J., Kalogerakis, E., Li, D.: Makeittalk: Speaker-aware talking-head animation. ACM Transactions on Graphics 39 (6) (2020)
Cited by
Sorted by citations
A Survey on Video Diffusion Models
Xing, Z., Feng, Q., Chen, H., Dai, Q., Hu, H., Xu, H., Wu, Z., & Jiang, Y. (2024) · ACM Computing Surveys · doi:10.1145/3696415 · Cited by 228
EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions
Chen, Z., Cao, J., Chen, Z., Li, Y., & Ma, C. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i3.32241 · Cited by 173
SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models
Guo, Y., Yang, C., Rao, A., Agrawala, M., Lin, D., & Dai, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72946-1_19 · Cited by 149
Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation
Ma, Y., Liu, H., Wang, H., Pan, H., He, Y., Yuan, J., Zeng, A., Cai, C., Shum, H., Liu, W., et al. (2024) · SIGGRAPH Asia Conference Papers · doi:10.1145/3680528.3687587 · Cited by 118
Wang, T., Wu, T., Liu, H., Brown, C., & Chen, Y. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3701198 · Cited by 39
Metadata
- Authors
- Tian, L., Wang, Q., Zhang, B., & Bo, L.
- Affiliations
- Alibaba Group (China) (CN)
- Corresponding author
- Tian, Linrui
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 244-260
- ISSN
- 0302-9743 (ISSN-L)
- arXiv
- 2402.17485
- OpenAlex
- W4404199654
- Cited by
- 92
- References
- 29
- Citations by year
- 2026: 34 · 2025: 48 · 2024: 10
- FWCI
- 19.05
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- facial expressions, audio-visual alignment, identity preservation, weak conditions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
