Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 65Generative Adversarial Networks and Image SynthesisFace recognition and analysisSpeech and Audio Processing

The DOI matches its DataCite record.

EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions

Open access
Chen, Zhiyuan, Cao, Jiajiong, Chen, Zhiquan, Li, Yuming, Ma, Chenguang·2024·arXiv
DOI 10.48550/arxiv.2407.08136arXiv 2407.08136
Open this paper in CoChat — free accountOpen PDF

Abstract

The area of portrait image animation, propelled by audio input, has witnessed notable progress in the generation of lifelike and dynamic portraits. Conventional methods are limited to utilizing either audios or facial key points to drive images into videos, while they can yield satisfactory results, certain issues exist. For instance, methods driven solely by audios can be unstable at times due to the relatively weaker audio signal, while methods driven exclusively by facial key points, although more stable in driving, can result in unnatural outcomes due to the excessive control of key point information. In addressing the previously mentioned challenges, in this paper, we introduce a novel approach which we named EchoMimic. EchoMimic is concurrently trained using both audios and facial landmarks. Through the implementation of a novel training strategy, EchoMimic is capable of generating portrait videos not only by audios and facial landmarks individually, but also by a combination of both audios and selected facial landmarks. EchoMimic has been comprehensively compared with alternative algorithms across various public datasets and our collected dataset, showcasing superior performance in both quantitative and qualitative evaluations. Additional visualization and access to the source code can be located on the EchoMimic project page.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.08136

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 28

  1. Chai, W.; Zheng, D.; Cao, J.; Chen, Z.; Wang, C.; and Ma, C. 2023. SpeedUpNet: A Plug-and-Play Hyper-Network for Accelerating Text-to-Image Diffusion Models. arXiv preprint arXiv:2312.08887. · arXiv
  2. Chung, J. S.; and Zisserman, A. 2017. Out of time: auto- mated lip sync in the wild. In Computer Vision-ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13, 251-263. Springer.
  3. Dhariwal, P.; and Nichol, A. 2021. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34: 8780-8794.
  4. Drobyshev, N.; Chelishev, J.; Khakhulin, T.; Ivakhnenko, A.; Lempitsky, V.; and Zakharov, E. 2022. Megaportraits: One- shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia, 2663- 2671.
  5. Esser, P.; Rombach, R.; and Ommer, B. 2021. Taming trans- formers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12873-12883.
  6. Guo, J.; Zhang, D.; Liu, X.; Zhong, Z.; Zhang, Y.; Wan, P.; and Zhang, D. 2024. Liveportrait: Efficient portrait anima- tion with stitching and retargeting control. arXiv preprint arXiv:2407.03168. · arXiv
  7. Ho, J.; Jain, A.; and Abbeel, P. 2020. Denoising diffusion probabilistic models. Advances in neural information pro- cessing systems, 33: 6840-6851.
  8. Hu, L. 2024. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8153-8163.
  9. Kingma, D. P.; and Welling, M. 2013. Auto-encoding varia- tional bayes. arXiv preprint arXiv:1312.6114. · arXiv
  10. Liu, H.; Han, X.; Jin, C.; Qian, L.; Wei, H.; Lin, Z.; Wang, F.; Dong, H.; Song, Y.; Xu, J.; et al. 2023. Human motion- former: Transferring human motions with vision transform- ers. arXiv preprint arXiv:2302.11306. · arXiv
  11. Luo, S.; Tan, Y.; Huang, L.; Li, J.; and Zhao, H. 2023. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378. · arXiv
  12. Ma, Y.; Liu, H.; Wang, H.; Pan, H.; He, Y.; Yuan, J.; Zeng, A.; Cai, C.; Shum, H.-Y.; Liu, W.; et al. 2024. Follow-Your- Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation. arXiv preprint arXiv:2406.01900. · arXiv
  13. Peebles, W.; and Xie, S. 2023. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, 4195-4205.
  14. Prajwal, K.; Mukhopadhyay, R.; Namboodiri, V. P.; and Jawahar, C. 2020. A lip sync expert is all you need for speech to lip generation in the wild. In Proceedings of the 28th ACM international conference on multimedia, 484- 492.
  15. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from nat- ural language supervision. In International conference on machine learning, 8748-8763. PMLR.
  16. Radford, A.; Kim, J. W.; Xu, T.; Brockman, G.; McLeavey, C.; and Sutskever, I. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, 28492-28518.
  17. PMLR.
  18. Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Om- mer, B. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, 10684- 10695.
  19. Ronneberger, O.; Fischer, P.; and Brox, T. 2015. U-net: Convolutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, 234-241. Springer.
  20. PMLR.
  21. Song, J.; Meng, C.; and Ermon, S. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. · arXiv
  22. Sun, X.; Zhang, L.; Zhu, H.; Zhang, P.; Zhang, B.; Ji, X.; Zhou, K.; Gao, D.; Bo, L.; and Cao, X. 2023. Vividtalk: One-shot audio-driven talking head generation based on 3d hybrid prior. arXiv preprint arXiv:2312.01841. · arXiv
  23. Tian, L.; Wang, Q.; Zhang, B.; and Bo, L. 2024. Emo: Emote portrait alive-generating expressive portrait videos with au- dio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.17485. · arXiv
  24. Wang, C.; Tian, K.; Zhang, J.; Guan, Y.; Luo, F.; Shen, F.; Jiang, Z.; Gu, Q.; Han, X.; and Yang, W. 2024. V-Express: Conditional Dropout for Progressive Training of Portrait Video Generation. arXiv preprint arXiv:2406.02511. · arXiv
  25. Wei, H.; Yang, Z.; and Wang, Z. 2024. Aniportrait: Audio- driven synthesis of photorealistic portrait animation. arXiv preprint arXiv:2403.17694. · arXiv
  26. Xu, M.; Li, H.; Su, Q.; Shang, H.; Zhang, L.; Liu, C.; Wang, J.; Van Gool, L.; Yao, Y.; and Zhu, S. 2024. Hallo: Hier- archical Audio-Driven Visual Synthesis for Portrait Image Animation. arXiv preprint arXiv:2406.08801. · arXiv
  27. Yu, L.; Lezama, J.; Gundavarapu, N. B.; Versari, L.; Sohn, K.; Minnen, D.; Cheng, Y.; Gupta, A.; Gu, X.; Hauptmann, A. G.; et al. 2023. Language Model Beats Diffusion- Tokenizer is Key to Visual Generation. arXiv preprint arXiv:2310.05737. · arXiv
  28. Zhang, W.; Cun, X.; Wang, X.; Zhang, Y.; Shen, X.; Guo, Y.; Shan, Y.; and Wang, F. 2023. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8652-8661.

Cited by

See all 65

Metadata

Authors
Chen, Z., Cao, J., Chen, Z., Li, Y., & Ma, C.
Affiliations
Ant Group (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
39
Issue
3
Pages
2403-2410
ISSN
2159-5399 (ISSN-L)
DOI
10.48550/arxiv.2407.08136
arXiv
2407.08136
OpenAlex
W4409369524
Cited by
65
References
0
Citations by year
2026: 29 · 2025: 36
FWCI
49.12
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions. https://app.cochat.ai/papers/10.48550/arxiv.2407.08136
The paper
Chen, Z., Cao, J., Chen, Z., Li, Y., & Ma, C. (2024). EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditions. arXiv. https://doi.org/10.48550/arxiv.2407.08136

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.