The DOI matches its Crossref record; 48 of the 50 references it cites were found in a registry or an index.
SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation
Open accessAbstract
Generating talking head videos through a face image and a piece of speech audio still contains many challenges. i.e., unnatural head movement, distorted expression, and identity modification. We argue that these issues are mainly caused by learning from the coupled 2D motion fields. On the other hand, explicitly using 3D information also suffers problems of stiff expression and incoherent video. We present SadTalker, which generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation. To learn the realistic motion coefficients, we explicitly model the connections between audio and different types of motion coefficients individually. Precisely, we present ExpNet to learn the accurate facial expression from audio by distilling both coefficients and 3D-rendered faces. As for the head pose, we design PoseVAE via a conditional VAE to synthesize head motion in different styles. Finally, the generated 3D motion coefficients are mapped to the unsupervised 3D keypoints space of the proposed face render to synthesize the final video. We conducted extensive experiments to demonstrate the superiority of our method in terms of motion and video quality.11The code and demo videos are available at https://sadtalker.github.io.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“SadTalker [44] first extracts 3DMM coefficients from audio and then renders the face in a 3D-aware manner.”
From MoCha: Towards Movie-Grade Talking Character Synthesis (2025) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00482 · Cited by 7,781
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. R. (1999) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/311535.311556 · Cited by 4,695
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 1×
“Both visible and invisible video watermarks will be inserted into the produced video for generated content identification similar to Dall-E [30] and Imagen [32].”
MethodVoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210
Tutorial on Variational Autoencoders
Doersch, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05908 · Cited by 1,470
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897
Accurate 3D Face Reconstruction With Weakly-Supervised Learning: From Single Image to Image Set
Deng, Y., Yang, J., Xu, S., Chen, D., Jia, Y., & Tong, X. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00038 · Cited by 776
Kim, H., Garrido, P., Tewari, A., Xu, W., Thies, J., Nießner, M., Pérez, P., Richardt, C., Zollhöfer, M., & Theobalt, C. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201283 · Cited by 702
Fine-Grained Head Pose Estimation Without Keypoints
Ruiz, N., Chong, E., & Rehg, J. M. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00281 · Cited by 677
A No-Reference Image Blur Metric Based on the Cumulative Probability of Blur Detection (CPBD)
Narvekar, N., & Karam, L. J. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2131660 · Cited by 595
First Order Motion Model for Image Animation
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., & Sebe, N. (2020) · arXiv · doi:10.48550/arxiv.2003.00196 · Cited by 585
Towards Real-World Blind Face Restoration with Generative Facial Prior
Wang, X., Li, Y., Zhang, H., & Shan, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00905 · Cited by 564
Self-citationGANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Bynagari, N. B. (2019) · Asian Journal of Applied Science and Engineering · doi:10.18034/ajase.v8i1.9 · Cited by 514
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
Wang, T.-C., Mallya, A., & Liu, M.-Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00991 · Cited by 496
Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss
Chen, L., Maddox, R. K., Duan, Z., & Xu, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00802 · Cited by 437
Zhou, Y., Han, X., Shechtman, E., Echevarria, J., Kalogerakis, E., & Li, D. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417774 · Cited by 431
Neural Voice Puppetry: Audio-Driven Facial Reenactment
Thies, J., Elgharib, M. A., Tewari, A., Theobalt, C., & Nießner, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_42 · Cited by 411
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
Zhou, H., Sun, Y., Wu, W., Loy, C. C., Wang, X., & Liu, Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00416 · Cited by 391
Self-citationAD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
Guo, Y., Chen, K., Liang, S., Liu, Y.-J., Bao, H., & Zhang, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00573 · Cited by 384
Self-citationFlow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset
Zhang, Z., Li, L., Ding, Y., & Fan, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00366 · Cited by 368
Learning Individual Styles of Conversational Gesture
Ginosar, S., Bar, A., Kohavi, G., Chan, C., Owens, A., & Malik, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00361 · Cited by 362
FaceFormer: Speech-Driven 3D Facial Animation with Transformers
Fan, Y., Lin, Z., Saito, J., Wang, W., & Komura, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01821 · Cited by 295
Few-shot Video-to-Video SynthesisPreprint
Wang, T.-C., Liu, M.-Y., Tao, A., Liu, G., Kautz, J., & Catanzaro, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12713 · Cited by 269
Motion Representations for Articulated Animation
Siarohin, A., Woodford, O. J., Ren, J., Chai, M., & Tulyakov, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01344 · Cited by 260
CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior
Xing, J., Xia, M., Zhang, Y., Cun, X., Wang, J., & Wong, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01229 · Cited by 238
Self-citationAudio-Driven Emotional Video Portraits
Ji, X., Zhou, H., Wang, K., Wu, W., Loy, C. C., Cao, X., & Xu, F. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01386 · Cited by 235
Thin-Plate Spline Motion Model for Image Animation
Zhao, J. M., & Zhang, H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00364 · Cited by 232
PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering
Ren, Y., Li, G., Chen, Y., Li, T. H., & Liu, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01350 · Cited by 224
HeadNeRF: A Realtime NeRF-based Parametric Head Model
Hong, Y., Peng, B., Xiao, H., Liu, L., & Zhang, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01973 · Cited by 207
Depth-Aware Generative Adversarial Network for Talking Head Video Generation
Hong, F.-T., Zhang, L., Shen, L., & Xu, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00339 · Cited by 190
Lu, Y., Chai, J., & Cao, X. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480484 · Cited by 188
StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN
Yin, F., Zhang, Y., Cun, X., Cao, M., Fan, Y., Wang, X., Bai, Q., Wu, B., Wang, J., & Yang, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19790-1_6 · Cited by 180
Self-citationAudio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion
Wang, S., Li, L., Ding, Y., Fan, C., & Yu, X. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/152 · Cited by 169
EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model
Ji, X., Zhou, H., Wang, K., Wu, Q., Wu, W., Xu, F., & Cao, X. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530745 · Cited by 155
Towards Automatic Face-to-Face Translation
R, P. K., Mukhopadhyay, R., Philip, J., Jha, A., Namboodiri, V. P., & Jawahar, C. V. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351066 · Cited by 151
FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning
Zhang, C., Zhao, Y., Huang, Y., Zeng, M., Ni, S., Budagavi, M., & Guo, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00384 · Cited by 140
HeadGAN: One-shot Neural Head Synthesis and Editing
Doukas, M. C., Zafeiriou, S., & Sharmanska, V. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01413 · Cited by 117
One-Shot Talking Face Generation from Single-Speaker Audio-Visual Correlation Learning
Wang, S., Li, L., Ding, Y., & Yu, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20154 · Cited by 115
VideoReTalking: Audio-based Lip Synchronization for Talking Head Video Editing In the Wild
Cheng, K., Cun, X., Zhang, Y., Xia, M., Yin, F., Zhu, M., Wang, X., Wang, J., & Wang, N. (2022) · SIGGRAPH Asia Conference Papers · doi:10.1145/3550469.3555399 · Cited by 112
Self-citationPhotorealistic Audio-driven Video Portraits
Wen, X. L., Wang, M., Richardt, C., Chen, Z.-Y., & Hu, S. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3023573 · Cited by 81
DPE: Disentanglement of Pose and Expression for General Video Portrait Editing
Pang, Y., Zhang, Y., Quan, W., Fan, Y., Cun, X., Shan, Y., & Dong-ming Yan, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00049 · Cited by 56
Self-citationLatent Image Animator: Learning to Animate Images via Latent Space NavigationPreprint
Wang, Y., Yang, D., Brémond, F., & Dantcheva, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.09043 · Cited by 52
Implicit Warping for Animation with Image SetsPreprint
Mallya, A., Wang, T.-C., & Liu, M.-Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.01794 · Cited by 12
Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from VideosPreprint
Filntisis, P. P., Retsinas, G., Papantoniou, F. P., Katsamanis, A., Roussos, A., & Maragos, P. A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.11094 · Cited by 7
Research works printed but not linked: 3
- kingma, 2014, A method for stochastic optimization, ArXiv Preprint
- seitzer, 2020, pytorch-fid FID Score for PyTorch
- li, 0, Bailando: 3d dance generation via actor-critic gpt with choreographic memory, CVPR
Cited by
Sorted by citations
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
Li, Z., Cao, M., Wang, X., Qi, Z., Cheng, M., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00825 · Cited by 233
Self-citationEvalCrafter: Benchmarking and Evaluating Large Video Generation Models
Liu, Y., Cun, X., Liu, X., Wang, X., Zhang, Y., Chen, H., Liu, Y., Zeng, T., Chan, R., & Shan, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02090 · Cited by 219
Self-citationTian, L., Wang, Q. J., Zhang, B., & Bo, L. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73010-8_15 · Cited by 211
Tian, L., Wang, Q., Zhang, B., & Bo, L. (2024) · arXiv · doi:10.48550/arxiv.2402.17485 · Cited by 92
Sun, Z., Lv, T. S., Ye, S., Lin, M. G., Sheng, J., Wen, Y., Yu, M., & Liu, Y. (2024) · ACM Transactions on Graphics · doi:10.1145/3658221 · Cited by 98
Metadata
- Authors
- Zhang, W., Cun, X., Wang, X., Zhang, Y., Shen, X., Guo, Y., Shan, Y., & Wang, F.
- Affiliations
- Xi'an Jiaotong University (CN)
- National Key Laboratory of Human-Machine Hybrid Augmented Intelligence
- Ant Group (China) (CN)
- Tencent (China) (CN)
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 8652-8661
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386072021
- Cited by
- 336
- References
- 50
- Citations by year
- 2026: 86 · 2025: 131 · 2024: 105 · 2023: 14
- FWCI
- 35.93
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- National Key Research and Development Program of China (2022YFB3303800)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- talking head generation, audio-driven face animation, facial expression synthesis, stylized animation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
