Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation
Open accessAbstract
Talking face generation has historically struggled to produce head movements and natural facial expressions without guidance from additional reference videos. Recent developments in diffusion-based generative models allow for more realistic and stable data synthesis and their performance on image and video generation has surpassed that of other generative models. In this work, we present an autoregressive diffusion model that requires only one identity image and audio sequence to generate a video of a realistic talking human head. Our solution is capable of hallucinating head movements, facial expressions, such as blinks, and preserving a given background. We evaluate our model on two different datasets, achieving state-of-the-art results on both of them.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Stypułkowski et al. developed the first diffusion model for generating videos of talking heads.
“Stypułkowski et al. (2023) have developed the first diffusion model for generating videos of talking heads.”
From Video Diffusion Models: A Survey (2024) · cites this paper as Stypułkowski et al. (2023)
The Introduction says the authors developed a diffusion model that generates talking-head videos and describes it as, to the best of their knowledge, the first diffusion-based solution for talking-face generation. The citing sentence omits that qualification and states the priority claim as an unqualified fact. (LLM, conf=0.82)
Experimental — accuracy not yet measuredDiffused Heads generates talking faces that move their lips in sync with the audio and display realistic facial expressions.
“The generated talking faces move their lips in sync with the audio and display realistic facial expressions.”
From Video Diffusion Models: A Survey (2024) · cites this paper as Stypułkowski et al. (2023)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Generative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384 · see citing works
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712 · see citing works
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365 · see citing works
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00813 · Cited by 6,332 · see citing works
Diffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362 · see citing works
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · see citing works
Denoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507 · see citing works
Towards Principled Methods for Training Generative Adversarial NetworksPreprint
Arjovsky, M., & Bottou, L. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1701.04862 · Cited by 2,373 · see citing works
Improved Denoising Diffusion Probabilistic Models
Nichol, A., & Dhariwal, P. (2021) · arXiv · doi:10.48550/arxiv.2102.09672 · Cited by 1,550 · see citing works
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357 · see citing works
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Patashnik, O., Wu, Z., Shechtman, E., Cohen‐Or, D., & Lischinski, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00209 · Cited by 1,079 · see citing works
MoCoGAN: Decomposing Motion and Content for Video Generation
Tulyakov, S., Liu, M.-Y., Yang, X., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00165 · Cited by 1,076 · see citing works
Synthesizing Obama: learning lip sync from audio
Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073640 · Cited by 1,057 · see citing works
CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2336244 · Cited by 949 · see citing works
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897 · see citing works
Out of Time: Automated Lip Sync in the Wild
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54427-4_19 · Cited by 689 · see citing works
GANimation: Anatomically-Aware Facial Animation from a Single Image
Pumarola, A., Agudo, A., Martinez, A. M., Sanfeliu, A., & Moreno-Noguer, F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_50 · Cited by 621 · see citing works
Looking to listen at the cocktail party
Ephrat, A., Mosseri, I., Lang, O., Dekel, T., Wilson, K., Hassidim, A., Freeman, W. T., & Rubinstein, M. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201357 · Cited by 620 · see citing works
Ho, J., Salimans, T., Gritsenko, A. A., Chan, W., Norouzi, M., & Fleet, D. J. (2022) · neural information processing systems · doi:10.52202/068431-0628 · Cited by 603 · see citing works
First Order Motion Model for Image Animation
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., & Sebe, N. (2020) · arXiv · doi:10.48550/arxiv.2003.00196 · Cited by 585 · see citing works
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537 · see citing works
Audio-driven facial animation by joint end-to-end learning of pose and emotion
Karras, T., Aila, T., Laine, S., Herva, A., & Lehtinen, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073658 · Cited by 435 · see citing works
Quantitative association of vocal-tract and facial behavior
Yehia, H. C., Rubin, P. E., & Vatikiotis‐Bateson, E. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00048-x · Cited by 420 · see citing works
Zhou, Y., Han, X., Shechtman, E., Echevarria, J., Kalogerakis, E., & Li, D. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417774 · Cited by 403 · see citing works
Imagen Video: High Definition Video Generation with Diffusion Models
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02303 · Cited by 399 · see citing works
Make-A-Video: Text-to-Video Generation without Text-Video Data
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.14792 · Cited by 395 · see citing works
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
Guo, Y., Chen, K., Liang, S., Liu, Y.-J., Bao, H., & Zhang, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00573 · Cited by 384 · see citing works
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
Zhou, H., Sun, Y., Wu, W., Loy, C. C., Wang, X., & Liu, Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00416 · Cited by 377 · see citing works
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation
Preechakul, K., Chatthee, N., Wizadwongsa, S., & Suwajanakorn, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01036 · Cited by 338 · see citing works
A deep learning approach for generalized speech animation
Taylor, S. L., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A. G., Hodgins, J. K., & Matthews, I. A. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073699 · Cited by 303 · see citing works
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., & Fleet, D. J. (2022) · arXiv · doi:10.48550/arxiv.2204.03458 · Cited by 301 · see citing works
Realistic Speech-Driven Facial Animation with GANs
Vougioukas, K., Petridis, S., & Pantić, M. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01251-8 · Cited by 299 · see citing works
Self-citationPIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering
Ren, Y., Li, G., Chen, Y., Li, T. H., & Liu, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01350 · Cited by 211 · see citing works
Diffusion Models for Video Prediction and InfillingPreprint
Hoppe, T., Mehrjou, A., Bauer, S., Nielsen, D., & Dittadi, A. (2022) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2206.07696 · Cited by 188 · see citing works
StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN
Yin, F., Zhang, Y., Cun, X., Cao, M., Fan, Y., Wang, X., Bai, Q., Wu, B., Wang, J., & Yang, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19790-1_6 · Cited by 180 · see citing works
Talking-Head Generation with Rhythmic Head Motion
Chen, L., Cui, G., Liu, C., Li, Z., Kou, Z., Xu, Y., & Xu, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7_3 · Cited by 179 · see citing works
EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model
Ji, X., Zhou, H., Wang, K., Wu, Q., Wu, W., Xu, F., & Cao, X. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530745 · Cited by 155 · see citing works
Everybody’s Talkin’: Let Me Talk as You Want
Song, L., Wu, W., Qian, C., He, R., & Loy, C. C. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3146783 · Cited by 142 · see citing works
Photo-real talking head with deep bidirectional LSTM
Fan, B., Wang, L., Soong, F. K., & Xie, L. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178899 · Cited by 126 · see citing works
Flexible Diffusion Modeling of Long Videos
Harvey, W., Naderiparizi, S., Masrani, V., Weilbach, C., & Wood, F. (2022) · neural information processing systems · doi:10.52202/068431-2027 · Cited by 120 · see citing works
Lahiri, A., Kwatra, V., Frueh, C., Lewis, J., & Bregler, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00278 · Cited by 101 · see citing works
Lip movement synthesis from speech based on Hidden Markov Models
Yamamoto, E., Nakamura, S., & Shikano, K. (1998) · Speech Communication · doi:10.1016/s0167-6393(98)00054-5 · Cited by 99 · see citing works
Speech-Driven 3D Facial Animation with Implicit Emotional Awareness: A Deep Learning Approach
Pham, H., Cheung, S., & Pavlović, V. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.287 · Cited by 85 · see citing works
FVD: A new Metric for Video Generation
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., & Gelly, S. (2019) · International Conference on Learning Representations · Cited by 80 · see citing works
A coupled HMM approach to video-realistic speech animation
Xie, L., & Liu, Z. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2006.12.001 · Cited by 70 · see citing works
Pham, H., Wang, Y., & Pavlović, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.07716 · Cited by 29 · see citing works
GENERATION OF MOUTHSHAPES FOR A SYNTHETIC TALKING HEAD
SIMONS, A., & COX, S. (2024) · Institute of Acoustics Speech and Hearing Conference · doi:10.25144/21430 · Cited by 13 · see citing works
Generation of realistic human behaviour
Vougioukas, K. (2022) · doi:10.25560/99407 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020. 2, 3, 4
- AD Simons. Generation of mouthshape for a synthetic talk- ing head. Proc. of the Institute of Acoustics, 1990. 2
- Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang. Styleheat: One-shot high-resolution ed- itable talking face generation via pretrained stylegan. arXiv preprint arXiv:2203.04036, 2022. 2 · arXiv
- Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevar- ria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020. 2, 6, 7
Cited by
Sorted by citations
Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis
Li, J., Zhang, J., Bai, X. P., Zhou, J. W., & Gu, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00696 · Cited by 116 · see citing works
Abbas, F., & Taeihagh, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124260 · Cited by 129 · see citing works
Face Generation and Editing With StyleGAN: A Survey
Melnik, A., Miasayedzenkau, M., Makaravets, D., Pirshtuk, D., Akbulut, E., Holzmann, D., Renusch, T., Reichert, G., & Ritter, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3350004 · Cited by 105 · see citing works
NetDiffus: Network traffic generation by diffusion models through time-series imaging
Sivaroopan, N., Bandara, D., Madarasingha, C., Jourjon, G., Jayasumana, A. P., & Thilakarathna, K. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110616 · Cited by 65 · see citing works
FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models
Aneja, S., Thies, J., Dai, A., & NieBner, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02009 · Cited by 55 · see citing works
AutoSplice: A Text-prompt Manipulated Image Dataset for Media Forensics
Jia, S., Huang, M., Zhou, Z., Ju, Y., Cai, J., & Lyu, S. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00096 · Cited by 44 · see citing works
Liz, H., Keïta, M., Ahmed, A. T., Hadid, A., Huertas‐Tato, J., & Camacho, D. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102103 · Cited by 42 · see citing works
Speech driven video editing via an audio-conditioned diffusion model
Bigioi, D., Basak, S., Stypułkowski, M., Zięba, M., Jordan, H., McDonnell, R., & Corcoran, P. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.104911 · Cited by 42 · see citing works
Self-citationHaque, K. I., & Yumak, Z. (2023) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3577190.3614157 · Cited by 40 · see citing works
DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars
Kirschstein, T., Giebenhain, S., & Nießner, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00524 · Cited by 34 · see citing works
EMOPortraits: Emotion-Enhanced Multimodal One-Shot Head Avatars
Drobyshev, N., Casademunt, A. B., Vougioukas, K., Landgraf, Z., Petridis, S., & Pantić, M. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00812 · Cited by 33 · see citing works
Self-citationA Survey of Cross-Modal Visual Content Generation
Nazarieh, F., Feng, Z., Awais, M., Wang, W., & Kittler, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3351601 · Cited by 29 · see citing works
Style2Talker: High-Resolution Talking Head Generation with Emotion Style and Art Style
Tan, S., Ji, B., & Pan, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28313 · Cited by 30 · see citing works
Deepfake Generation and Detection: A Benchmark and Survey
Pei, G., Zhang, J., Hu, M., Zhang, Z., Wang, C., Wu, Y., Zhai, G., Yang, J. J., & Tao, D. (2026) · ACM Computing Surveys · doi:10.1145/3801962 · Cited by 113 · see citing works
A Review of Deepfake and Its Detection: From Generative Adversarial Networks to Diffusion Models
Liu, B., Liu, B., Zhu, T., & Ding, M. (2025) · International Journal of Intelligent Systems · doi:10.1155/int/9987535 · Cited by 31 · see citing works
Generative technology for human emotion recognition: A scoping review
Ma, F., Yuan, Y., Xie, Y., Ren, H., Liu, I., He, Y., Ren, F., Yu, F. R., & Ni, S. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102753 · Cited by 37 · see citing works
Li, J., Zhang, J., Bai, X. P., Zheng, J., Zhou, J. W., & Gu, L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102456 · Cited by 22 · see citing works
Human Motion Video Generation: A Survey
Xue, H., Luo, X., Hu, Z., Zhang, X., Xiang, X., Dai, Y., Liu, J., Zhang, Z., Li, M., Yang, J. J., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3594034 · Cited by 23 · see citing works
Gotcha: Real-Time Video Deepfake Detection via Challenge-Response
Mittal, G., Hegde, C., & Memon, N. D. (2024) · IEEE European Symposium on Security and Privacy · doi:10.1109/eurosp60621.2024.00009 · Cited by 22 · see citing works
DaGAN++: Depth-Aware Generative Adversarial Network for Talking Head Video Generation
Hong, F.-T., Shen, L., & Xu, D. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3339964 · Cited by 22 · see citing works
MODA: Mapping-Once Audio-driven Portrait Animation with Dual Attentions
Liu, Y., Lin, L., Yu, F., Zhou, C., & Li, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02104 · Cited by 20 · see citing works
AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding
Liu, T., Chen, F., Fan, S., Du, C., Chen, Q., Chen, X., & Yu, K. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681198 · Cited by 21 · see citing works
An Improved Face Image Restoration Method Based on Denoising Diffusion Probabilistic Models
Pang, Y., Mao, J., He, L., Lin, H., & Qiang, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3349423 · Cited by 17 · see citing works
On the Vulnerability of Deepfake Detectors to Attacks Generated by Denoising Diffusion Models
Ivanovska, M., & Štruc, V. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00115 · Cited by 16 · see citing works
FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio
Xu, C., Liu, Y., Xing, J., Wang, W., Sun, M., Dan, J., Huang, T., Li, S., Cheng, Z.-Q., Tai, Y. Y., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00129 · Cited by 19 · see citing works
Liu, W., Wang, C., Cao, Y., Liang, D. S., Li, Y., & Mo, J. (2025) · Electric Power Systems Research · doi:10.1016/j.epsr.2025.111779 · Cited by 15 · see citing works
Wu, S., Haque, K. I., & Yumak, Z. (2024) · Motion in Games · doi:10.1145/3677388.3696320 · Cited by 15 · see citing works
Revisiting Generalizability in Deepfake Detection: Improving Metrics and Stabilizing Transfer
Kamat, S., Agarwal, S., Darrell, T., & Rohrbach, A. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00049 · Cited by 15 · see citing works
ConsistentID: Portrait Generation With Multimodal Fine-Grained Identity Preserving
Huang, J., Dong, X. M., Song, W., Zheng, C., Tang, Z., Zhou, J., Cheng, Y., Chen, L., Li, H., Yan, Y., et al. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3652557 · Cited by 21 · see citing works
Facial Animation Strategies for Improved Emotional Expression in Virtual Reality
Song, H., & Kwon, B. (2024) · Electronics · doi:10.3390/electronics13132601 · Cited by 12 · see citing works
Seeing the Sound: Multilingual Lip Sync for Real-Time Face-to-Face Translation
Oskooei, A. R., Aktaş, M. S., & Keleş, M. (2024) · Computers · doi:10.3390/computers14010007 · Cited by 13 · see citing works
Audio-Semantic Enhanced Pose-Driven Talking Head Generation
Liu, M., Li, D., Li, Y., Song, X., & Nie, L. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3414412 · Cited by 11 · see citing works
SingingHead: A Large-Scale 4D Dataset for Singing Head Animation
Wu, S., Li, Y., Zhang, W., Jia, J., Zhu, Y., Yan, Y., Zhai, G., & Yang, X. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3623560 · Cited by 10 · see citing works
PersonaTalk: Bring Attention to Your Persona in Visual Dubbing
Zhang, L., Liang, S., Ge, Z., & Hu, T. (2024) · SIGGRAPH Asia Conference Papers · doi:10.1145/3680528.3687618 · Cited by 9 · see citing works
VPT: Video portraits transformer for realistic talking face generation
Zhang, Z., Zhang, J., & Mai, W. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107122 · Cited by 9 · see citing works
DiffDub: Person-Generic Visual Dubbing Using Inpainting Renderer with Diffusion Auto-Encoder
Liu, T., Du, C., Fan, S., Chen, F., & Yu, K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446049 · Cited by 8 · see citing works
OSM-Net: One-to-Many One-Shot Talking Head Generation With Spontaneous Head Motions
Liu, J., Wang, X., Fu, X., Chai, Y., Cai, Y., Dai, J., & Han, J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3394850 · Cited by 8 · see citing works
DiffusionAct: Controllable Diffusion Autoencoder for One-shot Face Reenactment
Bounareli, S., Tzelepis, C., Argyriou, V., Patras, I., & Tzimiropoulos, G. (2025) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg61629.2025.11099159 · Cited by 14 · see citing works
Multimodal Fusion for Talking Face Generation Utilizing Speech-Related Facial Action Units
Liu, Z., Liu, X., Chen, S., Liu, J., Wang, L., & Bi, C. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3672565 · Cited by 7 · see citing works
Audio-Driven Talking Face Generation with Stabilized Synchronization Loss
Yaman, D., Eyiokur, F. İ., Bärmann, L., Ekenel, H. K., & Waibel, A. H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72655-2_24 · Cited by 7 · see citing works
Multimodal Feature-Guided Audio-Driven Emotional Talking Face Generation
Wang, X., Huo, Y., Liu, Y., Guo, X., Yan, F., & Zhao, G. (2025) · Electronics · doi:10.3390/electronics14132684 · Cited by 7 · see citing works
Decoupled Two-Stage Talking Head Generation via Gaussian-Landmark-Based Neural Radiance Fields
Ma, B., Cao, Y., & Zhang, L. (2025) · Computational Visual Media · doi:10.26599/cvm.2025.9450482 · Cited by 7 · see citing works
ReSyncer: Rewiring Style-Based Generator for Unified Audio-Visually Synced Facial Performer
Guan, J., Xu, Z., Zhou, H., Wang, K., He, S., Zhang, Z., Liang, B., Feng, H., Ding, E., Liu, J., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72940-9_20 · Cited by 7 · see citing works
Self-citationINFP: Audio-Driven Interactive Head Generation in Dyadic Conversations
Zhu, Y., Zhang, L., Rong, Z., Hu, T., Liang, S., & Ge, Z. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00997 · Cited by 6 · see citing works
Ten Natural Language Processing Tasks with Generative Artificial Intelligence
Golec, J., & Hachaj, T. (2025) · Applied Sciences · doi:10.3390/app15169057 · Cited by 13 · see citing works
Haque, K. I., Wu, S., & Yumak, Z. (2025) · Computers & Graphics · doi:10.1016/j.cag.2025.104358 · Cited by 6 · see citing works
Enhanced single-pixel imaging base on posterior mean prediction using a score-based diffusion model
Tang, Z., Yan, Q., Li, Y., & Yan, J. (2025) · Optics Communications · doi:10.1016/j.optcom.2025.131633 · Cited by 6 · see citing works
LatentColorization: Latent Diffusion-Based Speaker Video Colorization
Ward, R., Bigioi, D., Basak, S., Breslin, J. G., & Corcoran, P. (2024) · IEEE Access · doi:10.1109/access.2024.3406249 · Cited by 6 · see citing works
FD2Talk: Towards Generalized Talking Head Generation with Facial Decoupled Diffusion Model
Yao, Z., Cheng, X., & Huang, Z. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681238 · Cited by 5 · see citing works
Liu, X., Liu, Z., & Bi, C. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889538 · Cited by 5 · see citing works
Imai, Y., SENDA, T., & Kajiwara, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3525714 · Cited by 4 · see citing works
Keypoints and Action Units Jointly Drive Talking Head Generation for Video Conferencing
Shi, W., Xue, Z., & Wen, Y. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3560369 · Cited by 4 · see citing works
Rakesh, V. K., Mazumdar, S., Pal, S., Das, A. K., & Samanta, T. (2025) · The Visual Computer · doi:10.1007/s00371-025-04232-w · Cited by 4 · see citing works
Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation
Tan, J., Cheng, X., Xiong, L., Zhu, L., Li, X., Wu, X., Gong, K., Li, M., & Cai, Y. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10688218 · Cited by 4 · see citing works
GAMMA-FACE: GAussian Mixture Models Amend Diffusion Models for Bias Mitigation in Face Images
Pal, B., Kannan, A., Kathirvel, R. P., O’Toole, A. J., & Chellappa, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72855-6_27 · Cited by 4 · see citing works
Zhang, C., Wang, C., Zhang, J., Xu, H., Song, G., Xie, Y., Luo, L., Tian, Y., Feng, J., & Guo, X. (2025) · Computational Visual Media · doi:10.26599/cvm.2025.9450491 · Cited by 4 · see citing works
Simultaneous control of head pose and expressions in 3D facial keypoint-based GAN
Hatakeyama, T., Furuta, R., & Sato, Y. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18449-9 · Cited by 4 · see citing works
Gao, Y., Li, J., Chu, L., & Lü, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01824 · Cited by 4 · see citing works
Style Transfer for 2D Talking Head Generation
Pham, T.-T., Do, T., Le, N., Le, N., Nguyen, H. T., Tjiputra, E., Tran, Q., & Nguyen, A. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00745 · Cited by 4 · see citing works
“Wild West” of Evaluating Speech‐Driven 3D Facial Animation Synthesis: A Benchmark Study
Haque, K. I., Pavlou, A., & Yumak, Z. (2025) · Computer Graphics Forum · doi:10.1111/cgf.70073 · Cited by 3 · see citing works
Weidmann, B., & Xu, Y. (2025) · Journal of Intelligence · doi:10.3390/jintelligence13090116 · Cited by 7 · see citing works
GaPTalk: precision-controlled 3D Gaussian rendering for personalized talking-head synthesis
Huang, X., Li, X., Tan, S., Chen, W., & Li, G. (2025) · The Visual Computer · doi:10.1007/s00371-025-04195-y · Cited by 3 · see citing works
Revisiting Face Forgery Detection: From Facial Representation to Forgery Detection
Guo, Z., Liu, Y., Zhang, J., Zheng, H., & Shan, S. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3675018 · Cited by 4 · see citing works
Pal, B., Roy, A., Kathirvel, R. P., O’Toole, A. J., & Chellappa, R. (2024) · IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb62174.2024.10744513 · Cited by 4 · see citing works
ExDDV: A New Dataset for Explainable Deepfake Detection in Video
Hondru, V., Hogea, E., Onchiş, D., & Ionescu, R. T. (2026) · 2026 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv61042.2026.00416 · Cited by 3 · see citing works
Detection of Diffusion Model-Generated Faces by Assessing Smoothness and Noise Tolerance
Liu, B., Liu, B., Ding, M., & Zhu, T. (2024) · IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB) · doi:10.1109/bmsb62888.2024.10608232 · Cited by 3 · see citing works
Autoregressive GAN for Semantic Unconditional Head Motion Generation
Airale, L., Alameda-Pineda, X., Lathuilière, S., & Vaufreydaz, D. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3635154 · Cited by 3 · see citing works
FacialTalk: Audio-driven high-fidelity facial portrait generation using 3D facial prior
Yang, D., Liu, Y., Yang, Q., & Li, R. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111994 · Cited by 2 · see citing works
Alsafadi, F., Akins, A., & Wu, X. (2026) · Energy and AI · doi:10.1016/j.egyai.2026.100702 · Cited by 2 · see citing works
AI killed the Video Star. Audio-Driven Diffusion Model for Expressive Talking Head Generation
Chopin, B., Dhamija, T., Balaji, P., Wang, Y., & Dantcheva, A. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-026-02766-7 · Cited by 2 · see citing works
Samson, G. (2024) · International Journal of Electronics and Telecommunications · doi:10.24425/ijet.2024.149550 · Cited by 4 · see citing works
Talking Face Generation via Face Mesh - Controllability without Reference Videos
Köksal, A., Xu, Q., & Lim, J. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00246 · Cited by 2 · see citing works
Multi-Level Feature Dynamic Fusion Neural Radiance Fields for Audio-Driven Talking Head Generation
Song, W., Liu, Q., Liu, Y., Zhang, P., & Cao, J. (2025) · Applied Sciences · doi:10.3390/app15010479 · Cited by 2 · see citing works
Adaptive diffusion landmark dynamic rendering for realistic talking face video generation
Tang, Y., Yazhi, L., Xiong, L., & Li, W. (2025) · The Visual Computer · doi:10.1007/s00371-025-03907-8 · Cited by 2 · see citing works
Oskooei, A. R., Caglar, E., Şahin, I., Kayabay, A., & Aktaş, M. S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97606-3_15 · Cited by 2 · see citing works
Lip-Synchronized 3D Facial Animation Using Audio-Driven Graph Convolutional Autoencoder
Bozhilov, I., Tonchev, K., Neshov, N. N., & Manolova, A. H. (2023) · IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS) · doi:10.1109/idaacs58523.2023.10348935 · Cited by 2 · see citing works
Video Diffusion Models: A Survey
Melnik, A., Ljubljanac, M., Lu, C., Yan, Q., Ren, W., & Ritter, H. (2024) · arXiv · doi:10.48550/arxiv.2405.03150 · Cited by 4 · see citing works
UniFace++: Revisiting a Unified Framework for Face Reenactment and Swapping via 3D Priors
Xu, C., Qian, Y., Zhu, S., Sun, B., Zhao, J. M., Liu, Y., & Li, X. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02395-6 · Cited by 1 · see citing works
Boosting Pose Estimators via Cross-Representation Distillation
Liu, K., Yang, Z., Zhang, J., Wang, J., Wang, S., Yuan, C., & Guo, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-91575-8_21 · Cited by 1 · see citing works
Dai, Q., Feng, H., Cui, W., Cai, X., Zheng, Y., & Zeng, M. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733322 · Cited by 1 · see citing works
Geldhauser, C., Liljegren, J., & Nordqvist, P. (2025) · Electronics · doi:10.3390/electronics14173487 · Cited by 1 · see citing works
Cao, Z., Zhang, J., Chen, L., Li, T., Wu, H., & Zhang, Y. (2025) · Current Medical Imaging Formerly Current Medical Imaging Reviews · doi:10.2174/0115734056401610250827114351 · Cited by 1 · see citing works
Prompt-to-Animation: Generating Cognitively-Grounded Facial Expressions with LLMs
Durupınar, F., & Normoyle, A. (2025) · ACM SIGGRAPH Conference on Motion, Interaction and Games (MIG) · doi:10.1145/3769047.3769055 · Cited by 1 · see citing works
Generative AI for Video Translation: A Scalable Architecture for Multilingual Video Conferencing
Oskooei, A. R., Caglar, E., Şahin, I., Kayabay, A., & Aktaş, M. S. (2025) · Applied Sciences · doi:10.3390/app152312691 · Cited by 1 · see citing works
LSF-Animation: Label-Free Speech-Driven Facial Animation via Implicit Feature Representation
Lu, X., Zhuang, C., Jin, C., Lu, Z., Wang, Y., Liu, W., & Xiao, J. (2025) · SIGGRAPH Asia Conference Papers · doi:10.1145/3757377.3763887 · Cited by 1 · see citing works
A Survey of Talking Head Synthesis Techniques: Portrait Generation, Driving Mechanisms, and Editing
Meng, M., Zhao, Y., Zhang, B., Zhu, Y., Shi, W., Wen, Z., & Fan, Z. (2025) · ACM Computing Surveys · doi:10.1145/3785656 · Cited by 4 · see citing works
Splat-Portrait: Generalizing Talking Heads with Gaussian Splatting
Shi, T., de Almeida, M., Ivanova, D., Pugeault, N., & Henderson, P. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-6950-2_35 · Cited by 1 · see citing works
DH-FaceVid-1K: A Large-Scale High-Quality Dataset for Face Video Generation
Di, D., He, F., Sun, W., Ma, Y., Li, H., Chen, W., Fan, L., Su, T., & Yang, X. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01127 · Cited by 1 · see citing works
Oluwafemi, E. P. O., Tunde, B. D., Hoque, M., Briggs, D., Rahman, M. M., & Khalifa, F. (2025) · IEEE International Conference on Computing and Machine Intelligence (ICMI) · doi:10.1109/icmi65310.2025.11141284 · Cited by 1 · see citing works
High-Fidelity and Lip-Synced Talking Face Synthesis via Landmark-Based Diffusion Model
Zhong, W., Lin, J., Chen, P., Gao, F., Lin, L., & Li, G. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3676251 · Cited by 1 · see citing works
StableTalk: Advancing Audio-to-Talking Face Generation with Stable Diffusion and Vision Transformer
Nazarieh, F., Kittler, J., Awais, M., Kanojia, D., & Feng, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78172-8_18
DFNeRF: Disentangled Facial Neural Radiance Fields for Text-based Editing of Free-view Talking Head
Chen, B., Li, X., Wang, X., Zhang, Q., & Wang, H. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889463
SwapTalk: Audio-Driven Talking Face Generation with One-Shot Customization in Latent Space
Zhang, Z., Qin, H., Huang, J., Cheng, J.-K., Li, Y., Hui, L., Duan, Y., & Ma, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889186
High-fidelity Talking Face Synthesis Guided by Landmark
Liu, Z., Zhang, R., Li, B. H., & Liu, B. (2024) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic64248.2024.10913301
Blume, F., Qu, R., Bideau, P., Maier, M., Rahman, R. A., & Hellwich, O. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-85187-2_2
Xiao, Z., Liu, X., Guo, J., He, J., Hong, R., & Wang, M. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3742790
AnchorTalk: High-Fidelity Upper-Body Talking Human Generation From Speech
Cai, Y., Qiao, P., & Li, D. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733276
FREAK: Frequency-modulated High-fidelity and Real-time Audio-driven Talking Portrait Synthesis
Ni, Z., Fu, A., & Zhou, Y. (2025) · International Conference on Multimedia Retrieval · doi:10.1145/3731715.3733344
- Authors
- Stypułkowski, M., Vougioukas, K., He, S., Zięba, M., Petridis, S., & Pantic, M.
- Affiliations
- University of Wrocław (PL)
- Imperial College London (GB)
- Wrocław University of Science and Technology (PL)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 5089-5098
- ISSN
- 2472-6737 (ISSN-L)
- arXiv
- 2301.03396
- OpenAlex
- W4394597549
- Cited by
- 140
- References
- 51
- Citations by year
- 2026: 32 · 2025: 58 · 2024: 39 · 2023: 11
- FWCI
- 26.63
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Winter Conference on Applications of Computer Vision (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- talking head generation, diffusion models, facial expression synthesis, head movement, audio-driven video generation, background preservation, GaN comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
