Audio-Visual Face Reenactment
Open accessAbstract
This work proposes a novel method to generate realistic talking head videos using audio and visual streams. We animate a source image by transferring head motion from a driving video using a dense motion field generated using learnable keypoints. We improve the quality of lip sync using audio as an additional input, helping the network to attend to the mouth region. We use additional priors using face segmentation and face mesh to improve the structure of the reconstructed faces. Finally, we improve the visual quality of the generations by incorporating a carefully designed identity-aware generator module. The identity-aware generator takes the source image and the warped motion features as input to generate a high-quality output with fine-grained details. Our method produces state-of-the-art results and generalizes well to unseen faces, languages, and voices. We comprehensively evaluate our approach using multiple metrics and outperforming the current techniques both qualitative and quantitatively. Our work opens up several applications, including enabling low bandwidth video calls. We release a demo video and additional information at http://cvit.iiit.ac.in/research/projects/cvit-projects/avfr.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several face reenactment strategies including EAMM, AVFR-GAN, DiffTalk, ADNeRF and ATVGnet are not well suited for zero-shot lip-synced generation of unseen speakers.
“We investigated several face reenactment strategies including EAMM [26], AVFR-GAN [2], DiffTalk [54], ADNeRF [20] and ATVGnet [10] and concluded that these methods are not well suited for zero-shot lip-synced generation of unseen speakers.”
From AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset (2024) · cites this paper as [2]
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623 · see citing works
Instance Normalization: The Missing Ingredient for Fast StylizationPreprint
Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2016) · arXiv.org · doi:10.48550/arxiv.1607.08022 · Cited by 4,239 · see citing works
Spectral Normalization for Generative Adversarial Networks
Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018) · arXiv · doi:10.48550/arxiv.1802.05957 · Cited by 3,438 · see citing works
Semantic Image Synthesis With Spatially-Adaptive Normalization
Park, T., Liu, M.-Y., Wang, T.-C., & Zhu, J.-Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00244 · Cited by 2,974 · see citing works
Generative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494 · see citing works
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · see citing works
VoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210 · see citing works
Overview of the Versatile Video Coding (VVC) Standard and its Applications
Bross, B., Wang, Y.-K., Ye, Y., Liu, S., Chen, J., Sullivan, G. J., & Ohm, J.-R. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3101953 · Cited by 1,832 · see citing works
Bulat, A., & Tzimiropoulos, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.116 · Cited by 1,464 · see citing works
Bulat, A., & Tzimiropoulos, G. (2017) · Nottingham ePrints (University of Nottingham) · Cited by 1,277 · see citing works
MediaPipe: A framework for building perception pipelines
Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M. G., Lee, J., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.08172 · Cited by 1,043 · see citing works
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897 · see citing works
Self-citation(2016) · Cited by 844 · see citing works
Few-Shot Adversarial Learning of Realistic Neural Talking Head Models
Zakharov, E., Shysheya, A., Burkov, E., & Lempitsky, V. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00955 · Cited by 687 · see citing works
First Order Motion Model for Image Animation
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., & Sebe, N. (2020) · arXiv · doi:10.48550/arxiv.2003.00196 · Cited by 585 · see citing works
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537 · see citing works
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
Wang, T.-C., Mallya, A., & Liu, M.-Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00991 · Cited by 468 · see citing works
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation
Zhou, H., Liu, Y., Liu, Z., Luo, P., & Wang, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019299 · Cited by 416 · see citing works
Zhou, Y., Han, X., Shechtman, E., Echevarria, J., Kalogerakis, E., & Li, D. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417774 · Cited by 403 · see citing works
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
Zhou, H., Sun, Y., Wu, W., Loy, C. C., Wang, X., & Liu, Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00416 · Cited by 377 · see citing works
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset
Zhang, Z., Li, L., Ding, Y., & Fan, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00366 · Cited by 368 · see citing works
Zoom Exhaustion & Fatigue Scale
Fauville, G., Luo, M., Queiroz, A. C. M., Bailenson, J. N., & Hancock, J. T. (2021) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2021.100119 · Cited by 311 · see citing works
Motion Representations for Articulated Animation
Siarohin, A., Woodford, O. J., Ren, J., Chai, M., & Tulyakov, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01344 · Cited by 260 · see citing works
Audio-Driven Emotional Video Portraits
Ji, X., Zhou, H., Wang, K., Wu, W., Loy, C. C., Cao, X., & Xu, F. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01386 · Cited by 223 · see citing works
PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering
Ren, Y., Li, G., Chen, Y., Li, T. H., & Liu, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01350 · Cited by 211 · see citing works
Talking Face Generation by Conditional Recurrent Adversarial Network
Song, Y., Zhu, J., Li, D., Wang, A. Y.-D., & Qi, H. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/129 · Cited by 192 · see citing works
Depth-Aware Generative Adversarial Network for Talking Head Video Generation
Hong, F.-T., Zhang, L., Shen, L., & Xu, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00339 · Cited by 190 · see citing works
Lu, Y., Chai, J., & Cao, X. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480484 · Cited by 174 · see citing works
Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion
Wang, S., Li, L., Ding, Y., Fan, C., & Yu, X. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/152 · Cited by 169 · see citing works
Towards Automatic Face-to-Face Translation
R, P. K., Mukhopadhyay, R., Philip, J., Jha, A., Namboodiri, V. P., & Jawahar, C. V. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351066 · Cited by 151 · see citing works
Self-citationUnsupervised Part-Based Disentangling of Object Shape and Appearance
Lorenz, D., Bereska, L. F., Milbich, T., & Ommer, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01121 · Cited by 145 · see citing works
ICface: Interpretable and Controllable Face Reenactment Using GANs
Tripathy, S., Kannala, J., & Rahtu, E. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093474 · Cited by 88 · see citing works
Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation
Li, L., Wang, S., Zhang, Z., Ding, Y., Zheng, Y., Yu, X., & Fan, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16286 · Cited by 68 · see citing works
Faces à la Carte: Text-to-Face Generation via Attribute Disentanglement
Wang, T., Zhang, T., & Lovell, B. C. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00342 · Cited by 49 · see citing works
Txt2Vid: Ultra-Low Bitrate Compression of Talking-Head Videos via Text
Tandon, P., Chandak, S., Pataranutaporn, P., Liu, Y., Mapuranga, A. M., Maes, P., Weissman, T., & Sra, M. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3221953 · Cited by 32 · see citing works
Flagged first, then by citations
2 of the 40 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Deepfakes generation and detection: a short survey
Akhtar, Z. (2023) · Journal of Imaging · doi:10.3390/jimaging9010018 · Cited by 106 · see citing works
DeepFake on Face and Expression Swap: A Review
Waseem, S., Abu Bakar, S. A. R. S., Ahmed, B. A., Omar, Z., Eisa, T. A. E., & Dalam, M. E. E. (2023) · IEEE Access · doi:10.1109/access.2023.3324403 · Cited by 65 · see citing works
AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset
Cai, Z., Ghosh, S., Adatia, A. P., Hayat, M., Dhall, A., Gedeon, T., & Stefanov, K. (2024) · Proceedings of the 32nd ACM International Conference on Multimedia · doi:10.1145/3664647.3680795 · Cited by 59 · see citing works
Deepfakes as a threat to a speaker and facial recognition: An overview of tools and attack vectors
Firc, A., Malinka, K., & Hanáček, P. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e15090 · Cited by 54 · see citing works
Visual Deepfake Detection: Review of Techniques, Tools, Limitations, and Future Prospects
Ur Rehman Ahmed, N., Badshah, A., Adeel, H., Tajammul, A., Daud, A., & Alsahfi, T. (2025) · IEEE Access · doi:10.1109/access.2024.3523288 · Cited by 45 · see citing works
Video and Audio Deepfake Datasets and Open Issues in Deepfake Technology: Being Ahead of the Curve
Akhtar, Z., Pendyala, T. L., & Athmakuri, V. S. (2024) · Forensic Sciences · doi:10.3390/forensicsci4030021 · Cited by 40 · see citing works
Face-Adapter for Pre-trained Diffusion Models with Fine-Grained ID and Attribute Control
Han, Y., Zhu, J., He, K., Chen, X., Ge, Y., Li, W., Li, X., Zhang, J., Wang, C., & Liu, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72973-7_2 · Cited by 37 · see citing works
Deepfake Generation and Detection: A Benchmark and Survey
Pei, G., Zhang, J., Hu, M., Zhang, Z., Wang, C., Wu, Y., Zhai, G., Yang, J. J., & Tao, D. (2026) · ACM Computing Surveys · doi:10.1145/3801962 · Cited by 113 · see citing works
Ntire 2025 Xgc Quality Assessment Challenge: Methods and Results
Liu, X., Min, X., Qiang, H., Zhang, X., Guo, J., Zhai, G., Wang, S., Zhou, Y., Lü, L., Li, J., et al. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00128 · Cited by 26 · see citing works
Rochow, A., Schwarz, M., & Behnke, S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00737 · Cited by 24 · see citing works
Audio-Visual Segmentation via Unlabeled Frame Exploitation
Liu, J., Liu, Y., Zhang, F., Ju, C., Zhang, Y., & Wang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02487 · Cited by 14 · see citing works
Increasing Importance of Joint Analysis of Audio and Video in Computer Vision: A Survey
Shahabaz, A., & Sarkar, S. (2024) · IEEE Access · doi:10.1109/access.2024.3391817 · Cited by 13 · see citing works
LPIPS-AttnWav2Lip: Generic audio-driven lip synchronization for talking head generation in the wild
Chen, Z. H., Wang, X., Xie, L., Yuan, H., & Pan, H. (2023) · Speech Communication · doi:10.1016/j.specom.2023.103028 · Cited by 10 · see citing works
Talking Face Generation With Audio-Deduced Emotional Landmarks
Zhai, S., Liu, M., Li, Y., Gao, Z., Zhu, L., & Nie, L. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3274676 · Cited by 8 · see citing works
A review of generative models for virtual human motion driving
Wang, P., Peng, H., Mao, S., Wang, S., & Tang, J. (2025) · Science China Information Sciences · doi:10.1007/s11432-023-4284-x · Cited by 4 · see citing works
Rakesh, V. K., Mazumdar, S., Pal, S., Das, A. K., & Samanta, T. (2025) · The Visual Computer · doi:10.1007/s00371-025-04232-w · Cited by 5 · see citing works
Face Omron Ring: Proactive defense against face forgery with identity awareness
Dai, Y., Fei, J., Huang, F., & Xia, Z. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106639 · Cited by 4 · see citing works
Learning and Evaluating Human Preferences for Conversational Head Generation
Zhou, M., Bai, Y., Zhang, W., Yao, T., Zhao, T., & Mei, T. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612831 · Cited by 4 · see citing works
Gao, Y., Li, J., Chu, L., & Lü, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01824 · Cited by 4 · see citing works
Anchored Diffusion for Video Face Reenactment
Kligvasser, I., Cohen, R., Leifman, G., Rivlin, E., & Elad, M. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00402 · Cited by 3 · see citing works
Gowda, S. N., Pandey, D., & Gowda, S. N. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4573122 · Cited by 3 · see citing works
Talking Face Generation via Face Mesh - Controllability without Reference Videos
Köksal, A., Xu, Q., & Lim, J. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00246 · Cited by 2 · see citing works
PSAIP: Prior Structure-Assisted Identity-Preserving Network for Face Animation
Zhao, G., Xu, J., Wang, X., Yan, F., & Qiu, S. (2025) · Electronics · doi:10.3390/electronics14040784 · Cited by 2 · see citing works
Learning Semantic Facial Descriptors for Accurate Face Animation
Zhu, L., Chen, Y., Liu, X., Li, T. H., & Li, G. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889019 · Cited by 1 · see citing works
Total-Editing: Head Avatar with Editable Appearance, Motion, and Lighting
Zhao, Y., Liu, C., Chen, H., Raj, B., Xu, M., Baltrušaitis, T., Rundle, M., Wu, H., & Dizaji, K. G. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00273 · Cited by 1 · see citing works
Zhao, G., Liu, Y., Wang, X., & Yan, F. (2024) · International Conference on Multimedia Retrieval · doi:10.1145/3652583.3658055 · Cited by 1 · see citing works
ELEGANT: End-to-end Language Grounded Speech Denoiser for Efficient Generation of Talking Face
Chai, A.-F., Rajanala, S., Pal, A., Phan, R. C. ‐., & Ting, C. (2023) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc58517.2023.10317456
EasyAnim: 3D facial animation from in-the-wild videos for avatars with customized riggings
Song, H.-X., Qian, Y., Zhan, X., & Mu, T.-J. (2025) · Graphical Models · doi:10.1016/j.gmod.2025.101298
Decoding Deepfakes: Classification of Facial Manipulations and Emerging Generation Tools
Ahmed, N. U. R., Badshah, A., Adeel, H., Tajammul, A., & Siddique, E. (2025) · IEEE Conference on Engineering Informatics (ICEI) · doi:10.1109/icei68356.2025.11603702
Kumar, T., Vavekanand, R., Javed, M., Kumari, S., Bhutto, S. H., Curry, E., Umair, M., Raj, K., & Asghar, M. N. (2026) · Algorithms · doi:10.3390/a19110942
- Authors
- Agarwal, M., Mukhopadhyay, R., Namboodiri, V., & Jawahar, C. V.
- Affiliations
- International Institute of Information Technology, Hyderabad (IN)
- University of Bath (GB)
- Venue
- 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
- Publisher
- IEEE
- Published
- 2023-01
- Pages
- 5167-5176
- ISSN
- 2472-6737 (ISSN-L)
- OpenAlex
- W4319301061
- Cited by
- 28
- References
- 40
- Citations by year
- 2026: 1 · 2025: 9 · 2024: 10 · 2023: 8
- FWCI
- 2.99
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Winter Conference on Applications of Computer Vision (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- talking head generation, lip synchronization, dense motion field, face segmentation, face mesh, audio-driven video generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
