The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
Text-based editing of talking-head video
Open accessAbstract
Editing talking-head video to change the speech content or to remove filler words is challenging. We propose a novel method to edit talking-head video based on its transcript to produce a realistic output video in which the dialogue of the speaker has been modified, while maintaining a seamless audio-visual flow (i.e. no jump cuts). Our method automatically annotates an input talking-head video with phonemes, visemes, 3D face pose and geometry, reflectance, expression and scene illumination per frame. To edit a video, the user has to only edit the transcript, and an optimization strategy then chooses segments of the input corpus as base material. The annotated parameters corresponding to the selected segments are seamlessly stitched together and used to produce an intermediate video representation in which the lower half of the face is rendered with a parametric face model. Finally, a recurrent video generation network transforms this representation to a photorealistic video that matches the edited transcript. We demonstrate a large variety of edits, such as the addition, removal, and alteration of words, as well as convincing language translation and full sentence synthesis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387
Conditional Generative Adversarial Nets
Mirza, M., & Osindero, S. (2014) · arXiv · doi:10.48550/arxiv.1411.1784 · Cited by 8,863
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
A morphable model for the synthesis of 3D faces
Blanz, V., & Vetter, T. R. (1999) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/311535.311556 · Cited by 4,695
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791
TIMIT Acoustic-Phonetic Continuous Speech Corpus
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Pallett, D. S., Dahlgren, N. L., Zue, V. W., & Fiscus, J. G. (2020) · Americanae (AECID Library) · doi:10.35111/17gk-bn40 · Cited by 2,509
Research works printed but not linked: 10
- Chan Caroline, 2018, Efros
- Dou Pengfei, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Genova Kyle, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
- Ian J. Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems. Ian J. Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems.
- Karras Tero, 2018, International Conference on Learning Representations (ICLR).
- 10.1145/3197517.3201283 · DOI
- Levenshtein Vladimir I, Soviet physics doklady
- Ochshorn Robert, 2016, Gentle: A Forced Aligner. https://lowerquality.com/gentle/.
- Ting-Chun Wang Ming-Yu Liu Jun-Yan Zhu Andrew Tao Jan Kautz and Bryan Catanzaro. 2018b. High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. In CVPR. Ting-Chun Wang Ming-Yu Liu Jun-Yan Zhu Andrew Tao Jan Kautz and Bryan Catanzaro. 2018b. High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. In CVPR.
- Wiles O., European Conference on Computer Vision.
Cited by
Sorted by citations
Deepfakes and beyond: A Survey of face manipulation and fake detection
Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., & Ortega-Garcia, J. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.014 · Cited by 1,237
Nerfies: Deformable Neural Radiance Fields
Park, K., Sinha, U., Barron, J. T., Bouaziz, S., Goldman, D. B., Seitz, S. M., & Martin-Brualla, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00581 · Cited by 1,221
Self-citationMedia Forensics and DeepFakes: An Overview
Verdoliva, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3002101 · Cited by 979
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897
Park, K., Sinha, U., Hedman, P., Barron, J. T., Bouaziz, S., Goldman, D. B., Martin-Brualla, R., & Seitz, S. M. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480487 · Cited by 986
Self-citationMasood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543
Kietzmann, J., Lee, L. W., McCarthy, I. P., & Kietzmann, T. C. (2020) · Business Horizons · doi:10.1016/j.bushor.2019.11.006 · Cited by 504
Deep Learning for Deepfakes Creation and Detection: A Survey
Nguyen, T. T., Nguyen, Q. V. H., Nguyen, D. T., Nguyen, D. T., Huynh-The, T., Nahavandi, S., Nguyen, T. T., Pham, Q.-V., & Nguyen, C. M. (2019) · arXiv · doi:10.48550/arxiv.1909.11573 · Cited by 464
Deep learning for deepfakes creation and detection: A survey
Nguyen, T. T., Nguyen, Q. V. H., Nguyen, D. T., Nguyen, D. T., Huynh-The, T., Nahavandi, S., Nguyen, T. T., Pham, Q.-V., & Nguyen, C. M. (2022) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103525 · Cited by 464
“Audio-to-Video (A2V) [68] and Text-to-Video (T2V) [142].”
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
Wang, T.-C., Mallya, A., & Liu, M.-Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00991 · Cited by 496
State of the Art on Neural Rendering
Tewari, A., Fried, O., Thies, J., Sitzmann, V., Lombardi, S., Sunkavalli, K., Martin-Brualla, R., Simon, T., Saragih, J., Nießner, M., et al. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14022 · Cited by 480
Self-citationAbdal, R., Zhu, P., Mitra, N. J., & Wonka, P. (2021) · ACM Transactions on Graphics · doi:10.1145/3447648 · Cited by 514
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis
Guo, Y., Chen, K., Liang, S., Liu, Y.-J., Bao, H., & Zhang, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00573 · Cited by 384
Neural Voice Puppetry: Audio-Driven Facial Reenactment
Thies, J., Elgharib, M. A., Tewari, A., Theobalt, C., & Nießner, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_42 · Cited by 411
Self-citationFlow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset
Zhang, Z., Li, L., Ding, Y., & Fan, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00366 · Cited by 368
CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields
Wang, C., Chai, M., He, M., Chen, D., & Liao, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00381 · Cited by 403
MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation
Wang, K., Wu, Q., Song, L., Yang, Z., Wu, W., Qian, C., He, R., Qiao, Y., & Loy, C. C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58589-1_42 · Cited by 367
DeepFake Detection for Human Face Images and Videos: A Survey
Malik, A., Kuribayashi, M., Abdullahi, S. M., & Khan, A. N. (2022) · IEEE Access · doi:10.1109/access.2022.3151186 · Cited by 297
FaceFormer: Speech-Driven 3D Facial Animation with Transformers
Fan, Y., Lin, Z., Saito, J., Wang, W., & Komura, T. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01821 · Cited by 295
A Survey on Deepfake Video Detection
Yu, P., Xia, Z., Fei, J., & Lu, Y. (2021) · IET Biometrics · doi:10.1049/bme2.12031 · Cited by 253
Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
Liu, M.-Y., Huang, X., Yu, J., Wang, T.-C., & Mallya, A. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3049196 · Cited by 236
Detecting Deep-Fake Videos from Phoneme-Viseme Mismatches
Agarwal, S., Farid, H., Fried, O., & Agrawala, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00338 · Cited by 234
Self-citationAudio-Driven Emotional Video Portraits
Ji, X., Zhou, H., Wang, K., Wu, W., Loy, C. C., Cao, X., & Xu, F. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01386 · Cited by 235
CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior
Xing, J., Xia, M., Zhang, Y., Cun, X., Wang, J., & Wong, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01229 · Cited by 238
Anticipating and Addressing the Ethical Implications of Deepfakes in the Context of Elections
Diakopoulos, N. A., & Johnson, D. K. (2019) · New Media & Society · doi:10.1177/1461444820925811 · Cited by 235
Anticipating and Addressing the Ethical Implications of Deepfakes in the Context of Elections
Diakopoulos, N., & Johnson, D. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3474183 · Cited by 194
Talking-Head Generation with Rhythmic Head Motion
Chen, L., Cui, G., Liu, C., Li, Z., Kou, Z., Xu, Y., & Xu, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7_3 · Cited by 179
StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN
Yin, F., Zhang, Y., Cun, X., Cao, M., Fan, Y., Wang, X., Bai, Q., Wu, B., Wang, J., & Yang, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19790-1_6 · Cited by 180
Lu, Y., Chai, J., & Cao, X. (2021) · ACM Transactions on Graphics · doi:10.1145/3478513.3480484 · Cited by 188
Generative AI-Driven Semantic Communication Networks: Architecture, Technologies, and Applications
Liang, C., Du, H., Sun, Y., Niyato, D. T., Kang, J., Zhao, D., & Imran, M. A. (2024) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2024.3435524 · Cited by 193
Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion
Wang, S., Li, L., Ding, Y., Fan, C., & Yu, X. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/152 · Cited by 169
EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model
Ji, X., Zhou, H., Wang, K., Wu, Q., Wu, W., Xu, F., & Cao, X. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530745 · Cited by 155
ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis
He, Y., Gan, B., Chen, S., Zhou, Y., Yin, G., Song, L., Sheng, L., Shao, J., & Liu, Z. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00434 · Cited by 161
A comprehensive overview of Deepfake: generation, detection, datasets, and opportunities
Seow, J. W., Lim, M. K., Phan, R. C. W., & Liu, J. K. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.09.135 · Cited by 149
Use of a Capsule Network to Detect Fake Images and VideosPreprint
Nguyen, H. H., Yamagishi, J., & Echizen, I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12467 · Cited by 192
Everybody’s Talkin’: Let Me Talk as You Want
Song, L., Wu, W., Qian, C., He, R., & Loy, C. C. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3146783 · Cited by 142
Deep Learning for Deepfakes Creation and Detection: A Survey
Nguyen, T. T., Nguyen, M. C., Nguyen, D. T., Nguyen, D. T., & Nahavandi, S. (2020) · Figshare · doi:10.6084/m9.figshare.12731039.v1 · Cited by 122
One-Shot Talking Face Generation from Single-Speaker Audio-Visual Correlation Learning
Wang, S., Li, L., Ding, Y., & Yu, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20154 · Cited by 115
Abdal, R., Zhu, P., Mitra, N., & Wonka, P. (2021) · UCL Discovery (University College London) · Cited by 111
Improving Generalization by Commonality Learning in Face Forgery Detection
Yu, P., Fei, J., Xia, Z., Zhou, Z., & Weng, J. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3146781 · Cited by 117
Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis
Shen, S., Li, W., Zhu, Z. D., Duan, Y., Zhou, J., & Lu, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19775-8_39 · Cited by 107
CollaboVR: A Reconfigurable Framework for Creative Collaboration in Virtual Reality
He, Z., Du, R., & Perlin, K. (2020) · Proceedings - International Symposium on Mixed and Augmented Reality, ISMAR · doi:10.1109/ismar50242.2020.00082 · Cited by 103
Lahiri, A., Kwatra, V., Frueh, C., Lewis, J., & Bregler, C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00278 · Cited by 101
Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars
Sun, J., Wang, X., Wang, L., Li, X., Zhang, Y., Zhang, H., & Liu, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02011 · Cited by 98
Preventing DeepFake Attacks on Speaker Authentication by Dynamic Lip Movement Analysis
Yang, C.-Z., Ma, J., Wang, S., & Liew, A. W. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2020.3045937 · Cited by 103
Creating, Using, Misusing, and Detecting Deep Fakes
Farid, H. (2022) · Journal of Online Trust and Safety · doi:10.54501/jots.v1i4.56 · Cited by 122
Rewriting a Deep Generative Model
Bau, D., Liu, S., Tongzhou, W., Zhu, J.-Y., & Torralba, A. B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58452-8_21 · Cited by 101
DurIAN: Duration Informed Attention Network For Multimodal SynthesisPreprint
Yu, C., Lu, H., Hu, N., Yu, M., Weng, C., Xu, K., Liu, P., Tuo, D., Kang, S., Lei, G., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01700 · Cited by 84
Photorealistic Audio-driven Video Portraits
Wen, X. L., Wang, M., Richardt, C., Chen, Z.-Y., & Hu, S. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3023573 · Cited by 81
StyleTalk: One-Shot Talking Head Generation with Controllable Speaking Styles
Ma, Y., Wang, S., Hu, Z., Fan, C., Lv, T., Ding, Y., Deng, Z., & Yu, X. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25280 · Cited by 79
HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance FieldsPreprint
Park, K., Sinha, U., Hedman, P., Barron, J. T., Bouaziz, S., Goldman, D. B., Martin-Brualla, R., & Seitz, S. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.13228 · Cited by 77
Self-citationPavel, A., Reyes, G., & Bigham, J. P. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379337.3415864 · Cited by 74
Tolosana, R., Romero-Tapiador, S., Vera-Rodríguez, R., González-Sosa, E., & Fiérrez, J. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.104673 · Cited by 77
FSGANv2: Improved Subject Agnostic Face Swapping and Reenactment
Nirkin, Y., Keller, Y., & Hassner, T. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3155571 · Cited by 74
Human-Computer Interaction System: A Survey of Talking-Head Generation
Zhen, R., Song, W., He, Q., Cao, J., Shi, L., & Luo, J. (2023) · Electronics · doi:10.3390/electronics12010218 · Cited by 76
Privacy Intelligence: A Survey on Image Privacy in Online Social Networks
Liu, C., Zhu, T., Zhang, J., & Zhou, W. (2022) · ACM Computing Surveys · doi:10.1145/3547299 · Cited by 88
Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation
Li, L., Wang, S., Zhang, Z., Ding, Y., Zheng, Y., Yu, X., & Fan, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16286 · Cited by 72
A literature review and perspectives in deepfakes: generation, detection, and applications
Dagar, D., & Vishwakarma, D. K. (2022) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-022-00241-w · Cited by 83
DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video
Zhang, Z., Hu, Z., Deng, W., Fan, C., Lv, T., & Ding, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i3.25464 · Cited by 67
VirtualCube: An Immersive 3D Video Communication System
Zhang, Y., Yang, J., Liu, Z., Wang, R., Chen, G., Tong, X., & Guo, B. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3150512 · Cited by 66
LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing
Wang, B. K., Li, Y., Lv, Z., Xia, H., Xu, Y., & Sodhi, R. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645143 · Cited by 96
Liu, Y., Nadai, M. D., Cai, D., Li, H., Alameda-Pineda, X., Sebe, N., & Lepri, B. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413505 · Cited by 63
MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptation
Zhang, B., Qi, C., Zhang, P., Zhang, B., Wu, H., Chen, D., Chen, Q., Wang, Y., & Wen, F. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02116 · Cited by 58
DPE: Disentanglement of Pose and Expression for General Video Portrait Editing
Pang, Y., Zhang, Y., Quan, W., Fan, Y., Cun, X., Shan, Y., & Dong-ming Yan, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00049 · Cited by 56
Audio-Driven Talking Face Video Generation With Dynamic Convolution Kernels
Ye, Z., Xia, M., Yi, R., Zhang, J., Lai, Y., Huang, X., Zhang, G., & Liu, Y. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3142387 · Cited by 55
AVscript: Accessible Video Editing with Audio-Visual Scripts
Huh, M., Yang, S., Peng, Y.-H., Chen, X., Kim, Y., & Pavel, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581494 · Cited by 58
On the Generalization of Deep Learning Models in Video Deepfake Detection
Coccomini, D. A., Caldelli, R., Falchi, F., & Gennaro, C. (2023) · Preprint (MDPI) · doi:10.20944/preprints202303.0161.v1 · Cited by 46
“The fake images are manipulated using 15 different manipulations while the videos are manipulated using only 8 of them [27–36].”
MethodFollow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animation
Ma, Y., Liu, H., Wang, H., Pan, H., He, Y., Yuan, J., Zeng, A., Cai, C., Shum, H., Liu, W., et al. (2024) · SIGGRAPH Asia Conference Papers · doi:10.1145/3680528.3687587 · Cited by 118
Crosscast: Adding Visuals to Audio Travel Podcasts
Xia, H., Jacobs, J., & Agrawala, M. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379337.3415882 · Cited by 45
Self-citationCross-Forgery Analysis of Vision Transformers and CNNs for Deepfake Image Detection
Coccomini, D. A., Caldelli, R., Falchi, F., Gennaro, C., & Amato, G. (2022) · Proceedings of the 1st International Workshop on Multimedia AI against Disinformation · doi:10.1145/3512732.3533582 · Cited by 47
Learned Spatial Representations for Few-shot Talking-Head Synthesis
Meshry, M., Suri, S., Davis, L. S., & Shrivastava, A. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01357 · Cited by 44
The Ethical Implications of Generative Audio Models: A Systematic Literature Review
Barnett, J. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604686 · Cited by 61
Deep Person Generation: A Survey from the Perspective of Face, Pose, and Cloth Synthesis
Sha, T., Zhang, W., Shen, T., Li, Z., & Mei, T. (2022) · ACM Computing Surveys · doi:10.1145/3575656 · Cited by 46
SD-NeRF: Towards Lifelike Talking Head Animation via Spatially-Adaptive Dual-Driven NeRFs
Shen, S., Li, W., Huang, X., Zhu, Z. D., Zhou, J., & Lu, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3308441 · Cited by 39
Age of Incorrect Information in Semantic Communications for NOMA Aided XR Applications
Chen, J., Wang, J., Jiang, C., & Wang, J. (2023) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2023.3282836 · Cited by 38
Video and Audio Deepfake Datasets and Open Issues in Deepfake Technology: Being Ahead of the Curve
Akhtar, Z., Pendyala, T. L., & Athmakuri, V. S. (2024) · Forensic Sciences · doi:10.3390/forensicsci4030021 · Cited by 45
Generating Audio-Visual Slideshows from Text Articles Using Word Concreteness
Leake, M., Shin, H. V., Kim, J. O., & Agrawala, M. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376519 · Cited by 37
Self-citationAI-Powered MOOCs: Video Lecture Generation
Dao, X.-Q., Le, N.-B., & Nguyen, T.-M.-T. (2021) · International Conference on Image, Video and Signal Processing (IVSP) · doi:10.1145/3459212.3459227 · Cited by 35
Towards Realistic Visual Dubbing with Heterogeneous Sources
Xie, T., Liao, L., Bi, C., Tang, B., Yin, X., Yang, J., Wang, M., Yao, J., Zhang, Y., & Ma, Z. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475318 · Cited by 35
A comprehensive evaluation of feature-based AI techniques for deepfake detection
Sandotra, N., & Arora, B. (2024) · Neural Computing and Applications · doi:10.1007/s00521-023-09288-0 · Cited by 37
Multimodal Learning for Temporally Coherent Talking Face Generation With Articulator Synergy
Yu, L., Xie, H., & Zhang, Y. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3091863 · Cited by 34
PodReels: Human-AI Co-Creation of Video Podcast Teasers
Wang, S., Ning, Z., Truong, A., Dontcheva, M., Li, D., & Chilton, L. B. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661591 · Cited by 33
Text2video: Text-Driven Talking-Head Video Synthesis with Personalized Phoneme - Pose Dictionary
Zhang, S., Yuan, J., Liao, M., & Zhang, L. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747380 · Cited by 34
Xia, H. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379337.3415845 · Cited by 38
Txt2Vid: Ultra-Low Bitrate Compression of Talking-Head Videos via Text
Tandon, P., Chandak, S., Pataranutaporn, P., Liu, Y., Mapuranga, A. M., Maes, P., Weissman, T., & Sra, M. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3221953 · Cited by 36
Iterative Text-Based Editing of Talking-Heads Using Neural Retargeting
Yao, X., Fried, O., Fatahalian, K., & Agrawala, M. (2021) · ACM Transactions on Graphics · doi:10.1145/3449063 · Cited by 31
Self-citationF³A-GAN: Facial Flow for Face Animation With Generative Adversarial Networks
Wu, X., Zhang, Q., Wu, Y., Wang, H., Li, S., Sun, L., & Li, X. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3112059 · Cited by 31
Automatic Instructional Video Creation from a Markdown-Formatted Tutorial
Chi, P., Frey, N., Panovich, K., & Essa, I. A. (2021) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3472749.3474778 · Cited by 29
MINTIME: Multi-Identity Size-Invariant Video Deepfake Detection
Coccomini, D. A., Zilos, G. K., Amato, G., Caldelli, R., Falchi, F., Papadopoulos, S., & Gennaro, C. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3409054 · Cited by 33
Chen, W., Tan, X., Xia, Y., Qin, T., Wang, Y., & Liu, T.-Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413623 · Cited by 28
Watch Those Words: Video Falsification Detection Using Word-Conditioned Facial Motion
Agarwal, S., Hu, L., Ng, E., Darrell, T., Li, H., & Rohrbach, A. (2023) · 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv56688.2023.00469 · Cited by 28
ExpressEdit: Video Editing with Natural Language and Sketching
Tilekbay, B., Yang, S., Lewkowicz, M. A., Suryapranata, A., & Kim, J. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645164 · Cited by 38
Sparse to Dense Motion Transfer for Face Image Animation
Zhao, R., Wu, T., & Guo, G. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00226 · Cited by 27
Deep learning applications on cybersecurity: A practical approach
Miranda-García, A., Rego, A. Z., Pastor-López, I., Sanz, B., Tellaeche, A., de la Puerta, J. G., & Bringas, P. G. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126904 · Cited by 30
SliderGAN: Synthesizing Expressive Face Images by Sliding 3D Blendshape Parameters
Ververas, E., & Zafeiriou, S. P. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01338-7 · Cited by 26
Automatic Video Creation From a Web Page
Chi, P., Sun, Z., Panovich, K., & Essa, I. A. (2020) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3379337.3415814 · Cited by 25
Compositional Video Synthesis with Action GraphsPreprint
Bar, A., Herzig, R., Wang, X., Rohrbach, A., Chechik, G., Darrell, T., & Globerson, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15327 · Cited by 26
An Introduction to Digital Face Manipulation
Tolosana, R., Vera-Rodríguez, R., Fiérrez, J., Morales, A., & Ortega-García, J. (2022) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-030-87664-7_1 · Cited by 28
TellMeTalk: Multimodal-driven talking face video generation
Li, P., Zhao, H., Liu, Q., Tang, P., & Zhang, L. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2023.109049 · Cited by 22
Videoforensicshq: Detecting High-Quality Manipulated Face Videos
Fox, G., Liu, W., Kim, H., Seidel, H.-P., Elgharib, M., & Theobalt, C. (2021) · 2021 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428101 · Cited by 22
Self-citation
Metadata
- Authors
- Fried, O., Tewari, A., Zollhöfer, M., Finkelstein, A., Shechtman, E., Goldman, D. B., Genova, K., Jin, Z., Theobalt, C., & Agrawala, M.
- Affiliations
- Stanford University (US)
- Max Planck Institute for Informatics (DE)
- Princeton University (US)
- Adobe Systems (United States) (US)
- ORCID
- Venue
- ACM Transactions on Graphics
- Published
- 2019-07-12
- Volume
- 38
- Issue
- 4
- Pages
- 1-14
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2948284812
- Cited by
- 270
- References
- 73
- Citations by year
- 2026: 14 · 2025: 30 · 2024: 36 · 2023: 51 · 2022: 46 · 2021: 47 · 2020: 39 · 2019: 7
- FWCI
- 15.55
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Graphics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Adobe Systems
- European Research Council (Consolidator Grant 4DRepLy #770784)
- Princeton University
- Brown Institute for Media Innovation
- National Science Foundation (NSF GRFP #DGE-1656466)
- Office of the Dean for Research, Princeton University
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- speech modification, face geometry, reflectance estimation, facial expression estimation, video generation, speech translation, speech synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
