Synthesizing Obama: learning lip sync from audio
The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
Abstract
Given audio of President Barack Obama, we synthesize a high quality video of him speaking with accurate lip sync, composited into a target video clip. Trained on many hours of his weekly address footage, a recurrent neural network learns the mapping from raw audio features to mouth shapes. Given the mouth shape at each time instant, we synthesize high quality mouth texture, and composite it with proper 3D pose matching to change what he appears to be saying in a target video to match the input audio track. Our approach produces photorealistic results.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“A2V(Suwajanakorn et al. 2017) has used a recurrent neural network to train a model that can map from raw audio features of Obama's weekly address footage to mouth shapes.”
From Countering Malicious DeepFakes: Survey, Battleground, and Horizon (2022) · cites this paper as Suwajanakorn et al. 2017
In the citing paper
“It is a cross-modal method that leverages the pronunciation features of the target person to synthesize the correct lip shapes for given audio content.”
From Countering Malicious DeepFakes: Survey, Battleground, and Horizon (2022) · cites this paper as Suwajanakorn et al. 2017
In the citing paper
“It doesn't need an original video as expression-driven material.”
From Countering Malicious DeepFakes: Survey, Battleground, and Horizon (2022) · cites this paper as Suwajanakorn et al. 2017
In the citing paper
“For example, A2V (Suwajanakorn et al. 2017) successfully synthesized fake videos of Obama (i.e., the 44th president of the United States) according to the given audio.”
From Countering Malicious DeepFakes: Survey, Battleground, and Horizon (2022) · cites this paper as Suwajanakorn et al. 2017
In the citing paper
“Then, they generate photorealistic mouth texture based on the generated lip fiducials.”
From Countering Malicious DeepFakes: Survey, Battleground, and Horizon (2022) · cites this paper as Suwajanakorn et al. 2017
6 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,839
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
Dlib-ml: A Machine Learning Toolkit
King, D. E. (2009) · doi:10.5555/1577069.1755843 · Cited by 2,777
Recurrent Neural Network Regularization
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 2,123
Supervised Descent Method and Its Applications to Face Alignment
Xiong, X., & De la Torre, F. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.75 · Cited by 1,896
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Research works printed but not linked: 5
- Fabrice Bellard , M Niedermayer , and others. 2012. FFmpeg. Availabel from: http://ffm.peg.org ( 2012 ). Fabrice Bellard, M Niedermayer, and others. 2012. FFmpeg. Availabel from: http://ffm.peg.org (2012).
- G. Bradski . 2000. Dr. Dobb's Journal of Software Tools ( 2000 ). G. Bradski. 2000. Dr. Dobb's Journal of Software Tools (2000).
- Yarin Gal . 2015. A theoretically grounded application of dropout in recurrent neural networks. arXiv preprint arXiv:1512.05287 ( 2015 ). Yarin Gal. 2015. A theoretically grounded application of dropout in recurrent neural networks. arXiv preprint arXiv:1512.05287 (2015). · arXiv
- Shunsuke Saito , Tianye Li , and Hao Li. 2016. Real-Time Facial Segmentation and Performance Capture from RGB Input. arXiv preprint arXiv:1604.02647 ( 2016 ). Shunsuke Saito, Tianye Li, and Hao Li. 2016. Real-Time Facial Segmentation and Performance Capture from RGB Input. arXiv preprint arXiv:1604.02647 (2016). · arXiv
- Aäron van den Oord , Sander Dieleman , Heiga Zen , Karen Simonyan , Oriol Vinyals , Alex Graves , Nal Kalchbrenner , Andrew Senior , and Koray Kavukcuoglu . 2016 . Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016). Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 (2016). · arXiv
Cited by
Items 101–140 of 140
The Real Threat of Deepfake Pornography: A Review of Canadian Policy
Karasavva, V., & Noorbhai, A. (2021) · Cyberpsychology, Behavior, and Social Networking · doi:10.1089/cyber.2020.0272 · Cited by 85
Lewis, J. K., Toubal, I. E., Chen, H., Sandesera, V., Lomnitz, M., Hampel-Arias, Z., Prasad, C., & Palaniappan, K. (2020) · 2020 IEEE Applied Imagery Pattern Recognition Workshop (AIPR) · doi:10.1109/aipr50011.2020.9425167 · Cited by 80
Gamage, D., Ghasiya, P., Bonagiri, V., Whiting, M. E., & Sasahara, K. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517446 · Cited by 79
Detecting Deepfake Videos with Temporal Dropout 3DCNN
Zhang, D., Li, C., Lin, F., Zeng, D., & Ge, S. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/178 · Cited by 77
An efficient deepfake video detection using robust deep learning
Qadir, A., Mahum, R., El-Meligy, M. A., Ragab, A. E., AlSalman, A., & Awais, M. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25757 · Cited by 78
iCaps-Dfake: An Integrated Capsule-Based Model for Deepfake Image and Video Detection
Khalil, S. S., Youssef, S. M., & Saleh, S. N. (2021) · Future Internet · doi:10.3390/fi13040093 · Cited by 67
Qureshi, S. M., Saeed, A., Almotiri, S. H., Ahmad, F., & Al Ghamdi, M. A. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2037 · Cited by 71
DeepFake on Face and Expression Swap: A Review
Waseem, S., Abu Bakar, S. A. R. S., Ahmed, B. A., Omar, Z., Eisa, T. A. E., & Dalam, M. E. E. (2023) · IEEE Access · doi:10.1109/access.2023.3324403 · Cited by 65
Deep learning for visual speech analysis: A survey
Sheng, C., Kuang, G., Bai, L., Hou, C., Guo, Y., Xu, X., Pietikäinen, M., & Liu, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3376710 · Cited by 64
Human performance in detecting deepfakes: A systematic review and meta-analysis of 56 papers
Diel, A., Lalgi, T., Schröter, I. C., MacDorman, K. F., Teufel, M., & Bäuerle, A. (2024) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2024.100538 · Cited by 64
Enhanced deepfake detection with densenet and cross-vit
Siddiqui, F., Yang, J., Xiao, S., & Fahad, M. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126150 · Cited by 62
Review of audio deepfake detection techniques: Issues and prospects
Dixit, A., Kaur, N., & Kingra, S. (2023) · Expert Systems · doi:10.1111/exsy.13322 · Cited by 56
Locate and Verify: A Two-Stream Network for Improved Deepfake Detection
Shuai, C., Zhong, J., Wu, S., Lin, F., Wang, Z., Ba, Z., Liu, Z., Cavallaro, L., & Ren, K. (2023) · Proceedings of the 31st ACM International Conference on Multimedia · doi:10.1145/3581783.3612386 · Cited by 55
ADD: Attention-Based DeepFake Detection Approach
Khormali, A., & Yuan, J.-S. (2021) · Big Data and Cognitive Computing · doi:10.3390/bdcc5040049 · Cited by 55
Improving Generalization of Deepfake Detection With Data Farming and Few-Shot Learning
Korshunov, P., & Marcel, S. (2022) · IEEE Transactions on Biometrics, Behavior, and Identity Science · doi:10.1109/tbiom.2022.3143404 · Cited by 50
Deepfake Video Detection with Spatiotemporal Dropout Transformer
Zhang, D., Lin, F., Hua, Y., Wang, P., Zeng, D., & Ge, S. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3547913 · Cited by 50
DeepFake Disrupter: The Detector of DeepFake Is My Friend
Wang, X., Huang, J., Ma, S., Nepal, S., & Xu, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01450 · Cited by 45
Modality Dropout for Improved Performance-driven Talking Faces
Hussen Abdelaziz, A., Theobald, B.-J., Dixon, P., Knothe, R., Apostoloff, N., & Kajareker, S. (2020) · Proceedings of the 2020 International Conference on Multimodal Interaction · doi:10.1145/3382507.3418840 · Cited by 42
The future of online trust (and why Deepfake is advancing it)
Etienne, H. (2021) · AI and Ethics · doi:10.1007/s43681-021-00072-1 · Cited by 41
Face Forgery Detection by 3D Decomposition and Composition Search
Zhu, X., Fei, H., Zhang, B., Zhang, T., Zhang, X., Li, S. Z., & Lei, Z. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3233586 · Cited by 41
Joint Audio-Visual Attention with Contrastive Learning for More General Deepfake Detection
Zhang, Y., Lin, W., & Xu, J. (2024) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3625100 · Cited by 40
SafeEar: Content Privacy-Preserving Audio Deepfake Detection
Li, X., Li, K., Zheng, Y., Yan, C., Ji, X., & Xu, W. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670285 · Cited by 39
Deep Insights of Deepfake Technology : A Review
Mahmud, B. U., & Sharmin, A. (2021) · arXiv · doi:10.48550/arxiv.2105.00192 · Cited by 38
A comprehensive evaluation of feature-based AI techniques for deepfake detection
Sandotra, N., & Arora, B. (2024) · Neural Computing and Applications · doi:10.1007/s00521-023-09288-0 · Cited by 37
Emergence of deepfakes and video tampering detection approaches: A survey
Kingra, S., Aggarwal, N., & Kaur, N. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13100-x · Cited by 33
Watch Those Words: Video Falsification Detection Using Word-Conditioned Facial Motion
Agarwal, S., Hu, L., Ng, E., Darrell, T., Li, H., & Rohrbach, A. (2023) · 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv56688.2023.00469 · Cited by 28
Deepfake Detection by Exploiting Surface Anomalies: The Surfake Approach
Ciamarra, A., Caldelli, R., Becattini, F., Seidenari, L., & Del Bimbo, A. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00112 · Cited by 27
Deepfakes Detection Methods: A Literature Survey
Weerawardana, M., & Fernando, T. (2021) · 2021 10th International Conference on Information and Automation for Sustainability (ICIAfS) · doi:10.1109/iciafs52090.2021.9606067 · Cited by 23
Videoforensicshq: Detecting High-Quality Manipulated Face Videos
Fox, G., Liu, W., Kim, H., Seidel, H.-P., Elgharib, M., & Theobalt, C. (2021) · 2021 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428101 · Cited by 22
Ng, Y.-L. (2023) · Current Psychology · doi:10.1007/s12144-022-03621-x · Cited by 19
A defensive attention mechanism to detect deepfake content across multiple modalities
Asha, S., Vinod, P., & Menon, V. G. (2024) · Multimedia Systems · doi:10.1007/s00530-023-01248-x · Cited by 16
Sonic: Shifting Focus to Global Audio Perception in Portrait Animation
Ji, X., Hu, X., Xu, Z., Zhu, J., Lin, C., He, Q., Zhang, J., Luo, D., Chen, Y., Lin, Q., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.16331 · Cited by 14
Memories are One-to-Many Mapping Alleviators in Talking Face Generation
Tang, A., He, T., Tan, X., Ling, J., & Song, L. (2022) · arXiv · doi:10.48550/arxiv.2212.05005 · Cited by 12
Counteracting the contemporaneous proliferation of digital forgeries and fake news
FERREIRA, A., CARVALHO, T., ANDALÓ, F., & ROCHA, A. (2019) · Anais da Academia Brasileira de Ciências · doi:10.1590/0001-3765201820180149 · Cited by 11
“2016), by learning lip-sync from audio (Suwajanakorn et al. 2017), or by animating static images (Averbuch-Elor et al.”
BackgroundFine-grained talking face generation with video reinterpretation
Huang, X., Wang, M., & Gong, M. (2021) · The Visual Computer · doi:10.1007/s00371-020-01982-7 · Cited by 11
Multisemantic Path Neural Network for Deepfake Detection
Wu, N., Jin, X., Jiang, Q., Wang, P., Zhang, Y., Yao, S., & Zhou, W. (2022) · Security and Communication Networks · doi:10.1155/2022/4976848 · Cited by 9
Deepfake Video Forensics based on Transfer Learning
Student, D. O. C. S. A. E., U*, R., M, R., Student, D. O. C. S. A. E., Vignesh K, R., Student, D. O. C. S. A. E., K, T., & Department of Computer Science and Engineering, R. E. C. (. U. (2020) · International Journal of Recent Technology and Engineering (IJRTE) · doi:10.35940/ijrte.f9747.038620 · Cited by 9
“apos;s suggestion for an approach to recording real time facial shifts from one clip of video and re-enacting it on various RGB videos, any sound to lip join technique suggested by Suwajanakorn et al [2] has built a Recurrent Neural Network (RNN) capable of matching raw audio characteristics to certain mouths. Generat…”
MethodCan Deepfakes be created on a whim?
Mehta, P., Jagatap, G., Gallagher, K., Timmerman, B., Deb, P., Garg, S., Greenstadt, R., & Dolan-Gavitt, B. (2023) · Companion Proceedings of the ACM Web Conference 2023 · doi:10.1145/3543873.3587581 · Cited by 5
Chrominance and Luminance: a study to detect deepfakes
Talib, M. A., Nasir, Q., Nassif, A. B., Fadhl, N. B., & Gouda, O. M. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-025-20618-3 · Cited by 5
Quantitative Metrics for Evaluating Explanations of Video DeepFake Detectors
Baldassarre, F., Debard, Q., Pontiveros, G. F., & Wijaya, T. K. (2022) · arXiv · doi:10.48550/arxiv.2210.03683 · Cited by 1
Metadata
- Authors
- Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I.
- Affiliations
- University of Washington (US)
- Venue
- ACM Transactions on Graphics
- Published
- 2017-07-20
- Volume
- 36
- Issue
- 4
- Pages
- 1-13
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W2738406145
- Semantic Scholar
- 7aa88dafb5d5fd5645c0ada2539e9eaf5b2fe949
- Cited by
- 1,057
- References
- 53
- Citations by year
- 2026: 46 · 2025: 101 · 2024: 135 · 2023: 169 · 2022: 139 · 2021: 159 · 2020: 137 · 2019: 97 · 2018: 67 · 2017: 6
- FWCI
- 34.40
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Transactions on Graphics (publishedVersion)
- Funders
- Samsung
- Intel
- University of Washington Animation Research Labs
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- lip synchronization, audiovisual speech, video compositing, recurrent neural networks, Barack Obama
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.