You Said That?: Synthesising Talking Faces from Audio
Open accessThe DOI matches its Crossref record; 53 of the 54 references it cites were found in a registry or an index.
Abstract
We describe a method for generating a video of a talking face. The method takes still images of the target face and an audio speech segment as inputs, and generates a video of the target face lip synched with the audio. The method runs in real time and is applicable to faces and audio not seen at training time. To achieve this we develop an encoder–decoder convolutional neural network (CNN) model that uses a joint embedding of the face and audio to generate synthesised talking face video frames. The model is trained on unlabelled videos using cross-modal self-supervision. We also propose methods to re-dub videos by visually blending the generated face into the source video frame using a multi-stream CNN model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Thus, we believe that our model can enable a wide range of real-world applications where previous speaker-independent lipsyncing approaches [17, 18] struggle to produce satisfactory results.”
From A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild (2020) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We find that these models do not adequately penalize wrong lip shapes, either as a result of using only reconstruction losses or weak lip-sync discriminators.”
From A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild (2020) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To realize mapping driving sources to lip dynamics, representative traditional VSG methods mainly adopted cross-modal retrieval approaches [18, 109, 152, 153] and HMM-based approaches [154, 155].”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [155]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Wav2Lip [6] facial reenactment is used for this task as it has been shown to have better output quality than previous methods [61], [62], and has better generalization and robustness to unseen scenarios.”
From Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization (2022) · cites this paper as [61]
In the citing paper
“An outcome worth noting is that the previous works [17, 18] which produce several out-of-sync segments are less preferred over the unsynced version as the latter still preserves good Visual quality.”
From A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild (2020) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
(1981) · Cited by 11,963
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 2×
“Image to text is the most prominent, e.g. in caption generation (Vinyals et al. 2015; Karpathy and Fei-Fei 2015; Xu et al. 2015), but there is also text to image (Reed et al. 2016), video to sound (Owens et al. 2016), or in fact a combination of different mediums e.g. video and audio to text (Chung…”
BackgroundAccurate Image Super-Resolution Using Very Deep Convolutional Networks
Kim, J., Lee, J. K., & Lee, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.182 · Cited by 7,870
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
Image Style Transfer Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.265 · Cited by 6,025
Context Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
Parkhi, O. M., Vedaldi, A., & Zisserman, A. (2015) · Procedings of the British Machine Vision Conference 2015 · doi:10.5244/c.29.41 · Cited by 5,479
Self-citationDeep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Self-citationZhang, R., Isola, P., & Efros, A. A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_40 · Cited by 3,178
One millisecond face alignment with an ensemble of regression trees
Kazemi, V., & Sullivan, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.241 · Cited by 2,831
Dlib-ml: A Machine Learning Toolkit
King, D. E. (2009) · doi:10.5555/1577069.1755843 · Cited by 2,777
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · cited 2×
“The work on predicting co-occurrence (Isola et al. 2016), context (Doersch et al. 2015), and colorization (Zhang et al. 2016) fall under this category.”
BackgroundGenerative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494
MatConvNet: Convolutional Neural Networks for MATLAB
Vedaldi, A., & Lenc, K. (2015) · Proceedings of the 23rd ACM international conference on Multimedia · doi:10.1145/2733373.2807412 · Cited by 2,444
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
Self-citationPérez, P., Gangnet, M., & Blake, A. (2003) · ACM SIGGRAPH 2003 Papers · doi:10.1145/1201775.882269 · Cited by 2,020
Conditional Image Generation with PixelCNN Decoders
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 1,833
Pérez, P., Gangnet, M., & Blake, A. (2003) · ACM Transactions on Graphics · doi:10.1145/882262.882269 · Cited by 1,773
Pixel Recurrent Neural Networks
van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.06759 · Cited by 1,437
Synthesizing Obama: learning lip sync from audio
Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073640 · Cited by 1,057
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005
Photographic Image Synthesis with Cascaded Refinement Networks
Chen, Q., & Koltun, V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.168 · Cited by 887
Deep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801
Self-citationArandjelovic, R., & Zisserman, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.73 · Cited by 782
Self-citationShuffle and Learn: Unsupervised Learning Using Temporal Order Verification
Misra, I., Zitnick, C. L., & Hebert, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_32 · Cited by 779 · cited 1×
“Of more relevance is self-supervision from video, such asWangandGupta (2015), Fernandoet al. (2017),Misra et al. (2016),Xue et al. (2016), Pătrăucean et al. (2016) andDenton and Birodkar (2017).”
BackgroundDuke, D., & Scopigno, R. (2003) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2003.00715.x · Cited by 769
Lip reading sentences in the wild
Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.367 · Cited by 702 · cited 2×
“The difference in filter sizes can also be compensated by introducing some padding after the final layer, but since the results shown in Chung et al. (2017) have used these input and output dimensions, we chose to keep to these dimensions for a fair comparison.”
BackgroundSelf-citationOut of Time: Automated Lip Sync in the Wild
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54427-4_19 · Cited by 689 · cited 8×
“Active Speaker Detection and AV Synchronisation SyncNet (Chung and Zisserman 2016) provides a joint embedding of the audio and visual face sequences in a video, which can be used to determinewho is speaking in amulti-speaker video scene.”
InfluentialSelf-citationAudio-driven facial animation by joint end-to-end learning of pose and emotion
Karras, T., Aila, T., Laine, S., Herva, A., & Lehtinen, J. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073658 · Cited by 434
Self-Supervised Video Representation Learning with Odd-One-Out Networks
Fernando, B., Bilen, H., Gavves, E., & Gould, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.607 · Cited by 421
Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks
Xue, T., Wu, J., Bouman, K. L., & Freeman, W. T. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2854726 · Cited by 385
Owens, A., Isola, P., McDermott, J., Torralba, A., Adelson, E. H., & Freeman, W. T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.264 · Cited by 346
Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
Xue, T., Wu, J., Bouman, K. L., & Freeman, W. T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.02586 · Cited by 320
A deep learning approach for generalized speech animation
Taylor, S., Kim, T., Yue, Y., Mahler, M., Krahe, J., Rodriguez, A. G., Hodgins, J., & Matthews, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073699 · Cited by 303
RELIABLE TRANSITION DETECTION IN VIDEOS: A SURVEY AND PRACTITIONER'S GUIDE
LIENHART, R. (2001) · International Journal of Image and Graphics · doi:10.1142/s021946780100027x · Cited by 275
SoundNet: Learning Sound Representations from Unlabeled Video
Aytar, Y., Vondrick, C., & Torralba, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09001 · Cited by 222
Seeing Voices and Hearing Faces: Cross-Modal Biometric Matching
Nagrani, A., Albanie, S., & Zisserman, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00879 · Cited by 220
Self-citationSpatio-temporal video autoencoder with differentiable memory
Pătrăucean, V., Handa, A., & Cipolla, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06309 · Cited by 211
Unsupervised Learning of Disentangled Representations from Video
Denton, E., & Birodkar, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10915 · Cited by 205
Visual Speech Synthesis by Morphing Visemes
Ezzat, T., & Poggio, T. (2000) · International Journal of Computer Vision · doi:10.1023/a:1008166717597 · Cited by 140
VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track
Garrido, P., Valgaerts, L., Sarmadi, H., Steiner, I., Varanasi, K., Pérez, P., & Theobalt, C. (2015) · Computer Graphics Forum · doi:10.1111/cgf.12552 · Cited by 126
Photo-real talking head with deep bidirectional LSTM
Fan, B., Wang, L., Soong, F. K., & Xie, L. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178899 · Cited by 122
Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation
Chung, S.-W., Chung, J. S., & Kang, H.-G. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682524 · Cited by 106
Self-citationLearning visual groups from co-occurrences in space and time
Isola, P., Zoran, D., Krishnan, D., & Adelson, E. H. (2015) · arXiv · doi:10.48550/arxiv.1511.06811 · Cited by 90 · cited 1×
“The work on predicting co-occurrence [23], context [14], and colorization [50] fall under this category.”
BackgroundPHONEME-TO-VISEME MAPPING FOR VISUAL SPEECH RECOGNITION
Cappelletta, L., & Harte, N. (2012) · International Conference on Pattern Recognition Applications and Methods · doi:10.5220/0003731903220329 · Cited by 87
Chung, J. S., Jamaludin, A., & Zisserman, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.02966 · Cited by 58
Self-citationVirtual Immortality: Reanimating Characters from TV Shows
Charles, J., Magee, D., & Hogg, D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49409-8_71 · Cited by 11
Features and methods for improving large scale face recognition
Parkhi, O. (2015) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.5287/ora-r5jnrgega · Cited by 2
Research works printed but not linked: 1
- Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672-2680).
Cited by
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897
“This has led to the creation of speaker-independent speech to lip generation models [17, 18] that are trained on thousands of identities and voices.”
BackgroundMethodInfluentialThe Creation and Detection of Deepfakes: A Survey
Mirsky, Y., & Lee, W. (2022) · ACM Computing Surveys · doi:10.1145/3425780 · Cited by 863
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
Wang, T.-C., Mallya, A., & Liu, M.-Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00991 · Cited by 444
Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
Liu, M.-Y., Huang, X., Yu, J., Wang, T.-C., & Mallya, A. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3049196 · Cited by 236
Metadata
- Authors
- Jamaludin, A., Chung, J. S., & Zisserman, A.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Chung, Joon Son
- Venue
- International Journal of Computer Vision
- Published
- 2019-02-13
- Volume
- 127
- Issue
- 11-12
- Pages
- 1767-1779
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W2914217321
- Cited by
- 189
- References
- 54
- Citations by year
- 2026: 9 · 2025: 32 · 2024: 33 · 2023: 39 · 2022: 28 · 2021: 25 · 2020: 20 · 2019: 3
- FWCI
- 8.40
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- Locations
- International Journal of Computer Vision (publishedVersion, cc-by)
- Funders
- Engineering and Physical Sciences Research Council (EP/M013774/1)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- lip synchronization, encoder-decoder CNN, real-time video generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.