CoChat

A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild

Open access
Prajwal, K R, Mukhopadhyay, Rudrabha, Namboodiri, Vinay P., Jawahar, C.V. · 2020 · Proceedings of the 28th ACM International Conference on Multimedia

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 2 of the 22 sources its claims cite were checked for support.

Abstract

In this work, we investigate the problem of lip-syncing a talking face video of an arbitrary identity to match a target speech segment. Current works excel at producing accurate lip movements on a static image or videos of specific people seen during the training phase. However, they fail to accurately morph the lip movements of arbitrary identities in dynamic, unconstrained talking face videos, resulting in significant parts of the video being out-of-sync with the new audio. We identify key reasons pertaining to this and hence resolve them by learning from a powerful lip-sync discriminator. Next, we propose new, rigorous evaluation benchmarks and metrics to accurately measure lip synchronization in unconstrained videos. Extensive quantitative evaluations on our challenging benchmarks show that the lip-sync accuracy of the videos generated by our Wav2Lip model is almost as good as real synced videos. We provide a demo video clearly showing the substantial impact of our Wav2Lip model, and also publicly release the code, models, and evaluation benchmarks on our website.

Overview

CoChat Summary

TL;DR

Wav2Lip lip-syncs arbitrary talking-face videos in the wild by training a generator against a pretrained, accurate lip-sync expert discriminator rather than a weak GAN discriminator. It reports lip-sync accuracy almost as good as real synced videos and introduces new benchmarks and metrics for unconstrained evaluation.

Key findings

  • On LRS2, Wav2Lip reaches LSE-D 6.386 and LSE-C 7.789, close to real videos (6.736, 7.838) and much better than LipGAN (10.33, 3.199) and Speech2Vid (14.23, 1.587).
  • The expert lip-sync discriminator is 91% accurate on the LRS2 test set versus 56% for LipGAN’s discriminator; in ablation, the frozen T_v=5 expert reaches 91.6% off-sync accuracy with LSE-D 6.386/LSE-C 7.789, while fine-tuning it drops to 73.6% and LSE-D 9.953/LSE-C 3.508.
  • On ReSyncED, Wav2Lip models are preferred over existing methods or unsynced videos more than 90% of the time; adding the visual quality discriminator improves FID but causes a minor drop in sync accuracy.
  • The paper claims Wav2Lip is the first speaker-independent model to generate videos with lip-sync accuracy matching real synced videos.

Methods

  • Generator follows a LipGAN-like architecture with identity encoder, speech encoder, and face decoder; trained with L1 reconstruction loss, expert sync loss from a frozen pretrained lip-sync discriminator, and adversarial loss from a visual quality discriminator, with sync weight 0.03 and adversarial weight 0.07.
  • The expert discriminator modifies SyncNet using color images, deeper residual skip connections, and cosine-similarity with binary cross-entropy; trained on LRS2 train split (~29 hours), batch size 64, T_v=5, Adam with initial learning rate 1e-3.
  • Evaluation uses new LSE-D/LSE-C metrics from public SyncNet, benchmarks built from LRS2 (14K pairs), LRW (28K pairs), and LRS3 (14K pairs), plus ReSyncED real-world YouTube videos in Dubbed, Random, and TTS categories, with human evaluation by 14 evaluators.

Limitations

  • Stated in the paper: human evaluations indicate scope for improvement when lip-syncing TTS-generated speech, and using the visual quality discriminator causes a minor drop in sync accuracy, creating a trade-off between visual quality and sync accuracy.
  • Evident from the text: the model is trained only on the LRS2 train set, and the real-world benchmark is collected from YouTube with manually selected video types and synthetic translated speech.
  • Evident from the text: the human evaluation uses 14 evaluators, and the automatic lip-sync metrics rely on a public SyncNet model trained on a different, non-public dataset.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3394171.3413532

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 2 Not valid · 13 Not checked · 59 Own results and interpretations

54 more claims

Create a free account to see all 76 claims and to verify the 20 references still open.

See all claims — free account
  • It may state two separate claims as one. 10
  • Probably not a substantial claim. 9

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

  • Deep Residual Learning for Image Recognition

    He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×

    “ask. Firstly, instead of feeding grayscale images concatenated channel-wise as in the original model, we feed color images. Secondly, our model is significantly deeper, with residual skip connections [15]. Thirdly, inspired by this public implementation2, we use a different loss function: cosine-similarity with binary…”
    Method
  • Image quality assessment: from error visibility to structural similarity

    Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · cited 1×

    “nnot support a new metric or a future method that aims to study the temporal consistency aspect of this problem. 4.1.4 Current metrics are not specific to lip-sync. The existing metrics, such as SSIM [27] and PSNR, were developed to evaluate overall image quality and not fine-grained lip-sync errors. Although LMD [4]…”
    MethodInfluential
  • Untitled work

    (2010) · Cited by 8,261

  • Instance Normalization: The Missing Ingredient for Fast Stylization

    Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.08022 · Cited by 2,775 · cited 1×

    “Each block consists of a convolutional layer followed by an Instance Normalization layer [26] and Leaky ReLU activation [20].”
    Method
  • International Journal of Computer Vision

    (2013) · International Journal of Computer Vision · doi:10.1007/11263.1573-1405 · Cited by 2,340

  • DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection

    Tolosana, R., Vera-Rodríguez, R., Fierrez, J., Morales, A., & Ortega-García, J. (2020) · Information Fusion · doi:10.48550/arxiv.2001.00179 · Cited by 1,252 · cited 1×

    “In addition to the strong positive impact mentioned above, our intention to completely open-source our work is that it can simultaneously also encourage efforts [11, 16, 24, 25] in detecting manipulated video content and their misuse.”
    Background
  • Deepfakes and beyond: A Survey of face manipulation and fake detection

    Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., & Ortega-Garcia, J. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.014 · Cited by 1,237

  • An audio-visual corpus for speech perception and automatic speech recognition

    Cooke, M., Barker, J., Cunningham, S., & Shao, X. (2006) · The Journal of the Acoustical Society of America · doi:10.1121/1.2229005 · Cited by 1,144 · cited 1×

    “Several works [5, 26, 28] train on datasets with a limited set of words such as GRID [10] (56 words), TIMIT [14] and LRW [8] (1000 words) which significantly hampers a model from learning the vast diversity of phoneme-viseme mappings in real videos [18].”
    Background
  • Synthesizing Obama: learning lip sync from audio

    Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073640 · Cited by 1,057 · cited 2×

    “that are either constrained by the range of identities they can generate or the range of vocabulary they are limited to. Realistic generation of talking face videos was achieved by a few recent works [19, 22] on videos of Barack Obama. They learn a mapping between the input audio 1cvit.iiit.ac.in/research/projects/cvi…”
    BackgroundMethod
  • Deep Audio-Visual Speech Recognition

    Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801 · cited 5×

    “ideos. Also, it should not be fine-tuned further on the generated frames like it is done in LipGAN. One such network that has been used to correct lip-sync errors for creating large lip-sync datasets [1, 3] is the SyncNet [9] model. We propose to adapt and train a modified version of SyncNet [9] for our task. 3.3.1 Ov…”
    BackgroundMethodInfluential
  • Out of Time: Automated Lip Sync in the Wild

    Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54427-4_19 · Cited by 689 · cited 10×

    “We propose to use the pre-trained SyncNet [9] available publicly3 to measure the lip-sync error between the generated frames and the randomly chosen speech segment.”
    BackgroundMethodInfluential
  • Lip Reading in the Wild

    Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537 · cited 3×

    “t speaker. Another limitation of existing works is in terms of the vocabulary. Several works [5, 26, 28] train on datasets with a limited set of words such as GRID [10] (56 words), TIMIT [14] and LRW [8] (1000 words) which significantly hampers a model from learning the vast diversity of phoneme-viseme mappings in rea…”
    BackgroundMethodInfluential
  • Neural Voice Puppetry: Audio-driven Facial Reenactment

    Thies, J., Elgharib, M. A., Tewari, A., Theobalt, C., & Niessner, M. (2019) · European Conference on Computer Vision · doi:10.48550/arxiv.1912.05566 · Cited by 464 · cited 3×

    “More recent works [13, 23] in this line directly generate images from speech representations and show exceptional generation quality for specific speakers which they have been trained upon.”
    Background
  • Hierarchical Cross-Modal Talking Face Generation With Dynamic Pixel-Wise Loss

    Chen, L., Maddox, R. K., Duan, Z., & Xu, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00802 · Cited by 437 · cited 1×

    “lso have an additional overhead of requiring clean training data of each target speaker to generate for that speaker. Another limitation of existing works is in terms of the vocabulary. Several works [5, 26, 28] train on datasets with a limited set of words such as GRID [10] (56 words), TIMIT [14] and LRW [8] (1000 wo…”
    Background
  • The deepfake detection challenge (dfdc) dataset

    Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., & Ferrer, C. C. (2020) · arXiv · doi:10.48550/arxiv.2006.07397 · Cited by 427 · cited 1×

    “In addition to the strong positive impact mentioned above, our intention to completely open-source our work is that it can simultaneously also encourage efforts [11, 16, 24, 25] in detecting manipulated video content and their misuse.”
    Background
  • Talking Face Generation by Adversarially Disentangled Audio-Visual Representation

    Zhou, H., Liu, Y., Liu, Z., Luo, P., & Wang, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019299 · Cited by 416

  • Neural Voice Puppetry: Audio-Driven Facial Reenactment

    Thies, J., Elgharib, M., Tewari, A., Theobalt, C., & Nießner, M. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58517-4_42 · Cited by 372

  • Deep fake image detection based on pairwise learning

    Hsu, C.-C., Zhuang, Y.-X., & Lee, C.-Y. (2020) · Applied Sciences · doi:10.3390/app10010370 · Cited by 304 · cited 1×

    “In addition to the strong positive impact mentioned above, our intention to completely open-source our work is that it can simultaneously also encourage efforts [11, 16, 24, 25] in detecting manipulated video content and their misuse.”
    Background
  • Realistic Speech-Driven Facial Animation with GANs

    Vougioukas, K., Petridis, S., & Pantic, M. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01251-8 · Cited by 298 · cited 1×

    “lso have an additional overhead of requiring clean training data of each target speaker to generate for that speaker. Another limitation of existing works is in terms of the vocabulary. Several works [5, 26, 28] train on datasets with a limited set of words such as GRID [10] (56 words), TIMIT [14] and LRW [8] (1000 wo…”
    Background
  • Text-based Editing of Talking-head Video

    Fried, O., Tewari, A., Zollhöfer, M., Finkelstein, A., Shechtman, E., Goldman, D. B., Genova, K., Jin, Z., Theobalt, C., & Agrawala, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323028 · Cited by 270 · cited 3×

    “More recent works [13, 23] in this line directly generate images from speech representations and show exceptional generation quality for specific speakers which they have been trained upon.”
    Background
  • LRS3-TED: a large-scale dataset for visual speech recognition

    Afouras, T., Chung, J. S., & Zisserman, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00496 · Cited by 262 · cited 3×

    “p region, we found that lip landmarks can be quite inaccurate on generated faces. Thus, there is a need for a metric that is designed specifically for measuring lip-sync errors. LRW [8] LRS2 [1] LRS3 [3] Method LSE-D ↓ LSE-C ↑ FID ↓ LSE-D ↓ LSE-C ↑ FID ↓ LSE-D ↓ LSE-C ↑ FID ↓ Speech2Vid [17] 13.14 1.762 11.15 14.23 1.…”
    BackgroundMethodInfluential
  • Lip Movements Generation at a Glance

    Chen, L., Li, Z., Maddox, R. K., Duan, Z., & Xu, C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_32 · Cited by 258

  • TCD-TIMIT: An Audio-Visual Corpus of Continuous Speech

    Harte, N., & Gillen, E. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2407694 · Cited by 250 · cited 1×

    “erate for that speaker. Another limitation of existing works is in terms of the vocabulary. Several works [5, 26, 28] train on datasets with a limited set of words such as GRID [10] (56 words), TIMIT [14] and LRW [8] (1000 words) which significantly hampers a model from learning the vast diversity of phoneme-viseme ma…”
    Background
  • Untitled work

    (2020) · arXiv (Cornell University) · Cited by 204

  • You Said That?: Synthesising Talking Faces from Audio

    Jamaludin, A., Chung, J. S., & Zisserman, A. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01150-y · Cited by 189 · cited 9×

    “This has led to the creation of speaker-independent speech to lip generation models [17, 18] that are trained on thousands of identities and voices.”
    BackgroundMethodInfluential
  • ObamaNet: Photo-realistic lip-sync from text

    Kumar, R., Sotelo, J., Kumar, K., de Brébisson, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.01442 · Cited by 74 · cited 2×

    “Realistic generation of talking face videos was achieved by a few recent works [19, 22] on videos of Barack Obama.”
    BackgroundMethod
  • You said that?

    Chung, J. S., Jamaludin, A., & Zisserman, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.02966 · Cited by 58

  • Talking Face Generation by Adversarially Disentangled Audio-Visual RepresentationPreprint

    Zhou, H., Liu, Y., Liu, Z., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.07860 · Cited by 37 · cited 1×

    “Several works [5, 26, 28] train on datasets with a limited set of words such as GRID [10] (56 words), TIMIT [14] and LRW [8] (1000 words) which significantly hampers a model from learning the vast diversity of phoneme-viseme mappings in real videos [18].”
    Background
  • Towards Untrusted Social Video Verification to Combat Deepfakes via Face Geometry Consistency

    Tursman, E., George, M., Kamara, S., & Tompkin, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00335 · Cited by 19 · cited 1×

    “biguously present itself as synthetic. In addition to the strong positive impact mentioned above, our intention to completely open-source our work is that it can simultaneously also encourage efforts [11, 16, 24, 25] in detecting manipulated video content and their misuse. We believe that Wav2Lip can enable severalpos…”
    Background
  • Sound to Visual: Hierarchical Cross-Modal Talking Face Generation

    Chen, L., Zheng, H., Maddox, R. K., Duan, Z., & Xu, C. (2019) · Cited by 4

  • Deep Audio-Visual Speech Recognition

    Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2018) · arXiv · doi:10.48550/arxiv.1809.02108 · cited 5×

    “ideos. Also, it should not be fine-tuned further on the generated frames like it is done in LipGAN. One such network that has been used to correct lip-sync errors for creating large lip-sync datasets [1, 3] is the SyncNet [9] model. We propose to adapt and train a modified version of SyncNet [9] for our task. 3.3.1 Ov…”
    BackgroundMethodInfluential

Research works printed but not linked: 4

  1. [2] T. Afouras, J. S. Chung, and A. Zisserman. 2018. The Conversation: Deep AudioVisual Speech Enhancement. In INTERSPEECH . -
  2. [18] Prajwal KR, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, and CV Jawahar. 2019. Towards Automatic Face-to-Face Translation. In Proceedings of the 27th ACM International Conference on Multimedia . ACM, 1428-1436. -
  3. [20] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml , Vol. 30. 3. -
  4. [21] [NPD. 2016. 52 Percent of Millennial Smartphone Owners Use their Device for Video Calling, According to The NPD Group . https://www.npd.com/wps/portal/npd/us/ news/press-releases/2016/52-percent-of-millennial-smartphone-owners-usetheir-device-for-video-calling-according-to-the-npd-group/](https://www.npd.com/wps/portal/npd/us/news/press-releases/2016/52-percent-of-millennial-smartphone-owners-use-their-device-for-video-calling-according-to-the-npd-group/) -

Cited by

See all 897

Metadata

Authors
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V.
Affiliations
  • International Institute of Information Technology, Hyderabad (IN)
  • University of Bath (GB)
Venue
Proceedings of the 28th ACM International Conference on Multimedia
Publisher
ACM
Published
2020-10-12
Pages
484-492
OpenAlex
W3081492798
Cited by
897
References
31
Citations by year
2026: 157 · 2025: 235 · 2024: 211 · 2023: 166 · 2022: 75 · 2021: 50 · 2020: 3
FWCI
45.95
Type
conference paper
Language
en
Open access
Locations
  • ACM International Conference on Multimedia (ACM MM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Pure (University of Bath) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
Keywords
lip synchronization, unconstrained video, evaluation benchmark, audio-video synchronization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.