Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index; 2 of the 22 sources its claims cite were checked for support.

A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild

Open access
Prajwal, K R, Mukhopadhyay, Rudrabha, Namboodiri, Vinay P., Jawahar, C.V.·2020·Proceedings of the 28th ACM International Conference on Multimedia

Abstract

In this work, we investigate the problem of lip-syncing a talking face video of an arbitrary identity to match a target speech segment. Current works excel at producing accurate lip movements on a static image or videos of specific people seen during the training phase. However, they fail to accurately morph the lip movements of arbitrary identities in dynamic, unconstrained talking face videos, resulting in significant parts of the video being out-of-sync with the new audio. We identify key reasons pertaining to this and hence resolve them by learning from a powerful lip-sync discriminator. Next, we propose new, rigorous evaluation benchmarks and metrics to accurately measure lip synchronization in unconstrained videos. Extensive quantitative evaluations on our challenging benchmarks show that the lip-sync accuracy of the videos generated by our Wav2Lip model is almost as good as real synced videos. We provide a demo video clearly showing the substantial impact of our Wav2Lip model, and also publicly release the code, models, and evaluation benchmarks on our website.

Overview

CoChat Summary

TL;DR

Wav2Lip lip-syncs arbitrary talking-face videos in the wild by training a generator against a pretrained, accurate lip-sync expert discriminator rather than a weak GAN discriminator. It reports lip-sync accuracy almost as good as real synced videos and introduces new benchmarks and metrics for unconstrained evaluation.

Key findings

  • On LRS2, Wav2Lip reaches LSE-D 6.386 and LSE-C 7.789, close to real videos (6.736, 7.838) and much better than LipGAN (10.33, 3.199) and Speech2Vid (14.23, 1.587).
  • The expert lip-sync discriminator is 91% accurate on the LRS2 test set versus 56% for LipGAN’s discriminator; in ablation, the frozen T_v=5 expert reaches 91.6% off-sync accuracy with LSE-D 6.386/LSE-C 7.789, while fine-tuning it drops to 73.6% and LSE-D 9.953/LSE-C 3.508.
  • On ReSyncED, Wav2Lip models are preferred over existing methods or unsynced videos more than 90% of the time; adding the visual quality discriminator improves FID but causes a minor drop in sync accuracy.
  • The paper claims Wav2Lip is the first speaker-independent model to generate videos with lip-sync accuracy matching real synced videos.

Methods

  • Generator follows a LipGAN-like architecture with identity encoder, speech encoder, and face decoder; trained with L1 reconstruction loss, expert sync loss from a frozen pretrained lip-sync discriminator, and adversarial loss from a visual quality discriminator, with sync weight 0.03 and adversarial weight 0.07.
  • The expert discriminator modifies SyncNet using color images, deeper residual skip connections, and cosine-similarity with binary cross-entropy; trained on LRS2 train split (~29 hours), batch size 64, T_v=5, Adam with initial learning rate 1e-3.
  • Evaluation uses new LSE-D/LSE-C metrics from public SyncNet, benchmarks built from LRS2 (14K pairs), LRW (28K pairs), and LRS3 (14K pairs), plus ReSyncED real-world YouTube videos in Dubbed, Random, and TTS categories, with human evaluation by 14 evaluators.

Limitations

  • Stated in the paper: human evaluations indicate scope for improvement when lip-syncing TTS-generated speech, and using the visual quality discriminator causes a minor drop in sync accuracy, creating a trade-off between visual quality and sync accuracy.
  • Evident from the text: the model is trained only on the LRS2 train set, and the real-world benchmark is collected from YouTube with manually selected video types and synthetic translated speech.
  • Evident from the text: the human evaluation uses 14 evaluators, and the automatic lip-sync metrics rely on a public SyncNet model trained on a different, non-public dataset.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3394171.3413532

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 2 Not valid · 13 Not checked · 59 Own results and interpretations

54 more claims

Create a free account to see all 76 claims and to verify the 20 references still open.

See all claims — free account
  • It may state two separate claims as one. 10
  • Probably not a substantial claim. 9

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 31 references

Research works printed but not linked: 3

  1. Prajwal KR , Rudrabha Mukhopadhyay , Jerin Philip , Abhishek Jha , Vinay Namboodiri , and CV Jawahar . 2019 . Towards Automatic Face-to-Face Translation . In Proceedings of the 27th ACM International Conference on Multimedia. ACM, 1428--1436 . Prajwal KR, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, and CV Jawahar. 2019. Towards Automatic Face-to-Face Translation. In Proceedings of the 27th ACM International Conference on Multimedia. ACM, 1428--1436.
  2. Andrew L Maas , Awni Y Hannun , and Andrew Y Ng . 2013 . Rectifier nonlinearities improve neural network acoustic models . In Proc. icml , Vol. 30 . 3. Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, Vol. 30. 3.
  3. NPD. 2016. 52 Percent of Millennial Smartphone Owners Use their Device for Video Calling According to The NPD Group. https://www.npd.com/wps/portal/npd/us/news/press-releases/2016/52-percent-of-millennial-smartphone-owners-use-their-device-for-video-calling-according-to-the-npd-group/ NPD. 2016. 52 Percent of Millennial Smartphone Owners Use their Device for Video Calling According to The NPD Group. https://www.npd.com/wps/portal/npd/us/news/press-releases/2016/52-percent-of-millennial-smartphone-owners-use-their-device-for-video-calling-according-to-the-npd-group/

Cited by

See all 897

Metadata

Authors
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V.
Affiliations
  • International Institute of Information Technology, Hyderabad (IN)
  • University of Bath (GB)
Venue
Proceedings of the 28th ACM International Conference on Multimedia
Publisher
ACM
Published
2020-10-12
Pages
484-492
OpenAlex
W3081492798
Cited by
897
References
31
Citations by year
2026: 157 · 2025: 235 · 2024: 211 · 2023: 166 · 2022: 75 · 2021: 50 · 2020: 3
FWCI
45.95
Type
conference paper
Language
en
Open access
Locations
  • ACM International Conference on Multimedia (ACM MM) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Pure (University of Bath) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
Keywords
lip synchronization, unconstrained video, evaluation benchmark, audio-video synchronization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild. https://app.cochat.ai/papers/10.1145/3394171.3413532
The paper
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020). A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild. Proceedings of the 28th ACM International Conference on Multimedia. https://doi.org/10.1145/3394171.3413532

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.