Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; none of the 22 sources its claims cite is checked for support yet.

Talking Face Generation by Adversarially Disentangled Audio-Visual Representation

Open access
Zhou, Hang, Liu, Yu, Liu, Ziwei, Luo, Ping, Wang, Xiaogang · 2019 · Proceedings of the AAAI Conference on Artificial Intelligence

Abstract

Talking face generation aims to synthesize a sequence of face images that correspond to a clip of speech. This is a challenging task because face appearance variation and semantics of speech are coupled together in the subtle movements of the talking face regions. Existing works either construct specific face appearance model on specific subjects or model the transformation between lip motion and speech. In this work, we integrate both aspects and enable arbitrary-subject talking face generation by learning disentangled audio-visual representation. We find that the talking face sequence is actually a composition of both subject-related information and speech-related information. These two spaces are then explicitly disentangled through a novel associative-and-adversarial training process. This disentangled representation has an advantage where both audio and video can serve as inputs for generation. Extensive experiments show that the proposed approach generates realistic talking face sequences on arbitrary subjects with much clearer lip motion patterns than previous work. We also demonstrate the learned audio-visual representation is extremely useful for the tasks of automatic lip reading and audio-video retrieval.

Overview

CoChat Summary

TL;DR

This paper proposes DAVS, an end-to-end talking-face generator that learns disentangled audio-visual representations so it can synthesize arbitrary subjects from either audio or video. On LRW, the learned representation improves lip-reading and retrieval, and the generation results are reported as more realistic and better synchronized than the compared baseline.

Key findings

  • On LRW, adding the full set of supervisions improved audio-visual speech recognition to 67.5% visual acc., 91.8% audio acc., and 95.2% combined acc.; the baseline from Chung and Zisserman (2016a) was 61.1% visual acc.
  • For audio-to-video retrieval, the full model reached R@1 67.7, R@10 85.8, MedR 1.0; for video-to-audio retrieval, it reached R@1 64.2, R@10 84.7, MedR 1.0.
  • In generation, replacing \(L_{L1}\) with the reconstruction loss \(L_{Re}\) improved PSNR/SSIM from 25.4/0.859 to 26.7/0.883 for audio-based generation, and from 25.7/0.865 to 26.8/0.884 for video-based generation.
  • In the user study, the reproduced baseline was rated 44.1% realistic and 58.0% lip-audio sync, while the proposed method scored 51.5% / 72.3% for audio guidance and 87.8% / 88.4% for video guidance.

Methods

  • The paper introduces Disentangled Audio-Visual System (DAVS), which learns a shared Word-ID (wid) audio-visual embedding and disentangles it from a Person-ID (pid) space using adversarial training.
  • Data/sample: training and evaluation use the LRW dataset (described as 1-of-500 word labels, with more than 800 training samples and 50 validation/test samples per class); the identity-preserving module is trained on a subset of MS-Celeb-1M. The user study uses 10 celebrity photos and clips from LRW and VoxCeleb.
  • Main analysis: experiments report PSNR/SSIM, audio-video retrieval at R@1/R@10/MedR, audio-visual speech recognition accuracy, and an ablation measuring how much word-ID information remains in the person-ID branch via an SVM.

Limitations

  • The paper notes that, because different subjects may move their lips differently for the same audio clip, the generated result is not a one-to-one mapping to ground truth, so final outputs may differ from the exact target motion.
  • It also states that the directly generated results may suffer from a scale changing problem due to alignment in the training set, and therefore applies subspace video stabilization for smoothness.
  • Evident from the text: the method is evaluated primarily on word-level LRW clips and 10 participants in the user study, so the reported evaluation scope is relatively narrow.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1609/aaai.v33i01.33019299

Cites

33 of 33 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 2 Not valid · 14 Not checked · 29 Own results and interpretations

21 more claims

Create a free account to see all 45 claims and to verify the 22 references still open.

See all claims — free account
  • It may state two separate claims as one. 13
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 10

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 33 references

Research works printed but not linked: 3

  1. Chung, J. S.; Jamaludin, A.; and Zisserman, A. 2017. You said that? In BMVC .
  2. Fan, B.; Wang, L.; Soong, F. K.; and Xie, L. 2015. Photoreal talking head with deep bidirectional lstm. In ICASSP .
  3. Liu, Y.; Song, G.; Shao, J.; Jin, X.; and Wang, X. 2018a. Transductive centroid projection for semi-supervised largescale recognition. In ECCV .

Cited by

See all 416

Metadata

Authors
Zhou, H., Liu, Y., Liu, Z., Luo, P., & Wang, X.
Affiliations
Chinese University of Hong Kong (HK)
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Published
2019-07-17
Volume
33
Issue
01
Pages
9299-9306
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
OpenAlex
W2963290645
Cited by
416
References
33
Citations by year
2026: 6 · 2025: 33 · 2024: 65 · 2023: 97 · 2022: 71 · 2021: 78 · 2020: 49 · 2019: 16
FWCI
17.45
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
Keywords
talking face generation, disentangled representation learning, adversarial training, audiovisual representation, lip movement, automatic lip reading

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Talking Face Generation by Adversarially Disentangled Audio-Visual Representation. https://app.cochat.ai/papers/10.1609/aaai.v33i01.33019299
The paper
Zhou, H., Liu, Y., Liu, Z., Luo, P., & Wang, X. (2019). Talking Face Generation by Adversarially Disentangled Audio-Visual Representation. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v33i01.33019299

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.