The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index; none of the 22 sources its claims cite is checked for support yet.
Talking Face Generation by Adversarially Disentangled Audio-Visual Representation
Open accessAbstract
Talking face generation aims to synthesize a sequence of face images that correspond to a clip of speech. This is a challenging task because face appearance variation and semantics of speech are coupled together in the subtle movements of the talking face regions. Existing works either construct specific face appearance model on specific subjects or model the transformation between lip motion and speech. In this work, we integrate both aspects and enable arbitrary-subject talking face generation by learning disentangled audio-visual representation. We find that the talking face sequence is actually a composition of both subject-related information and speech-related information. These two spaces are then explicitly disentangled through a novel associative-and-adversarial training process. This disentangled representation has an advantage where both audio and video can serve as inputs for generation. Extensive experiments show that the proposed approach generates realistic talking face sequences on arbitrary subjects with much clearer lip motion patterns than previous work. We also demonstrate the learned audio-visual representation is extremely useful for the tasks of automatic lip reading and audio-video retrieval.
Overview
CoChat Summary
TL;DR
This paper proposes DAVS, an end-to-end talking-face generator that learns disentangled audio-visual representations so it can synthesize arbitrary subjects from either audio or video. On LRW, the learned representation improves lip-reading and retrieval, and the generation results are reported as more realistic and better synchronized than the compared baseline.
Key findings
- On LRW, adding the full set of supervisions improved audio-visual speech recognition to 67.5% visual acc., 91.8% audio acc., and 95.2% combined acc.; the baseline from Chung and Zisserman (2016a) was 61.1% visual acc.
- For audio-to-video retrieval, the full model reached R@1 67.7, R@10 85.8, MedR 1.0; for video-to-audio retrieval, it reached R@1 64.2, R@10 84.7, MedR 1.0.
- In generation, replacing \(L_{L1}\) with the reconstruction loss \(L_{Re}\) improved PSNR/SSIM from 25.4/0.859 to 26.7/0.883 for audio-based generation, and from 25.7/0.865 to 26.8/0.884 for video-based generation.
- In the user study, the reproduced baseline was rated 44.1% realistic and 58.0% lip-audio sync, while the proposed method scored 51.5% / 72.3% for audio guidance and 87.8% / 88.4% for video guidance.
Methods
- The paper introduces Disentangled Audio-Visual System (DAVS), which learns a shared Word-ID (wid) audio-visual embedding and disentangles it from a Person-ID (pid) space using adversarial training.
- Data/sample: training and evaluation use the LRW dataset (described as 1-of-500 word labels, with more than 800 training samples and 50 validation/test samples per class); the identity-preserving module is trained on a subset of MS-Celeb-1M. The user study uses 10 celebrity photos and clips from LRW and VoxCeleb.
- Main analysis: experiments report PSNR/SSIM, audio-video retrieval at R@1/R@10/MedR, audio-visual speech recognition accuracy, and an ablation measuring how much word-ID information remains in the person-ID branch via an SVM.
Limitations
- The paper notes that, because different subjects may move their lips differently for the same audio clip, the generated result is not a one-to-one mapping to ground truth, so final outputs may differ from the exact target motion.
- It also states that the directly generated results may suffer from a scale changing problem due to alignment in the training set, and therefore applies subspace video stabilization for smoothness.
- Evident from the text: the method is evaluated primarily on word-level LRW clips and 10 participants in the user study, so the reported evaluation scope is relatively narrow.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 14 Not checked · 29 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Extensive experiments show that the proposed approach generates realistic talking face sequences on arbitrary subjects with much clearer lip motion patterns than previous work.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also demonstrate the learned audio-visual representation is extremely useful for the tasks of automatic lip reading and audio-video retrieval.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Experiments show that the jointembedding improves the baseline of lip reading result on LRW dataset (Chung and Zisserman 2016a).”
What the sources say
- Not checked yetcites Chung and Zisserman 2016a
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we can see in Table 3, with all supervisions, the highest results can be achieved.”
plainstates a causen = 25,000- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Combining with the adversarial training makes the image quality improves.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
21 more claims
Create a free account to see all 45 claims and to verify the 22 references still open.
- It may state two separate claims as one. 13
- The quote only partly states this claim. 1
- Probably not a substantial claim. 10
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, DAVS relies on extra Word-ID labels and Person-ID labels in the training stage.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [72]
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747
Self-citationLearning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
Dlib-ml: A Machine Learning Toolkit
King, D. E. (2009) · doi:10.5555/1577069.1755843 · Cited by 2,777
VoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210
MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition
Guo, Y., Zhang, L., Hu, Y., He, X., & Gao, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_6 · Cited by 2,122
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Bulat, A., & Tzimiropoulos, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.116 · Cited by 1,464
Video Frame Synthesis Using Deep Voxel Flow
Liu, Z., Yeh, R. A., Tang, X., Liu, Y., & Agarwala, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.478 · Cited by 743
Self-citation
Research works printed but not linked: 3
- Chung, J. S.; Jamaludin, A.; and Zisserman, A. 2017. You said that? In BMVC .
- Fan, B.; Wang, L.; Soong, F. K.; and Xie, L. 2015. Photoreal talking head with deep bidirectional lstm. In ICASSP .
- Liu, Y.; Song, G.; Shao, J.; Jin, X.; and Wang, X. 2018a. Transductive centroid projection for semi-supervised largescale recognition. In ECCV .
Cited by
Deepfakes and beyond: A Survey of face manipulation and fake detection
Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., & Ortega-Garcia, J. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.014 · Cited by 1,237
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897
The Creation and Detection of Deepfakes: A Survey
Mirsky, Y., & Lee, W. (2022) · ACM Computing Surveys · doi:10.1145/3425780 · Cited by 863
The Creation and Detection of Deepfakes: A Survey
Mirsky, Y., & Lee, W. (2020) · arXiv · doi:10.48550/arxiv.2004.11138 · Cited by 863
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543
Metadata
- Authors
- Zhou, H., Liu, Y., Liu, Z., Luo, P., & Wang, X.
- Affiliations
- Chinese University of Hong Kong (HK)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2019-07-17
- Volume
- 33
- Issue
- 01
- Pages
- 9299-9306
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2963290645
- Semantic Scholar
- 1816f98e2a4dd54690c2689cf529699d8843e847
- Cited by
- 416
- References
- 33
- Citations by year
- 2026: 6 · 2025: 33 · 2024: 65 · 2023: 97 · 2022: 71 · 2021: 78 · 2020: 49 · 2019: 16
- FWCI
- 17.45
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- talking face generation, disentangled representation learning, adversarial training, audiovisual representation, lip movement, automatic lip reading
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Talking Face Generation by Adversarially Disentangled Audio-Visual Representation. https://app.cochat.ai/papers/10.1609/aaai.v33i01.33019299
- The paper
- Zhou, H., Liu, Y., Liu, Z., Luo, P., & Wang, X. (2019). Talking Face Generation by Adversarially Disentangled Audio-Visual Representation. Proceedings of the AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai.v33i01.33019299
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.