CoChat
Sign Up
Cited by 88Generative Adversarial Networks and Image SynthesisSpeech and Audio ProcessingSpeech Recognition and Synthesis

SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory

Open access
Park, Se Jin, Kim, Minsu, Hong, Joanna, Choi, Jeongsoo, Ro, Yong Man·2022·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v36i2.20102
Open this paper in CoChat — free account

Abstract

The challenge of talking face generation from speech lies in aligning two different modal information, audio and video, such that the mouth region corresponds to input audio. Previous methods either exploit audio-visual representation learning or leverage intermediate structural information such as landmarks and 3D models. However, they struggle to synthesize fine details of the lips varying at the phoneme level as they do not sufficiently provide visual information of the lips at the video synthesis step. To overcome this limitation, our work proposes Audio-Lip Memory that brings in visual information of the mouth region corresponding to input audio and enforces fine-grained audio-visual coherence. It stores lip motion features from sequential ground truth images in the value memory and aligns them with corresponding audio features so that they can be retrieved using audio input at inference time. Therefore, using the retrieved lip motion features as visual hints, it can easily correlate audio with visual dynamics in the synthesis step. By analyzing the memory, we demonstrate that unique lip features are stored in each memory slot at the phoneme level, capturing subtle lip motion based on memory addressing. In addition, we introduce visual-visual synchronization loss which can enhance lip-syncing performance when used along with audio-visual synchronization loss in our model. Extensive experiments are performed to verify that our method generates high-quality video with mouth shapes that best align with the input audio, outperforming previous state-of-the-art methods.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v36i2.20102

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SyncTalkFace ensures fine-grained audio-visual coherence by aligning visual information of the mouth region with input audio.

    “SyncTalkFace (Park et al., 2022) introduces an Audio-Lip Memory mechanism to align visual information of the mouth region with input audio, ensuring fine-grained audio-visual coherence.”

    From The AI Hippocampus: How Far are We From Human Memory? (2026) · cites this paper as Park et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 36

  1. Afouras, T.; Chung, J. S.; Senior, A.; Vinyals, O.; and Zisserman, A. 2018. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence .
  2. Cai, Q.; Pan, Y.; Yao, T.; Yan, C.; and Mei, T. 2018. Memory matching networks for one-shot image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , 4080-4088.
  3. Chen, L.; Li, Z.; Maddox, R. K.; Duan, Z.; and Xu, C. 2018. Lip movements generation at a glance. In Proceedings of the European Conference on Computer Vision (ECCV) , 520535.
  4. Chen, L.; Maddox, R. K.; Duan, Z.; and Xu, C. 2019. Hierarchical cross-modal talking face generation with dynamic pixel-wise loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 78327841.
  5. Chung, J. S.; and Zisserman, A. 2016a. Lip reading in the wild. In Asian conference on computer vision , 87-103. Springer.
  6. Chung, S.-W.; Chung, J. S.; and Kang, H.-G. 2019. Perfect match: Improved cross-modal embeddings for audiovisual synchronisation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 3965-3969. IEEE.
  7. Das, D.; Biswas, S.; Sinha, S.; and Bhowmick, B. 2020. Speech-driven facial animation using cascaded gans for learning of motion and texture. In European Conference on Computer Vision , 408-424. Springer.
  8. Gao, R.; and Grauman, K. 2021. Visualvoice: Audio-visual speech separation with cross-modal consistency. arXiv preprint arXiv:2101.03149 . · arXiv
  9. Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative adversarial nets. Advances in neural information processing systems , 27.
  10. Jamaludin, A.; Chung, J. S.; and Zisserman, A. 2019. You said that?: Synthesising talking faces from audio. International Journal of Computer Vision , 127(11): 1767-1779.
  11. Kaiser, Ł.; Nachum, O.; Roy, A.; and Bengio, S. 2017. Learning to remember rare events. arXiv preprint arXiv:1703.03129 . · arXiv
  12. Kim, J. U.; Park, S.; and Ro, Y. M. 2021. Robust SmallScale Pedestrian Detection With Cued Recall via Memory Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 3050-3059.
  13. Kim, M.; Hong, J.; Park, S. J.; and Ro, Y. M. 2021. Multimodality associative bridging through memory: Speech sound recollected from face video. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 296-306.
  14. Kim, S.; and Ro, Y. M. 2021. M-CAM: Visual Explanation of Challenging Conditioned Dataset with Bias-reducing Memory. In The 32nd British Machine Vision Conference,
  15. BMVC 2021 . British Machine Vision Association (BMVA).
  16. King, D. E. 2009. Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research , 10: 1755-1758.
  17. KR, P.; Mukhopadhyay, R.; Philip, J.; Jha, A.; Namboodiri, V.; and Jawahar, C. 2019. Towards automatic face-to-face translation. In Proceedings of the 27th ACM International Conference on Multimedia , 1428-1436.
  18. Lee, S.; Kim, H. G.; Choi, D. H.; Kim, H.-I.; and Ro, Y. M. 2021. Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 3054-3063.
  19. Lee, S.; Sung, J.; Yu, Y.; and Kim, G. 2018. A memory network approach for story-based temporal summarization of 360 videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 1410-1419.
  20. Miller, A.; Fisch, A.; Dodge, J.; Karimi, A.-H.; Bordes, A.; and Weston, J. 2016. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126 . · arXiv
  21. Mittal, G.; and Wang, B. 2020. Animating face using disentangled audio representations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 3290-3298.
  22. Nagrani, A.; Chung, J. S.; Albanie, S.; and Zisserman, A. 2020. Disentangled speech embeddings using cross-modal self-supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 6829-6833. IEEE.
  23. Pei, W.; Zhang, J.; Wang, X.; Ke, L.; Shen, X.; and Tai, Y.-W. 2019. Memory-attended recurrent network for video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 8347-8356.
  24. Song, L.; Wu, W.; Qian, C.; He, R.; and Loy, C. C. 2020. Everybody's talkin': Let me talk as you want. arXiv preprint arXiv:2001.05201 . · arXiv
  25. Song, Y.; Zhu, J.; Li, D.; Wang, X.; and Qi, H. 2018. Talking face generation by conditional recurrent adversarial network. arXiv preprint arXiv:1804.04786 . · arXiv
  26. Thies, J.; Elgharib, M.; Tewari, A.; Theobalt, C.; and Nießner, M. 2020. Neural voice puppetry: Audio-driven facial reenactment. In European Conference on Computer Vision , 716-731. Springer.
  27. Vougioukas, K.; Petridis, S.; and Pantic, M. 2020. Realistic speech-driven facial animation with gans. International Journal of Computer Vision , 128(5): 1398-1413.
  28. Wang, T.-C.; Mallya, A.; and Liu, M.-Y. 2021. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 10039-10049.
  29. Weston, J.; Chopra, S.; and Bordes, A. 2014. Memory networks. arXiv preprint arXiv:1410.3916 . · arXiv
  30. Yi, R.; Ye, Z.; Zhang, J.; Bao, H.; and Liu, Y.-J. 2020. Audio-driven talking face video generation with learning-based personalized head pose. arXiv preprint arXiv:2002.10137 . · arXiv
  31. Yu, L.; Yu, J.; Li, M.; and Ling, Q. 2020. Multimodal Inputs Driven Talking Face Generation With Spatial-Temporal Dependency. IEEE Transactions on Circuits and Systems for Video Technology , 31(1): 203-216.
  32. Zhang, Z.; Li, L.; Ding, Y.; and Fan, C. 2021. Flow-Guided One-Shot Talking Face Generation With a High-Resolution Audio-Visual Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 3661-3670.
  33. Zhou, H.; Liu, Y.; Liu, Z.; Luo, P.; and Wang, X. 2019. Talking face generation by adversarially disentangled audiovisual representation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, 9299-9306.
  34. Zhou, H.; Sun, Y.; Wu, W.; Loy, C. C.; Wang, X.; and Liu, Z. 2021. Pose-controllable talking face generation by implicitly modularized audio-visual representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 4176-4186.
  35. Zhou, Y.; Han, X.; Shechtman, E.; Echevarria, J.; Kalogerakis, E.; and Li, D. 2020. MakeltTalk: speaker-aware talking-head animation. ACM Transactions on Graphics (TOG) , 39(6): 1-15.
  36. Zhu, M.; Pan, P.; Chen, W.; and Yang, Y. 2019. Dm-gan: Dynamic memory generative adversarial networks for textto-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 58025810.

Cited by

Sorted by citations

Authors
Park, S. J., Kim, M., Hong, J., Choi, J., & Ro, Y. M.
Affiliations
Korea Advanced Institute of Science and Technology (KR)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2022-06-28
Volume
36
Issue
2
Pages
2062-2070
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v36i2.20102
OpenAlex
W4283818626
Cited by
88
References
4
Citations by year
2026: 11 · 2025: 17 · 2024: 34 · 2023: 22 · 2022: 4
FWCI
6.98
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
Keywords
talking face generation, lip synchronization, audio-visual alignment, memory retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review