CoChat

Speech2Video: Cross-Modal Distillation for Speech to Video Generation

The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index; none of the 19 sources its claims cite is checked for support yet.

Abstract

This paper investigates a novel task of talking face video generation solely from speeches.The speech-to-video generation technique can spark interesting applications in entertainment, customer service, and human-computer-interaction industries.Indeed, the timbre, accent and speed in speeches could contain rich information relevant to speakers' appearance.The challenge mainly lies in disentangling the distinct visual attributes from audio signals.In this article, we propose a light-weight, cross-modal distillation method to extract disentangled emotional and identity information from unlabelled video inputs.The extracted features are then integrated by a generative adversarial network into talking face video clips.With carefully crafted discriminators, the proposed framework achieves realistic generation results.Experiments with observed individuals demonstrated that the proposed framework captures the emotional expressions solely from speeches, and produces spontaneous facial motion in the video output.Compared to the baseline method where speeches are combined with a static image of the speaker, the results of the proposed framework is almost indistinguishable.User studies also show that the proposed method outperforms the existing algorithms in terms of emotion expression in the generated videos.

Overview

CoChat Summary

## TL;DR

The paper proposes speech2video, a two-stage framework that generates a talking-face video from speech alone by first distilling identity and emotion from unlabeled talking-face videos and then composing frames with a GAN. On CREMA-D and VoxCeleb2, the method produces visually competitive results and user studies reportedly favor it for emotion expression, though it cannot match methods that use a reference image in reconstruction metrics.

## Key findings

  • The framework uses a two-stage pipeline: cross-modal distillation to extract identity and emotion features from speech, followed by an adversarial video composer.
  • On CREMA-D, the reported quantitative results for speech2video are SSIM 0.541, PSNR 20.540, and confidence 5.1; the corresponding SDFA baseline scores are 0.705, 23.568, and 5.4.
  • On VoxCeleb2, speech2video reports SSIM 0.5707, PSNR 20.891, and confidence 5.1.
  • The paper states that qualitative results are “almost indistinguishable” from the baseline using a static image, and that user studies show the proposed method outperforms existing algorithms in emotion expression.

## Methods

  • Design: A two-stage model: (1) a cross-modal distillation module disentangles identity and emotional features from speech; (2) a GAN-based adversarial composer generates talking-face video frames using temporal speech features plus the disentangled identity/emotion vectors.
  • Data/sample: Trained and evaluated on two talking-face datasets: CREMA-D (7,442 clips from 91 actors, 6 emotions, split 70%/15%/15%) and VoxCeleb2 (more than 1 million utterances from 6,112 celebrities).
  • Main analysis: Qualitative comparisons against SDFA and S2F+SDFA, and quantitative evaluation using SSIM, PSNR, and AV confidence.

## Limitations

  • The paper explicitly notes that the proposed method scores below SDFA on SSIM and PSNR because it does not use a ground-truth/reference image, making perfect facial reconstruction impossible with a GAN-based approach.
  • It also states that for unobserved voices, the generated appearance does not match ground truth, though gender, complexion, and age are roughly preserved.
  • Evident from the text: several parts of the method rely on unlabeled talking-face videos and pretrained teacher networks, so performance may depend on the availability and quality of such training data and pretrained models.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.21437/interspeech.2021-1996

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 0 Not valid · 7 Not checked · 26 Own results and interpretations

18 more claims

Create a free account to see all 33 claims and to verify the 19 references still open.

See all claims — free account
  • It may state two separate claims as one. 5
  • Probably not a substantial claim. 6

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 34 references

Research works printed but not linked: 3

  1. [7][7] A. Nagrani, J. S. Chung, S. Albanie, and A. Zisserman, 'Disentangled speech embeddings using cross-modal self-supervision,' in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 6829-6833. -
  2. [26][26] O. M. Parkhi, A. Vedaldi, and A. Zisserman, 'Deep face recognition,' Proceedings of the British Machine Vision Conference (BMVC) , pp. 41.1-41.12, 2015. -
  3. [29][29] O. M. Parkhi, A. Vedaldi, and A. Zisserman, 'Deep face recognition,' in British Machine Vision Conference , 2015. -

Cited by

See all 15

Metadata

Authors
Si, S., Wang, J., Qu, X., Cheng, N., Wei, W., Zhu, X., & Xiao, J.
Affiliations
Ping An (China) (CN)
Corresponding author
Wang, Jianzong
Venue
Interspeech 2021
Publisher
ISCA
Published
2021-08-30
Pages
1629-1633
ISSN
2308-457X (ISSN-L)
OpenAlex
W3197087768
Cited by
12
References
32
Citations by year
2025: 1 · 2024: 3 · 2023: 6 · 2022: 2
FWCI
0.90
Type
conference paper
Language
en
Open access
closed
Locations
Interspeech (USB) (publishedVersion)
Funders
National Key Research and Development Program of China (2018YFB0204403, 2018YFB1003500, No. 2018YFB0204403, 2017YFB1401202)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
Keywords
cross-modal distillation, disentangled representation learning, generative adversarial networks, audio-visual synthesis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Speech2Video: Cross-Modal Distillation for Speech to Video Generation. https://app.cochat.ai/papers/10.21437/interspeech.2021-1996
The paper
Si, S., Wang, J., Qu, X., Cheng, N., Wei, W., Zhu, X., & Xiao, J. (2021). Speech2Video: Cross-Modal Distillation for Speech to Video Generation. Interspeech 2021. https://doi.org/10.21437/interspeech.2021-1996

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.