Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 35 of the 37 references it cites were found in a registry or an index; 9 of the 11 sources its claims cite were checked for support.

Integrating Audio-Visual Features For Multimodal Deepfake Detection

Open access
Muppalla, Sneha, Jia, Shan, Lyu, Siwei · 2023 · 2023 IEEE MIT Undergraduate Research Technology Conference (URTC)

Abstract

Deepfakes are AI-generated media in which an image or video has been digitally modified. The advancements made in deepfake technology have led to privacy and security issues. Most deepfake detection techniques rely on the detection of a single modality. Existing methods for audio-visual detection do not always surpass that of the analysis based on single modalities. Therefore, this paper proposes an audio visual based method for deepfake detection, which integrates fine-grained deepfake identification with binary classification. We categorize the samples into four types by combining labels specific to each single modality. This method enhances the detection under intra-domain and cross-domain testing.

Overview

CoChat Summary

TL;DR

This paper proposes an audio-visual deepfake detector that combines binary classification for each modality with a fine-grained four-class identification module for real/fake audio-video combinations. It reports improved intra-domain and cross-domain detection on FakeAVCeleb and TMC over single-modality and traditional fusion baselines.

Key findings

  • On FakeAVCeleb intra-domain testing, the proposed score-fusion method with Capsule Forensics achieved overall AUC 99.30 and ACC 99.20, outperforming reported baselines such as AVoiD-DF (89.20 AUC, 83.70 ACC) and AV-Lip-Sync (94.00 ACC).
  • On TMC intra-domain testing, the proposed score-fusion method with Capsule Forensics achieved overall AUC 93.67 and ACC 93.15; with Swin Transformer it achieved 80.11 AUC and 85.57 ACC.
  • Under cross-domain testing, the proposed method achieved the best performance in both scenarios; for example, TMC→FakeAVCeleb Ours-S reached 82.17 AUC and 89.73 ACC, while FakeAVCeleb→TMC Ours-S reached 70.41 AUC and 79.68 ACC in one reported setting.
  • In cross-type testing on TMC mismatched real audio/video, the score-fusion Swin Transformer model achieved 63.83 AUC and 69.63 ACC, while the naive fusion baseline achieved 44.00 ACC; most mismatched samples were classified as “Real Video Fake Audio.”

Methods

  • Video preprocessing extracts frames (1 frame per second for videos longer than a minute, all frames for shorter videos), detects and crops faces with MTCNN; audio is converted to WAV and then to mel-spectrogram images with frequency scale 0–8000 Hz and length 4 seconds.
  • Features are extracted using Capsule network and Swin Transformer backbones; fusion is either feature fusion (concatenating averaged visual features with audio features) or score fusion (averaging video and audio network scores), followed by a four-class classification head.
  • The multi-task loss combines binary cross-entropy losses for audio and video with a four-class cross-entropy loss for the four audio-video types; models are trained for 50 epochs with batch size 10, Adam optimizer, and learning rate 5×10⁻⁴, evaluated on FakeAVCeleb and TMC with an 80-20 split using Accuracy and AUC.

Limitations

  • The paper does not include a dedicated limitations section; it states future work will focus on more robust multimodal networks and improving generalizability to unseen deepfakes.
  • Evident from the reported results: cross-domain performance drops substantially compared with intra-domain performance (e.g., FakeAVCeleb→TMC Ours-S around 65–70 AUC versus 99.30 AUC on FakeAVCeleb intra-domain), indicating a generalization gap.
  • The authors attribute lower TMC results to the dataset’s high diversity and additional perturbations in both video and audio tracks.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/urtc60662.2023.10534969

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

3 Valid · 1 Not valid · 2 Not checked · 19 Own results and interpretations

14 more claims

Create a free account to see all 25 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 3
  • Probably not a substantial claim. 3

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 37 references

Research works printed but not linked: 1

  1. Roberts, 2022, Understanding the mel spectrogram

Cited by

See all 46

Metadata

Authors
Muppalla, S., Jia, S., & Lyu, S.
Affiliations
  • Cupertino High School,Cupertino,CA
  • University at Buffalo, State University of New York (US)
Venue
2023 IEEE MIT Undergraduate Research Technology Conference (URTC)
Publisher
IEEE
Published
2023-10-06
Pages
1-5
OpenAlex
W4398789263
Cited by
46
References
37
Citations by year
2026: 12 · 2025: 22 · 2024: 12
FWCI
4.90
Type
conference paper
Language
en
Open access
Locations
  • IEEE MIT Undergraduate Research Technology Conference (URTC) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
Keywords
deepfake detection, multimodal fusion, audio-visual features, binary classification, domain generalization, AI-generated content

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Integrating Audio-Visual Features For Multimodal Deepfake Detection. https://app.cochat.ai/papers/10.1109/urtc60662.2023.10534969
The paper
Muppalla, S., Jia, S., & Lyu, S. (2023). Integrating Audio-Visual Features For Multimodal Deepfake Detection. 2023 IEEE MIT Undergraduate Research Technology Conference (URTC). https://doi.org/10.1109/urtc60662.2023.10534969

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.