Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index; 35 of the 51 sources its claims cite were checked for support.

Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection

Open access
Masood, Momina, Javed, Ali, Irtaza, Aun · 2023 · Preprint (Research Square)

Abstract

Abstract Deepfake media proliferated on the internet has major societal consequences for politicians, celebrities, and even common people. Recent advancements in deepfake videos include the creation of realistic talking faces and the usage of synthetic human voices. Numerous deepfake detection approaches have been proposed in response to the potential harm caused by deepfakes. However, the majority of deepfake detection methods process audio and video modality independently and have low identification accuracy. In this work, we propose an ensemble multimodal deepfake detection method that can identify both auditory and facial manipulations by exploiting correspondence between audio-visual modalities. The proposed framework comprises unimodal and cross-modal learning networks to exploit intra- and inter-modality inconsistencies introduced as a result of manipulation. The suggested multimodal approach employs an ensemble of deep convolutional neural-network based on an attention mechanism that extracts representative features and effectively determines if a video is fake or real. We evaluated the proposed approach on several benchmark multimodal deepfake datasets including FakeAVCeleb, DFDC-p, and DF-TIMIT. Experimental results demonstrate that an ensemble of deep learners based on unimodal and cross-modal network mechanisms exploit highly semantic information between audio and visual signals and outperforms independently trained audio and visual classifiers. Moreover, it can effectively identify different unseen types of deepfakes as well as robust under various post-processing attacks. The results confirm that our approach outperforms existing unimodal/multimodal classifiers for audio-visual manipulated video identification.

Overview

CoChat Summary

## TL;DR

The paper proposes an ensemble multimodal framework (AVDFDNet) for audio-visual deepfake detection that combines a video network, an audio network, and a cross-modal audio-visual network with hard voting. On the FakeAVCeleb dataset, the proposed model reports 94.08% accuracy and also shows cross-dataset generalization and robustness to perturbations.

## Key findings

  • On FakeAVCeleb, the proposed AVDFDNet achieved 94.08% accuracy, with 92.91% precision, 95.43% recall, and 94.15% F1-score; this outperformed the video network (90.79%), audio network (83.40%), and audio-visual network (92.64%).
  • The audio-visual network alone outperformed the unimodal models on FakeAVCeleb, reaching 92.64% accuracy versus 90.79% for video-only and 83.40% for audio-only.
  • On different manipulation types in FakeAVCeleb, the model reported its best accuracies on Faceswap (97.51%), Faceswap-wav2lip (98.11%), and Fsgan-wav2lip (95.33%); lower accuracies were reported for Fsgan (86.94%) and wav2lip (88.42%).
  • In cross-dataset evaluation, the model achieved accuracy greater than 75% on both DFDC and DF-TIMIT, including 88.44% on DF-TIMIT(HQ) and 85.03% on DF-TIMIT(LQ); on DFDC it reported 76.62%.
  • Under perturbation testing on FakeAVCeleb test videos, accuracy dropped from 94.08% to 88.24%, and after retraining with data augmentation it increased to 91.81%.

## Methods

  • The study frames audio-video manipulation detection as a binary classification problem and builds an ensemble AVDFDNet with three components: a 3D CNN-based video network with spatial-temporal attention, a ResNet-50-based audio network on mel-spectrograms, and a cross-modal audio-visual network using cross-attention and hybrid fusion.
  • Data come from FakeAVCeleb (reported as 500 real videos and 19,500 manipulated videos), DFDC, and DF-TIMIT; the paper says it supplemented FakeAVCeleb with 1,600 real videos from VoxCeleb1 and used an 85:15 train-test split for selected videos.
  • Training used TensorFlow/Keras, AdamW, binary cross entropy loss, 2-second segments, 20 FPS video sampling, and 16 kHz audio sampling; evaluation used precision, recall, F1-score, accuracy, and AUC, plus cross-dataset and robustness tests.

## Limitations

  • The paper states that DFDC is challenging because it contains partially forged videos, varied poses/lighting/backgrounds, and labels videos as fake even when only the voice is replaced; this helps explain the lower cross-dataset accuracy (76.62%) on DFDC.
  • The paper notes that DF-TIMIT contains videos manipulated through face-swap with frontal pose without occlusion or rotation, which makes them easier to detect; performance was lower on DF-TIMIT(LQ) (85.03%) than DF-TIMIT(HQ) (88.44%).
  • Evident from the text: some reported performance details are incomplete or inconsistent in the pasted tables/sections, and the abstract is truncated mid-sentence; also, the paper’s evaluation relies heavily on publicly available benchmark datasets, so real-world generalization beyond those datasets is not directly established in the text.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.21203/rs.3.rs-3415144/v1

Cites

90 of 90 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

27 Valid · 6 Not valid · 6 Not checked · 69 Own results and interpretations

65 more claims · 14 with their cited sources checked

Create a free account to see all 108 claims and to verify the 5 references still open.

See all claims — free account
  • Probably not a substantial claim. 6
  • It may state two separate claims as one. 35

Create a free account to see these claims.

Citations

Cites

Show all 77 references

Research works printed but not linked: 1

  1. J. Khochare, C. Joshi, B. Yenarkar, S. Suratkar, and F. Kazi, "A Deep Learning Framework for Audio Deepfake Detection," Arabian Journal for Science and Engineering, vol. 1, pp. 1–12, 2021.

Cited by

See all 6

Metadata

Authors
Masood, M., Javed, A., & Irtaza, A.
Affiliations
University of Engineering and Technology
Corresponding author
Javed, Ali
Venue
Preprint (Research Square)
Published
2023-10-11
OpenAlex
W4387527117
Cited by
6
References
77
Citations by year
2026: 2 · 2025: 3 · 2024: 1
Type
posted content
Language
en
Open access
Locations
Research Square (acceptedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
deepfake detection, multimodal learning, attention mechanism, ensemble convolutional neural networks, cross-modal learning, sound manipulation, face manipulation, post-processing attacks, synthetic speech

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection. https://app.cochat.ai/papers/10.21203/rs.3.rs-3415144/v1
The paper
Masood, M., Javed, A., & Irtaza, A. (2023). Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection. Preprint (Research Square). https://doi.org/10.21203/rs.3.rs-3415144/v1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.