Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index; 18 of the 41 sources its claims cite were checked for support.

Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization

Open access
Katamneni, Vinaya Sree, Rattani, Ajita · 2024 · 2024 IEEE International Joint Conference on Biometrics (IJCB)

Abstract

In the digital age, the emergence of deepfakes and synthetic media presents a significant threat to societal and political integrity. Deepfakes based on multi-modal manipulation, such as audio-visual, are more realistic and pose a greater threat. Current multi-modal deepfake detectors are often based on the attention-based fusion of heterogeneous data streams from multiple modalities. However, the heterogeneous nature of the data (such as audio and visual signals) creates a distributional modality gap and poses a significant challenge in effective fusion and hence multi-modal deepfake detection. In this paper, we propose a novel multi-modal attention framework based on recurrent neural networks (RNNs) that leverages contextual information for audio-visual deepfake detection. The proposed approach applies attention to multi-modal multi-sequence representations and learns the contributing features among them for deepfake detection and localization. Thorough experimental validations on audio-visual deepfake datasets, namely FakeAVCeleb, AV-Deepfake1M, TVIL, and LAV-DF datasets, demonstrate the efficacy of our approach. Cross-comparison with the published studies demonstrates superior performance of our approach with an improved accuracy and precision by 3.47% and 2.05% in deepfake detection and localization, respectively. Thus, obtaining state-of-the-art performance. To facilitate reproducibility, the code and the datasets information is available at https://github.com/vcbsl/audiovisual-deepfake/.

Overview

CoChat Summary

TL;DR

The paper proposes MMMS-BA, a recurrent neural network-based multi-modal multi-sequence bi-modal attention framework for audio-visual deepfake detection and localization. It reports state-of-the-art performance on FakeAVCeleb, AV-Deepfake1M, LAV-DF, and TVIL, with improved detection and localization over published methods.

Key findings

  • On intra-dataset evaluation, MMMS-BA achieved AUC/ACC of 0.989/0.979 on FakeAVCeleb and 0.979/0.968 on AV-Deepfake1M; average intra-dataset AUC/ACC was 0.984/0.973. Cross-dataset average AUC/ACC was 0.944/0.919.
  • On FakeAVCeleb, MMMS-BA outperformed published audio-visual detectors with AUC 0.989 and ACC 0.979, an average increment of 0.0341 AUC and 3.47% ACC; the abstract reports improved accuracy and precision by 3.47% and 2.05% for detection and localization, respectively.
  • For localization, MMMS-BA obtained the best AP@0.5/AR@50 on AV-Deepfake1M (62.75/57.49), LAV-DF (97.56/93.45), and TVIL (96.87/90.47); however, UMMAFormer had higher AP@0.75 and AP@0.95 on TVIL (84.70 and 62.43).
  • Ablation showed full visual+lip+audio (V+L+A) gave the best AUC 0.989 and EER 0.029, versus V+L AUC 0.814, L+A AUC 0.924, and V+A AUC 0.955; MMMS-BA’s EER 0.029 was lower than MMUS-SA (0.033) and MS-SA (0.045).

Methods

  • MMMS-BA processes full visual face, lip, and audio sequences with separate Bi-GRU layers (300 neurons each) and dense layers (100 neurons), dropout 0.3, then computes pairwise bi-modal attention for (V,L), (L,A), and (A,V), multi-sequence attention, multiplicative gating, and concatenation with residual skip connections. Classification uses focal loss; localization uses a regression head with differentiable IoU/DIoU loss and non-maximum suppression.
  • Evaluation used FakeAVCeleb, AV-Deepfake1M, TVIL, and LAV-DF. Detection metrics were AUC, pAUC at 10% FPR, and EER; localization metrics were AP and AR at multiple thresholds. For AV-Deepfake1M, the training set was split 85%/15% for training/validation because the test set was not released, and the actual validation set was used for testing.
  • Ablations varied the attention mechanism (MMUS-SA and MS-SA) and modality combinations (V+A, V+L, L+A, V+L+A).

Limitations

  • Stated: future work will extend the framework to incorporate text analysis and adapt it to account for missing modalities during training and inference. The paper also attributes lower AV-Deepfake1M localization performance to highly realistic content-driven fake content.
  • Evident: AV-Deepfake1M’s test set was not released, so the actual validation set was used for testing. Cross-dataset detection performance dropped relative to intra-dataset evaluation, e.g., average AUC/ACC decreases of 0.053/0.076 when trained on FakeAVCeleb and 0.027/0.029 when trained on AV-Deepfake1M.
  • Evident: on TVIL, UMMAFormer achieved higher AP@0.75 (84.70) and AP@0.95 (62.43) than MMMS-BA (81.33 and 28.43), indicating weaker high-threshold localization. The paper has no explicit limitations section.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/ijcb62174.2024.10744480

Cites

77 of 77 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

11 Valid · 5 Not valid · 9 Not checked · 55 Own results and interpretations

37 more claims · 4 with their cited sources checked

Create a free account to see all 80 claims and to verify the 3 references still open.

See all claims — free account
  • Probably not a substantial claim. 11
  • It may state two separate claims as one. 26
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Show all 77 references

Research works printed but not linked: 3

  1. Citron, How deepfakes undermine truth and threaten democracy
  2. Khalid, 2021, Fakeavceleb: A novel audio-video multimodal deepfake dataset
  3. 10.1109/ICASSP49357.2023.10095247 · DOI

Cited by

See all 20

Metadata

Authors
Katamneni, V. S., & Rattani, A.
Affiliations
University of North Texas (US)
Venue
2024 IEEE International Joint Conference on Biometrics (IJCB)
Publisher
IEEE
Published
2024-09-15
Pages
1-11
OpenAlex
W4404239158
Cited by
20
References
77
Citations by year
2026: 12 · 2025: 8
FWCI
4.11
Type
conference paper
Language
en
Open access
Locations
  • IEEE International Joint Conference on Biometrics (IJCB) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
cross-modal attention, multimodal fusion, recurrent neural networks, contextual information, modality gap, synthetic media

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization. https://app.cochat.ai/papers/10.1109/ijcb62174.2024.10744480
The paper
Katamneni, V. S., & Rattani, A. (2024). Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization. 2024 IEEE International Joint Conference on Biometrics (IJCB). https://doi.org/10.1109/ijcb62174.2024.10744480

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.