CoChat

A Robust Approach to Multimodal Deepfake Detection

Open access

The DOI matches its Crossref record; 52 of the 54 references it cites were found in a registry or an index; 10 of the 23 sources its claims cite were checked for support.

Abstract

The widespread use of deep learning techniques for creating realistic synthetic media, commonly known as deepfakes, poses a significant threat to individuals, organizations, and society. As the malicious use of these data could lead to unpleasant situations, it is becoming crucial to distinguish between authentic and fake media. Nonetheless, though deepfake generation systems can create convincing images and audio, they may struggle to maintain consistency across different data modalities, such as producing a realistic video sequence where both visual frames and speech are fake and consistent one with the other. Moreover, these systems may not accurately reproduce semantic and timely accurate aspects. All these elements can be exploited to perform a robust detection of fake content. In this paper, we propose a novel approach for detecting deepfake video sequences by leveraging data multimodality. Our method extracts audio-visual features from the input video over time and analyzes them using time-aware neural networks. We exploit both the video and audio modalities to leverage the inconsistencies between and within them, enhancing the final detection performance. The peculiarity of the proposed method is that we never train on multimodal deepfake data, but on disjoint monomodal datasets which contain visual-only or audio-only deepfakes. This frees us from leveraging multimodal datasets during training, which is desirable given their lack in the literature. Moreover, at test time, it allows to evaluate the robustness of our proposed detector on unseen multimodal deepfakes. We test different fusion techniques between data modalities and investigate which one leads to more robust predictions by the developed detectors. Our results indicate that a multimodal approach is more effective than a monomodal one, even if trained on disjoint monomodal datasets.

Overview

CoChat Summary

TL;DR

The paper proposes a multimodal deepfake video detector that fuses visual and audio features using time-aware neural networks, trained only on disjoint monomodal deepfake datasets. It reports that Early Fusion is the most robust fusion strategy and that multimodal detection outperforms monomodal baselines on unseen multimodal datasets.

Key findings

  • Preliminary monomodal detectors achieved AUC 0.91 and BA0.5 0.83 for visual-only, and AUC 0.96 and BA0.5 0.90 for audio-only.
  • Early Fusion was the most effective fusion strategy on average, achieving AUCs ≥ 0.90 for two of three datasets and being best on the remaining dataset; it exceeded other fusion strategies by 7% and 10% on FakeAVceleb and DFDC, respectively, while being competitive on TIMIT.
  • The multimodal approach consistently outperformed the monomodal detectors, supporting the benefit of multimodal analysis.
  • In mixed-class experiments, the Early Fusion detector achieved best average results when trained with the All Class strategy: average AUC = 0.80 vs. average AUC = 0.76 for Same Class. A fixed threshold t = 0.5 did not considerably change BA values, though FakeAVceleb showed a 5% accuracy improvement when using the best threshold for DEF; Early Fusion on FakeAVceleb was unbalanced toward “Real” (TPR0.5 = 0.954 vs. TNR0.5 = 0.580).

Methods

  • Two-stage pipeline: visual features extracted with an EfficientNetB4 ensemble modified per prior work, and audio features extracted as TDNN x-vectors via SpeechBrain; extraction at 10 Hz, with 100 ms non-overlapped audio windows, time window TW = 3.0 s, N = 30 time instants, MV = 1072 visual features, MA = 512 audio features. The detector uses a Transformer-based classifier with a single hidden layer, 8 attention heads, 0.1 dropout, and GELU, evaluated with Late, Mid, and Early Fusion strategies.
  • Training data: FaceForensics++ (5000 videos, QP 23 and QP 40 merged) for visual-only training, and ASVspoof 2019 Logical Access (>120,000 audio tracks at 16 kHz) for audio-only training. Testing was cross-dataset on FakeAVCeleb (~20,000 deepfake videos from 500 real VoxCeleb2 videos), DFDC (12,547 samples from folders 45–49), and TIMIT (VidTIMIT + DeepfakeTIMIT + TIMIT-TTS DTW subset with almost 20,000 synthetic speech tracks).
  • Training and analysis: 150 epochs with early stopping patience 15, weighted cross-entropy loss, Adam optimization, learning rate e-3, weight decay e-4, and learning-rate reduction on plateau by factor 0.1; class balancing by oversampling the less represented class. Evaluation used ROC curves, AUC, and Balanced Accuracy at threshold t = 0.5 and at the best threshold.

Limitations

  • The paper states that multimodal deepfake datasets are lacking, and that training on monomodal datasets means the detectors cannot leverage all intra-modality relationships within the content. It also states the system cannot detect synthetic content that appears realistic in individual modalities but lacks audio-video synchronization, even if simpler detectors trained explicitly for this could spot such inconsistencies.
  • The paper notes that training on monomodal datasets could impact performance on multimodal ones, and that poor AUC on DFDC is likely due to distinct characteristics between training and test data.
  • Evident from the text: evaluation is limited to three multimodal test corpora (FakeAVCeleb, DFDC, TIMIT) without multimodal training data; fixed threshold t = 0.5 yields unbalanced TPR/TNR on FakeAVCeleb, though best-threshold selection improves accuracy by 5% for DEF. The recursive Late Fusion approach was not considered further due to limited novelty and computational overhead.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/jimaging9060122

Cites

52 of 52 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Proposed pipeline for multimodal deepfake detection.
Figure 1 Proposed pipeline for multimodal deepfake detection.CC BY · PubMed Central
Figure 2 Architecture of the classifier Cm.
Figure 2 Architecture of the classifier Cm.CC BY · PubMed Central
Figure 3 Different fusion strategies considered to perform multimodal deepfake detection.
Figure 3 Different fusion strategies considered to perform multimodal deepfake detection.CC BY · PubMed Central
Figure 4 Evaluation of the considered detectors on monomodal datasets.
Figure 4 Evaluation of the considered detectors on monomodal datasets.CC BY · PubMed Central
Figure 5 Evaluation of the considered detectors on multimodal datasets considering different fusion strategies.
Figure 5 Evaluation of the considered detectors on multimodal datasets considering different fusion strategies.CC BY · PubMed Central
Figure 6 Confusion matrices obtained by considering the DLF detector (Late Fusion) on the FakeAVceleb (left), TIMIT (center) and DFDC (right) datasets. The corresponding BA0.5 values are 0.75, 0.87, and 0.69, respectively.
Figure 6 Confusion matrices obtained by considering the DLF detector (Late Fusion) on the FakeAVceleb (left), TIMIT (center) and DFDC (right) datasets. The corresponding BA0.5 values are 0.75, 0.87, and 0.69, respectively.CC BY · PubMed Central
Figure 7 Confusion matrices obtained by considering the DMF detector (Mid Fusion) on the FakeAVceleb (left), TIMIT (center) and DFDC (right) datasets. The corresponding BA0.5 values are 0.75, 0.87, and 0.70, respectively.
Figure 7 Confusion matrices obtained by considering the DMF detector (Mid Fusion) on the FakeAVceleb (left), TIMIT (center) and DFDC (right) datasets. The corresponding BA0.5 values are 0.75, 0.87, and 0.70, respectively.CC BY · PubMed Central
Figure 8 Confusion matrices obtained by considering the DEF detector (Early Fusion) on the FakeAVceleb (left), TIMIT (center) and DFDC (right) datasets. The corresponding BA0.5 values are 0.77, 0.88, and 0.74, respectively.
Figure 8 Confusion matrices obtained by considering the DEF detector (Early Fusion) on the FakeAVceleb (left), TIMIT (center) and DFDC (right) datasets. The corresponding BA0.5 values are 0.77, 0.88, and 0.74, respectively.CC BY · PubMed Central
Figure 9 Evaluation of the considered detectors on multimodal datasets considering monomodal (i.e., visual-only or audio-only) against multimodal approaches.
Figure 9 Evaluation of the considered detectors on multimodal datasets considering monomodal (i.e., visual-only or audio-only) against multimodal approaches.CC BY · PubMed Central
Figure 10 Evaluation of the DEF detector (Early Fusion) on mixed classes (real audio and fake video and viceversa). The case where both video and audio are fake is excluded.
Figure 10 Evaluation of the DEF detector (Early Fusion) on mixed classes (real audio and fake video and viceversa). The case where both video and audio are fake is excluded.CC BY · PubMed Central

Verification

Claims the paper makes

6 Valid · 0 Not valid · 6 Not checked · 29 Own results and interpretations

18 more claims

Create a free account to see all 41 claims and to verify the 2 references still open.

See all claims — free account
  • It may state two separate claims as one. 15
  • The quote only partly states this claim. 1
  • Probably not a substantial claim. 2

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Research works printed but not linked: 2

  1. Koopman, M., Rodriguez, A.M., and Geradts, Z. (2018, January 29–31). Detection of deepfake video manipulation. Proceedings of the 20th Irish Machine Vision and Image Processing Conference (IMVIP), Belfast, UK.
  2. Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., and Zhong, J. (2021). SpeechBrain: A General-Purpose Speech Toolkit. arXiv.

Cited by

See all 96

Metadata

Authors
Salvi, D., Liu, H., Mandelli, S., Bestagini, P., Zhou, W., Zhang, W., & Tubaro, S.
Affiliations
  • Politecnico di Milano (IT)
  • University of Science and Technology of China (CN)
Corresponding author
  • Salvi, Davide
  • Bestagini, Paolo
Venue
Journal of Imaging
Publisher
MDPI AG
Published
2023-06-19
Volume
9
Issue
6
Pages
122
ISSN
2313-433X (ISSN-L)
OpenAlex
W4381140704
Cited by
96
References
54
Citations by year
2026: 34 · 2025: 35 · 2024: 24 · 2023: 3
FWCI
8.68
Type
journal article
Language
en
Open access
Locations
  • Journal of Imaging (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Virtual Community of Pathological Anatomy (University of Castilla La Mancha) (submittedVersion, other-oa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Defense Advanced Research Projects Agency (FA8750-20-2-1004, PE00000001)
  • European Union (FA8750-20-2-1004, PE00000001)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
deepfake detection, multimodal fusion, audio-visual consistency, synthetic media, fusion techniques

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.