The DOI matches its Crossref record; 52 of the 54 references it cites were found in a registry or an index; 9 of the 23 sources its claims cite were checked for support.
A Robust Approach to Multimodal Deepfake Detection
Open accessAbstract
The widespread use of deep learning techniques for creating realistic synthetic media, commonly known as deepfakes, poses a significant threat to individuals, organizations, and society. As the malicious use of these data could lead to unpleasant situations, it is becoming crucial to distinguish between authentic and fake media. Nonetheless, though deepfake generation systems can create convincing images and audio, they may struggle to maintain consistency across different data modalities, such as producing a realistic video sequence where both visual frames and speech are fake and consistent one with the other. Moreover, these systems may not accurately reproduce semantic and timely accurate aspects. All these elements can be exploited to perform a robust detection of fake content. In this paper, we propose a novel approach for detecting deepfake video sequences by leveraging data multimodality. Our method extracts audio-visual features from the input video over time and analyzes them using time-aware neural networks. We exploit both the video and audio modalities to leverage the inconsistencies between and within them, enhancing the final detection performance. The peculiarity of the proposed method is that we never train on multimodal deepfake data, but on disjoint monomodal datasets which contain visual-only or audio-only deepfakes. This frees us from leveraging multimodal datasets during training, which is desirable given their lack in the literature. Moreover, at test time, it allows to evaluate the robustness of our proposed detector on unseen multimodal deepfakes. We test different fusion techniques between data modalities and investigate which one leads to more robust predictions by the developed detectors. Our results indicate that a multimodal approach is more effective than a monomodal one, even if trained on disjoint monomodal datasets.
Overview
CoChat Summary
TL;DR
The paper proposes a multimodal deepfake video detector that fuses visual and audio features using time-aware neural networks, trained only on disjoint monomodal deepfake datasets. It reports that Early Fusion is the most robust fusion strategy and that multimodal detection outperforms monomodal baselines on unseen multimodal datasets.
Key findings
- Preliminary monomodal detectors achieved AUC 0.91 and BA0.5 0.83 for visual-only, and AUC 0.96 and BA0.5 0.90 for audio-only.
- Early Fusion was the most effective fusion strategy on average, achieving AUCs ≥ 0.90 for two of three datasets and being best on the remaining dataset; it exceeded other fusion strategies by 7% and 10% on FakeAVceleb and DFDC, respectively, while being competitive on TIMIT.
- The multimodal approach consistently outperformed the monomodal detectors, supporting the benefit of multimodal analysis.
- In mixed-class experiments, the Early Fusion detector achieved best average results when trained with the All Class strategy: average AUC = 0.80 vs. average AUC = 0.76 for Same Class. A fixed threshold t = 0.5 did not considerably change BA values, though FakeAVceleb showed a 5% accuracy improvement when using the best threshold for DEF; Early Fusion on FakeAVceleb was unbalanced toward “Real” (TPR0.5 = 0.954 vs. TNR0.5 = 0.580).
Methods
- Two-stage pipeline: visual features extracted with an EfficientNetB4 ensemble modified per prior work, and audio features extracted as TDNN x-vectors via SpeechBrain; extraction at 10 Hz, with 100 ms non-overlapped audio windows, time window TW = 3.0 s, N = 30 time instants, MV = 1072 visual features, MA = 512 audio features. The detector uses a Transformer-based classifier with a single hidden layer, 8 attention heads, 0.1 dropout, and GELU, evaluated with Late, Mid, and Early Fusion strategies.
- Training data: FaceForensics++ (5000 videos, QP 23 and QP 40 merged) for visual-only training, and ASVspoof 2019 Logical Access (>120,000 audio tracks at 16 kHz) for audio-only training. Testing was cross-dataset on FakeAVCeleb (~20,000 deepfake videos from 500 real VoxCeleb2 videos), DFDC (12,547 samples from folders 45–49), and TIMIT (VidTIMIT + DeepfakeTIMIT + TIMIT-TTS DTW subset with almost 20,000 synthetic speech tracks).
- Training and analysis: 150 epochs with early stopping patience 15, weighted cross-entropy loss, Adam optimization, learning rate e-3, weight decay e-4, and learning-rate reduction on plateau by factor 0.1; class balancing by oversampling the less represented class. Evaluation used ROC curves, AUC, and Balanced Accuracy at threshold t = 0.5 and at the best threshold.
Limitations
- The paper states that multimodal deepfake datasets are lacking, and that training on monomodal datasets means the detectors cannot leverage all intra-modality relationships within the content. It also states the system cannot detect synthetic content that appears realistic in individual modalities but lacks audio-video synchronization, even if simpler detectors trained explicitly for this could spot such inconsistencies.
- The paper notes that training on monomodal datasets could impact performance on multimodal ones, and that poor AUC on DFDC is likely due to distinct characteristics between training and test data.
- Evident from the text: evaluation is limited to three multimodal test corpora (FakeAVCeleb, DFDC, TIMIT) without multimodal training data; fixed threshold t = 0.5 yields unbalanced TPR/TNR on FakeAVCeleb, though best-threshold selection improves accuracy by 5% for DEF. The recursive Late Fusion approach was not considered further due to limited novelty and computational overhead.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures










Verification
Claims the paper makes
6 Valid · 0 Not valid · 6 Not checked · 29 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Moreover, the results of [38] show that an ensemble of audio and visual baselines outperforms monomodal counterparts.”
§ A Robust Approach to Multimodal Deepfake Detection > 2. Deepfake Detection > 2.3. Audio-Visual Deepfake Detection
What the sources say
- Supportedcites [38] Khalid, H., Kim, M., Tariq, S., and Woo, S.S. (2021, January 24). Evaluation of an audio-video multimodal deepfake dataset using unimodal and multimodal detectors. Proceedings of the 1st Workshop on Synthetic Multimedia-Audiovisual Deepfake Generation and Detection, Virtual Event. source
The Abstract (p. 1) says that audio- or video-only unimodal methods do not perform well compared with ensemble-based methods. Section 4.2.2 (p. 5) identifies those ensembles as combinations of one audio and one video classifier, matching the claim. (LLM, conf=0.91)
states a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In the recent years, few multimodal datasets have been proposed, containing both counterfeited video and audio tracks.”
- Also stated in: § 2.3. Audio-Visual Deepfake Detection · plain
“These are DFDC [42], FakeAVCeleb [43], and DeepfakeTIMIT [44] with TIMIT-TTS [6].”
§ A Robust Approach to Multimodal Deepfake Detection > 2. Deepfake Detection > 2.3. Audio-Visual Deepfake Detection
What the sources say
- Partly supportedcites [42] Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., and Ferrer, C.C. (2020). The deepfake detection challenge (DFDC) dataset. arXiv. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) and Dataset Construction (p. 2) identify DFDC as a dataset of videos with manipulated faces, supporting its identification as a counterfeited-video dataset. They describe face swaps but do not establish that the audio tracks are also counterfeited. This source also does not establish the claim about FakeAVCeleb or DeepFakeTIMIT with TIMIT-TTS. (LLM, conf=0.84)
- Partly supportedcites [43] Khalid, H., Tariq, S., Kim, M., and Woo, S.S. (2021, January 6–14). FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset. Proceedings of the Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), Virtual. sourceExperimental — accuracy not yet measured
The Introduction (p. 2) identifies DFDC as a dataset containing deepfake video and synthesized audio, and describes FakeAVCeleb as containing both. That supports the citing sentence’s general point that such datasets were few. But Background and Motivation (p. 3) says Deepfake TIMIT’s audio was real, not fake; our copy does not identify a TIMIT-TTS variant, so the claim’s three-dataset list is not fully supported. (LLM, conf=0.86)
- Not matchedcites [44] Korshunov, P., and Marcel, S. (2018). Deepfakes: A new threat to face recognition? Assessment and detection. arXiv.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.3. Audio-Visual Deepfake Detection · plain
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“To overcome these limitations, multimodal approaches have been recently proposed, able to combine information from various domains to enhance the accuracy of the detection process [7,8].”
§ A Robust Approach to Multimodal Deepfake Detection > 1. Introduction
What the sources say
- Supportedcites [7] Agarwal, S., Hu, L., Ng, E., Darrell, T., Li, H., and Rohrbach, A. (2023, January 2–7). Watch those words: Video falsification detection using word-conditioned facial motion. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA. source
The Related Work section (p. 3) says multimodal techniques use both audio and visual signals to detect deepfakes. The Discussion (p. 8) describes the paper’s own multimodal approach, combining spoken words with facial gestures, and reports that it outperformed other methods in most tested cases. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [8] Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., and Manocha, D. (2020, January 12–16). Emotions don’t lie: An audio-visual deepfake detection method using affective cues. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, WA, USA. source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“These results are consistent with those of many cutting-edge detectors reported in the literature [20,34], indicating that the proposed monomodal classifiers are suitable for subsequent multimodal experiments.”
§ A Robust Approach to Multimodal Deepfake Detection > 5. Results > 5.2. Monomodal Results
What the sources say
- Supportedcites [20] Bonettini, N., Cannas, E.D., Mandelli, S., Bondi, L., Bestagini, P., and Tubaro, S. (2021, January 10–15). Video face manipulation detection through ensemble of CNNs. Proceedings of the International Conference on Pattern Recognition (ICPR), Virtual Event. source
The citation supports the comparison with published face-manipulation detectors, not the citing authors’ inference about their later multimodal experiments. The Introduction (p. 2) reports that the proposed CNN ensemble outperformed an FF++ baseline on two datasets, and the Results (p. 6) compare detector performance using AUC and LogLoss. (LLM, conf=0.86)
- Not checked — no full text of the sourcecites [34] Attorresi, L., Salvi, D., Borrelli, C., Bestagini, P., and Tubaro, S. (2022, January 21–25). Combining Automatic Speaker Verification and Prosody Analysis for Synthetic Speech Detection. Proceedings of the International Conference on Pattern Recognition, Montréal, QC, Canada. source
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The DEF model achieves the best average results when trained following the All Class strategy (average AUC = 0.80 vs. average AUC = 0.76 of the Same Class).”
§ A Robust Approach to Multimodal Deepfake Detection > 5. Results > 5.3. Multimodal Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
18 more claims
Create a free account to see all 41 claims and to verify the 2 references still open.
- It may state two separate claims as one. 15
- The quote only partly states this claim. 1
- Probably not a substantial claim. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A Robust Approach to Multimodal Deepfake Detection > 5. Results > 5.3. Multimodal Results
“Early Fusion results on FakeAVceleb are strongly unbalanced towards the “Real” class (i.e., TPR0.5=0.954 vs. TNR0.5=0.580).”
§ A Robust Approach to Multimodal Deepfake Detection > 5. Results > 5.1. Evaluation Metrics
“If the likelihood exceeds the threshold, the sequence is classified as “Fake”, otherwise it is classified as “Real”.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
X-Vectors: Robust DNN Embeddings for Speaker Recognition
Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., & Khudanpur, S. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461375 · Cited by 2,620
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
VoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210
MesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Deepfake Video Detection Using Recurrent Neural Networks
Guera, D., & Delp, E. J. (2018) · 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2018.8639163 · Cited by 1,285
DARPA TIMIT: acoustic-phonetic continuous speech corpus CD-ROM, NIST speech disc 1-1.1
Garofolo, J. S., Lamel, L. F., Fisher, W. M., Fiscus, J. G., Pallett, D. S., & Dahlgren, N. L. (1993) · doi:10.6028/nist.ir.4930 · Cited by 1,267
Exposing Deep Fakes Using Inconsistent Head Poses
Yang, X., Li, Y., & Lyu, S. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683164 · Cited by 1,196
Research works printed but not linked: 2
- Koopman, M., Rodriguez, A.M., and Geradts, Z. (2018, January 29–31). Detection of deepfake video manipulation. Proceedings of the 20th Irish Machine Vision and Image Processing Conference (IMVIP), Belfast, UK.
- Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., and Zhong, J. (2021). SpeechBrain: A General-Purpose Speech Toolkit. arXiv.
Cited by
Abbas, F., & Taeihagh, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124260 · Cited by 184
Deepfake Media Forensics: Status and Future Challenges
Amerini, I., Barni, M., Battiato, S., Bestagini, P., Boato, G., Bruni, V., Caldelli, R., De Natale, F., De Nicola, R., Guarnera, L., et al. (2025) · Journal of Imaging · doi:10.3390/jimaging11030073 · Cited by 113
Self-citationA novel approach for detecting deep fake videos using graph neural network
El-Gayar, M. M., Abouhawwash, M., Askar, S., & Sweidan, S. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00884-y · Cited by 82
Deepfake video detection methods, approaches, and challenges
Alrashoud, M. (2025) · Alexandria Engineering Journal · doi:10.1016/j.aej.2025.04.007 · Cited by 49
Audio–Visual Synchronization and Lip Movement Analysis for Real-Time Deepfake Detection
Javed, M., Zhang, Z., Dahri, F. H., Laghari, A. A., Krajčík, M., & Almadhor, A. S. (2025) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-025-00911-7 · Cited by 62
Metadata
- Authors
- Salvi, D., Liu, H., Mandelli, S., Bestagini, P., Zhou, W., Zhang, W., & Tubaro, S.
- Affiliations
- Politecnico di Milano (IT)
- University of Science and Technology of China (CN)
- Corresponding author
- Salvi, Davide
- Bestagini, Paolo
- Venue
- Journal of Imaging
- Publisher
- MDPI AG
- Published
- 2023-06-19
- Volume
- 9
- Issue
- 6
- Pages
- 122
- ISSN
- 2313-433X (ISSN-L)
- PMC
- PMC10299653
- OpenAlex
- W4381140704
- Cited by
- 96
- References
- 54
- Citations by year
- 2026: 34 · 2025: 35 · 2024: 24 · 2023: 3
- FWCI
- 8.68
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Journal of Imaging (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Virtual Community of Pathological Anatomy (University of Castilla La Mancha) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Defense Advanced Research Projects Agency (FA8750-20-2-1004, PE00000001)
- European Union (FA8750-20-2-1004, PE00000001)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- deepfake detection, multimodal fusion, audio-visual consistency, synthetic media, fusion techniques
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Robust Approach to Multimodal Deepfake Detection. https://app.cochat.ai/papers/10.3390/jimaging9060122
- The paper
- Salvi, D., Liu, H., Mandelli, S., Bestagini, P., Zhou, W., Zhang, W., & Tubaro, S. (2023). A Robust Approach to Multimodal Deepfake Detection. Journal of Imaging. https://doi.org/10.3390/jimaging9060122
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.