The DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index; 35 of the 51 sources its claims cite were checked for support.
Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection
Open accessAbstract
Abstract Deepfake media proliferated on the internet has major societal consequences for politicians, celebrities, and even common people. Recent advancements in deepfake videos include the creation of realistic talking faces and the usage of synthetic human voices. Numerous deepfake detection approaches have been proposed in response to the potential harm caused by deepfakes. However, the majority of deepfake detection methods process audio and video modality independently and have low identification accuracy. In this work, we propose an ensemble multimodal deepfake detection method that can identify both auditory and facial manipulations by exploiting correspondence between audio-visual modalities. The proposed framework comprises unimodal and cross-modal learning networks to exploit intra- and inter-modality inconsistencies introduced as a result of manipulation. The suggested multimodal approach employs an ensemble of deep convolutional neural-network based on an attention mechanism that extracts representative features and effectively determines if a video is fake or real. We evaluated the proposed approach on several benchmark multimodal deepfake datasets including FakeAVCeleb, DFDC-p, and DF-TIMIT. Experimental results demonstrate that an ensemble of deep learners based on unimodal and cross-modal network mechanisms exploit highly semantic information between audio and visual signals and outperforms independently trained audio and visual classifiers. Moreover, it can effectively identify different unseen types of deepfakes as well as robust under various post-processing attacks. The results confirm that our approach outperforms existing unimodal/multimodal classifiers for audio-visual manipulated video identification.
Overview
CoChat Summary
## TL;DR
The paper proposes an ensemble multimodal framework (AVDFDNet) for audio-visual deepfake detection that combines a video network, an audio network, and a cross-modal audio-visual network with hard voting. On the FakeAVCeleb dataset, the proposed model reports 94.08% accuracy and also shows cross-dataset generalization and robustness to perturbations.
## Key findings
- On FakeAVCeleb, the proposed AVDFDNet achieved 94.08% accuracy, with 92.91% precision, 95.43% recall, and 94.15% F1-score; this outperformed the video network (90.79%), audio network (83.40%), and audio-visual network (92.64%).
- The audio-visual network alone outperformed the unimodal models on FakeAVCeleb, reaching 92.64% accuracy versus 90.79% for video-only and 83.40% for audio-only.
- On different manipulation types in FakeAVCeleb, the model reported its best accuracies on Faceswap (97.51%), Faceswap-wav2lip (98.11%), and Fsgan-wav2lip (95.33%); lower accuracies were reported for Fsgan (86.94%) and wav2lip (88.42%).
- In cross-dataset evaluation, the model achieved accuracy greater than 75% on both DFDC and DF-TIMIT, including 88.44% on DF-TIMIT(HQ) and 85.03% on DF-TIMIT(LQ); on DFDC it reported 76.62%.
- Under perturbation testing on FakeAVCeleb test videos, accuracy dropped from 94.08% to 88.24%, and after retraining with data augmentation it increased to 91.81%.
## Methods
- The study frames audio-video manipulation detection as a binary classification problem and builds an ensemble AVDFDNet with three components: a 3D CNN-based video network with spatial-temporal attention, a ResNet-50-based audio network on mel-spectrograms, and a cross-modal audio-visual network using cross-attention and hybrid fusion.
- Data come from FakeAVCeleb (reported as 500 real videos and 19,500 manipulated videos), DFDC, and DF-TIMIT; the paper says it supplemented FakeAVCeleb with 1,600 real videos from VoxCeleb1 and used an 85:15 train-test split for selected videos.
- Training used TensorFlow/Keras, AdamW, binary cross entropy loss, 2-second segments, 20 FPS video sampling, and 16 kHz audio sampling; evaluation used precision, recall, F1-score, accuracy, and AUC, plus cross-dataset and robustness tests.
## Limitations
- The paper states that DFDC is challenging because it contains partially forged videos, varied poses/lighting/backgrounds, and labels videos as fake even when only the voice is replaced; this helps explain the lower cross-dataset accuracy (76.62%) on DFDC.
- The paper notes that DF-TIMIT contains videos manipulated through face-swap with frontal pose without occlusion or rotation, which makes them easier to detect; performance was lower on DF-TIMIT(LQ) (85.03%) than DF-TIMIT(HQ) (88.44%).
- Evident from the text: some reported performance details are incomplete or inconsistent in the pasted tables/sections, and the abstract is truncated mid-sentence; also, the paper’s evaluation relies heavily on publicly available benchmark datasets, so real-world generalization beyond those datasets is not directly established in the text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
27 Valid · 6 Not valid · 6 Not checked · 69 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“It does not indicate whether the visual component or speech is manipulated in the video.”
What the sources say
- Disputedcites [9] B. Dolhansky, R. Howes, B. Pflaum, N. Baram, and C. C. Ferrer, "The Deepfake Detection Challenge (DFDC) Preview Dataset," arXiv preprint arXiv:1910.08854, 2019. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says the dataset’s labels describe whether videos were generated using facial manipulation techniques. Dataset Construction (p. 3) also says its metadata lists swapped and target identities for each video. Thus, the paper does indicate visual face manipulation, contrary to the claim; it does not discuss speech-manipulation labels. (LLM, conf=0.83)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“From the results, it can be noticed that using both audio and video modality for the deepfake video detection task generalizes extremely well as compared to utilizing only a single modality (visual information) [27].”
What the sources say
- Disputedcites [27] A. Haliassos, K. Vougioukas, S. Petridis, and M. Pantic, "Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5039–5049. sourceExperimental — accuracy not yet measured
The Related Work section (p. 2) explicitly says LipForensics is visual-only and does not use audio during training or inference. Its cross-dataset results (Section 4.2, p. 5) show that this visual-only method generalizes well against the tested baselines, not that an audio-and-video method outperforms a visual-only one. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Existing low-level approaches concentrate on inconsistent behaviors, including eye blinking [14], facial expression, head movement [15, 16], lip movement [27, 28], gestural mannerisms [29], and misalignment between the eyes [21].”
What the sources say
- Not checked — no full text of the sourcecites [14] Y. Li, M.-C. Chang, and S. Lyu, "In ictu oculi: Exposing ai created fake videos by detecting eye blinking," in 2018 IEEE International Workshop on Information Forensics and Security (WIFS), 2018, pp. 1–7: IEEE. source
- Not checked — no full text of the sourcecites [15] X. Yang, Y. Li, and S. Lyu, "Exposing deep fakes using inconsistent head poses," in 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 8261–8265: IEEE. source
- Not matchedcites [16] S. Agarwal, H. Farid, Y. Gu, M. He, K. Nagano, and H. Li, "Protecting World Leaders Against Deep Fakes," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 38–45.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Partly supportedcites [27] A. Haliassos, K. Vougioukas, S. Petridis, and M. Pantic, "Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5039–5049. sourceExperimental — accuracy not yet measured
The citation is attached to lip movement, not the other behaviors in the list. The Overview (p. 3) confirms that the paper detects anomalous mouth movements, but explicitly describes these cues as semantically high-level and says low-level patterns are likely insufficient. Thus it supports the lip-movement example, not its classification as a low-level approach. (LLM, conf=0.88)
- Not matchedcites [28] S. Agarwal, H. Farid, O. Fried, and M. Agrawala, "Detecting deep-fake videos from phoneme-viseme mismatches," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020, pp. 660–661.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Another area of research makes use of biometric signals to identify unique facial motion patterns in a particular person, however, the ability of such ID-specific methods to generalize to new identities is limited [46].”
What the sources say
- Partly supportedcites [46] D. Cozzolino, A. Rössler, J. Thies, M. Nießner, and L. Verdoliva, "ID-Reveal: Identity-aware DeepFake Video Detection," presented at the International Conference on Computer Vision, Montreal, QC, Canada, 2021. sourceExperimental — accuracy not yet measured
The “Identity-based features” section describes methods that use individual biometric traits, including distinctive facial and head movements. It does not say that these methods have limited ability to generalize to new identities; the “Generalization and robustness analysis” section instead discusses generalization across manipulation methods. Page numbers are unavailable in our copy. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The FakeAVCeleb is a recently released multimodal dataset having a total of 500 real videos and 19,500 manipulated videos.”
What the sources say
- Supportedcites [36] H. Khalid, S. Tariq, M. Kim, and S. S. Woo, "FakeAVCeleb: a novel audio-video multimodal deepfake dataset," in Thirty-fifth Conference on Neural Information Processing Systems, 2021. source
Section 3 (p. 4) says FakeAVCeleb includes 500 real videos. Section 3.1 (pp. 6–7) reports 500 fake-audio/real-video, 9,000 real-audio/fake-video, and 10,000 fake-audio/fake-video samples, totaling 19,500 manipulated videos. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
65 more claims · 14 with their cited sources checked
Create a free account to see all 108 claims and to verify the 5 references still open.
- Probably not a substantial claim. 6
- It may state two separate claims as one. 35
Citations
Cites
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
A Closer Look at Spatiotemporal Convolutions for Action Recognition
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2017) · arXiv · doi:10.48550/arxiv.1711.11248 · Cited by 3,664
A Closer Look at Spatiotemporal Convolutions for Action Recognition
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., & Paluri, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00675 · Cited by 3,516
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791
VoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210
MesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Deepfake Video Detection Using Recurrent Neural Networks
Guera, D., & Delp, E. J. (2018) · 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2018.8639163 · Cited by 1,285
Exposing Deep Fakes Using Inconsistent Head Poses
Yang, X., Li, Y., & Lyu, S. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683164 · Cited by 1,196
Research works printed but not linked: 1
- J. Khochare, C. Joshi, B. Yenarkar, S. Suratkar, and F. Kazi, "A Deep Learning Framework for Audio Deepfake Detection," Arabian Journal for Science and Engineering, vol. 1, pp. 1–12, 2021.
Cited by
Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization
Katamneni, V. S., & Rattani, A. (2024) · 2024 IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb62174.2024.10744480 · Cited by 20
A review of deep learning based multimodal forgery detection for video and audio
Tan, D., Yang, Y., Niu, C., Li, S., Yang, D., & Tan, B. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07629-3 · Cited by 23
America, U. S. O., & Polagani, S. S. (2025) · International Journal of Innovative Research in Science Engineering and Technology · doi:10.15680/ijirset.2025.1406001 · Cited by 1
Multi-modal Deepfake Detection: A Hybrid Approach Utilizing Video, Audio, and Textual Features
Bhatiya, U. K., Dixit, A., & Dixit, R. (2026) · Lecture notes in networks and systems · doi:10.1007/978-981-96-9239-2_28
Deepfake Attacks Detection and Mitigation for Strengthening Cybersecurity: A Comparative AssessmentPreprint
Wairagade, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5093293
Metadata
- Authors
- Masood, M., Javed, A., & Irtaza, A.
- Affiliations
- University of Engineering and Technology
- Corresponding author
- Javed, Ali
- Venue
- Preprint (Research Square)
- Publisher
- Research Square Platform LLC
- Published
- 2023-10-11
- OpenAlex
- W4387527117
- Cited by
- 6
- References
- 77
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- Research Square (acceptedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- deepfake detection, multimodal learning, attention mechanism, ensemble convolutional neural networks, cross-modal learning, sound manipulation, face manipulation, post-processing attacks, synthetic speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection. https://app.cochat.ai/papers/10.21203/rs.3.rs-3415144/v1
- The paper
- Masood, M., Javed, A., & Irtaza, A. (2023). Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection. Preprint (Research Square). https://doi.org/10.21203/rs.3.rs-3415144/v1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.