The DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index; 18 of the 41 sources its claims cite were checked for support.
Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization
Open accessAbstract
In the digital age, the emergence of deepfakes and synthetic media presents a significant threat to societal and political integrity. Deepfakes based on multi-modal manipulation, such as audio-visual, are more realistic and pose a greater threat. Current multi-modal deepfake detectors are often based on the attention-based fusion of heterogeneous data streams from multiple modalities. However, the heterogeneous nature of the data (such as audio and visual signals) creates a distributional modality gap and poses a significant challenge in effective fusion and hence multi-modal deepfake detection. In this paper, we propose a novel multi-modal attention framework based on recurrent neural networks (RNNs) that leverages contextual information for audio-visual deepfake detection. The proposed approach applies attention to multi-modal multi-sequence representations and learns the contributing features among them for deepfake detection and localization. Thorough experimental validations on audio-visual deepfake datasets, namely FakeAVCeleb, AV-Deepfake1M, TVIL, and LAV-DF datasets, demonstrate the efficacy of our approach. Cross-comparison with the published studies demonstrates superior performance of our approach with an improved accuracy and precision by 3.47% and 2.05% in deepfake detection and localization, respectively. Thus, obtaining state-of-the-art performance. To facilitate reproducibility, the code and the datasets information is available at https://github.com/vcbsl/audiovisual-deepfake/.
Overview
CoChat Summary
TL;DR
The paper proposes MMMS-BA, a recurrent neural network-based multi-modal multi-sequence bi-modal attention framework for audio-visual deepfake detection and localization. It reports state-of-the-art performance on FakeAVCeleb, AV-Deepfake1M, LAV-DF, and TVIL, with improved detection and localization over published methods.
Key findings
- On intra-dataset evaluation, MMMS-BA achieved AUC/ACC of 0.989/0.979 on FakeAVCeleb and 0.979/0.968 on AV-Deepfake1M; average intra-dataset AUC/ACC was 0.984/0.973. Cross-dataset average AUC/ACC was 0.944/0.919.
- On FakeAVCeleb, MMMS-BA outperformed published audio-visual detectors with AUC 0.989 and ACC 0.979, an average increment of 0.0341 AUC and 3.47% ACC; the abstract reports improved accuracy and precision by 3.47% and 2.05% for detection and localization, respectively.
- For localization, MMMS-BA obtained the best AP@0.5/AR@50 on AV-Deepfake1M (62.75/57.49), LAV-DF (97.56/93.45), and TVIL (96.87/90.47); however, UMMAFormer had higher AP@0.75 and AP@0.95 on TVIL (84.70 and 62.43).
- Ablation showed full visual+lip+audio (V+L+A) gave the best AUC 0.989 and EER 0.029, versus V+L AUC 0.814, L+A AUC 0.924, and V+A AUC 0.955; MMMS-BA’s EER 0.029 was lower than MMUS-SA (0.033) and MS-SA (0.045).
Methods
- MMMS-BA processes full visual face, lip, and audio sequences with separate Bi-GRU layers (300 neurons each) and dense layers (100 neurons), dropout 0.3, then computes pairwise bi-modal attention for (V,L), (L,A), and (A,V), multi-sequence attention, multiplicative gating, and concatenation with residual skip connections. Classification uses focal loss; localization uses a regression head with differentiable IoU/DIoU loss and non-maximum suppression.
- Evaluation used FakeAVCeleb, AV-Deepfake1M, TVIL, and LAV-DF. Detection metrics were AUC, pAUC at 10% FPR, and EER; localization metrics were AP and AR at multiple thresholds. For AV-Deepfake1M, the training set was split 85%/15% for training/validation because the test set was not released, and the actual validation set was used for testing.
- Ablations varied the attention mechanism (MMUS-SA and MS-SA) and modality combinations (V+A, V+L, L+A, V+L+A).
Limitations
- Stated: future work will extend the framework to incorporate text analysis and adapt it to account for missing modalities during training and inference. The paper also attributes lower AV-Deepfake1M localization performance to highly realistic content-driven fake content.
- Evident: AV-Deepfake1M’s test set was not released, so the actual validation set was used for testing. Cross-dataset detection performance dropped relative to intra-dataset evaluation, e.g., average AUC/ACC decreases of 0.053/0.076 when trained on FakeAVCeleb and 0.027/0.029 when trained on AV-Deepfake1M.
- Evident: on TVIL, UMMAFormer achieved higher AP@0.75 (84.70) and AP@0.95 (62.43) than MMMS-BA (81.33 and 28.43), indicating weaker high-threshold localization. The paper has no explicit limitations section.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
11 Valid · 5 Not valid · 9 Not checked · 55 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Given that deepfakes have become more content-driven generating partially fake media [9, 8, 64], integrated deepfake detection as well as localization is crucial.”
What the sources say
- Supportedcites [9] source
The Abstract (p. 1) states that 'a sophisticated deepfake may include small segments of audio or audio-visual manipulations that can completely change the meaning of the video content', and the Introduction (p. 1) says fake content 'may only constitute a small part(s) of a long real video' while calling this 'this new type of manipulations' — i.e. content-driven forgeries that leave most of the media real. Section 2.1/Figure 1 (p. 2) and the paper's own contribution (multimodal content-driven temporal forgery localization and detection, LAV-DF) supply exactly the detection-plus-localization framing the citing sentence draws. The only nuance is that the source presents content-driven partial forgery as an emerging type/gap rather than documenting a measured increase over time, which does not alter the attributed content. (LLM, conf=0.80)
- Supportedcites [8] source
The Introduction describes small manipulated segments embedded in otherwise real content. Related Work identifies the emergence of content-driven deepfake datasets, and Section 3 describes this paper’s own content-driven audio-visual manipulations. Together, these support the citing sentence’s description of partially fake, content-driven media. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites [64] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Current multi-modal deepfake detectors are often based on the attention-based fusion of heterogeneous data streams from multiple modalities.”
- Also stated in: § 1. Introduction · p. 1 · plain
“Existing audio-visual deepfake detectors are often based on the fusion of heterogeneous streams using feature concatenation and employing attention mechanism [68, 47, 67, 37, 59, 5, 45].”
What the sources say
- Not matchedcites [68]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [47] source
- Not matchedcites [67]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [37] source
- Could not verifycites [59] sourceExperimental — accuracy not yet measured
Section 3.2 describes attention-based matching between images and text for image generation, not multimodal deepfake detection. Searches for deepfake detectors and audio-visual fusion found no passage addressing whether current detectors often use attention-based fusion. (LLM, conf=0.77)
- Not checked — no full text of the sourcecites [5] source
- Not checked — no full text of the sourcecites [45] source
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1. Introduction · p. 1 · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Existing unimodal deepfake detectors are primarily designed to detect a single type of manipulation, such as visual, acoustic, and text.”
What the sources say
- Not matchedcites [32] Li, Exposing deepfake videos by detecting face warping artifacts
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [23] source
- Partly supportedcites [2] sourceExperimental — accuracy not yet measured
Section 2.2 (page not provided) describes visual deepfake detectors aimed at particular artifacts or face-swap manipulations, which supports part of the claim. It does not characterize unimodal detectors generally or establish that acoustic and text detectors are primarily designed for a single type of manipulation. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [39] source
- Not checked — no full text of the sourcecites [13] source
- Partly supportedcites [24] sourceExperimental — accuracy not yet measured
The “Deepfake Detection: Strengths and Weaknesses” section describes detectors targeting visual artifacts and says systems that identify known creation methods may generalize poorly to others. It does not establish that existing detectors are primarily unimodal or describe detectors separately designed for acoustic and text manipulations; the Introduction says text generation is outside the paper’s review. (LLM, conf=0.78)
- Not checked — no full text of the sourcecites [46] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“The Unsupervised Cross-Modal Inconsistencies [57] model obtained the second-best performance, with an AUC of 0 . 968 .”
What the sources say
- Could not verifycites [57] sourceExperimental — accuracy not yet measured
The Manipulation Detection section compares detection methods and AUCs on other datasets, including Celeb-DF, but does not report a result for FakeAVCeleb. Searches for “Unsupervised Cross-Modal Inconsistencies,” “FakeAVCeleb,” and “0.968” found no passage establishing the model’s AUC or second-best ranking. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“NPVForensics is based on mining the correlation between non-critical phonemes and visemes using a Swin Transformer and cross-modal fusion, but it achieves lower performance over MMMS-BA due to its limited ability to capture temporal dependencies and interactions across multiple sequences.”
What the sources say
- Partly supportedcites [11] Chen, 2023, Npvforensics: Jointing non-critical phonemes and visemes for deepfake detection sourceExperimental — accuracy not yet measured
The Method (p. 3) supports the description of NPVForensics as using a Swin Transformer and cross-modal fusion to mine non-critical phoneme–viseme correlations. The Introduction (p. 2) says it captures long-range dependencies, and the paper does not establish a performance comparison with MMMS-BA or attribute one to the stated limitation. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
37 more claims · 4 with their cited sources checked
Create a free account to see all 80 claims and to verify the 3 references still open.
- Probably not a substantial claim. 11
- It may state two separate claims as one. 26
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.3. Performance of Deepfake Detection · p. 5
“For the AV-Deepfake1M dataset, the model obtained the lowest performance with an AUC and ACC of 0 . 909 and 0 . 893 , respectively.”
§ 4.3. Performance of Deepfake Detection · p. 5
“| AV-Deepfake1M | 0.909 | 0.884 | 0.178 | 0.821 | 0.893 | 0.249 |”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.1. Varying the Attention at Sequence and Modality Level · p. 7
“| MMMS-BA | 0.989 | 0.977 | 0.029 | 0.979 | 0.965 | 0.039 |”
§ 5.2. Performance of MMMS-BA on Different Combination of Modalities · p. 7
“| MMMS-BA (V+L+A) | 0.989 | 0.978 | 0.033 | 0.979 | 0.963 | 0.039 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.5. Deepfake Localization Performance · p. 6
“The Enc-Dec model obtained the lowest performance across all datasets and metrics.”
§ 5.1. Varying the Attention at Sequence and Modality Level · p. 7
“| TVIL | Enc-Dec | 23.08 | 08.45 | 05.32 | 17.16 | 23.47 | 43.98 | 45.18 | | | ActionFormer [63] | 86.27 | 83.03 | 28.17 | 84.82 | 85.77 | 88.10 | 88.49 | | | BA-TFD+ [9] | 76.90 | 38.50 | 0.25 | 66.90 | 64.08 | 60.77 | 58.42 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.4. Comparison with the Published Audio-Visual Deepfake Detectors · p. 6
“The Unsupervised Cross-Modal Inconsistencies [57] model obtained the second-best performance, with an AUC of 0 . 968 .”
§ 4.3. Performance of Deepfake Detection · p. 5
“| Multi-modal contrastive learning [36] | 0.978 | 0.965 |”
Citations
Cites
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556
Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., & Savarese, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00075 · Cited by 6,612
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., & Choo, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00916 · Cited by 4,163
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation
Choi, Y., Choi, M.-J., Kim, M., Ha, J.-W., Kim, S., & Choo, J. (2017) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1711.09020 · Cited by 4,057
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
Soft-NMS — Improving Object Detection with One Line of Code
Bodla, N., Singh, B., Chellappa, R., & Davis, L. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.593 · Cited by 2,274 · cited 1×
“These segments are further processed using Non-Maximum Suppression (NMS) [7] to remove highly overlapping instances, leading to the final localization timestamps.”
MethodVoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923
MesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Research works printed but not linked: 3
- Citron, How deepfakes undermine truth and threaten democracy
- Khalid, 2021, Fakeavceleb: A novel audio-video multimodal deepfake dataset
- 10.1109/ICASSP49357.2023.10095247 · DOI
Cited by
Shahzad, S. A., Hashmi, A., Peng, Y., Tsao, Y., & Wang, H. (2025) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2025.3618409 · Cited by 31
“MMMS-BA framework [77] applies attention over audio-visual sequences to address the modality gap and improve deepfake detection and localization, reaching 97.90% accuracy.”
InfluentialSingh, L., Kumar, A., & Golash, R. (2026) · The European Physical Journal Plus · doi:10.1140/epjp/s13360-026-07296-2 · Cited by 3
CLFormer: a cross-lingual transformer framework for temporal forgery localization
Cheng, H., Liu, H., Cai, J., & Ye, L. (2025) · Visual Intelligence · doi:10.1007/s44267-025-00084-z · Cited by 2
TCMabNet: A Spatio-Temporal Network for Fine-Grained Fake Video
Shi, G., Tan, A., Tao, H., & Du, Y. (2025) · International Conference on Artificial Intelligence Technologies and Applications (ICAITA) · doi:10.1109/icaita67588.2025.11137964 · Cited by 1
“In addition, the article [11] proposes a contextual cross-attention mechanism fusing audio - video modalities for deep forgery detection and localization, achieving SOTA performance on datasets such as FakeAVCeleb.”
GrDT: Towards Robust Deepfake Detection Using Geometric Representation Distribution and Texture
Xie, H., He, H., Fu, B., & Sanchez, V. (2025) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw65960.2025.00083 · Cited by 10
“For example, there are detection meth-ods that rely on capsule networks [48], autoencoders [27], CNNs [4, 49, 64], multi-modal approaches [32, 46, 61] and transformers [20, 55, 59].”
Metadata
- Authors
- Katamneni, V. S., & Rattani, A.
- Affiliations
- University of North Texas (US)
- ORCID
- Rattani, Ajita
- Venue
- 2024 IEEE International Joint Conference on Biometrics (IJCB)
- Publisher
- IEEE
- Published
- 2024-09-15
- Pages
- 1-11
- OpenAlex
- W4404239158
- Cited by
- 20
- References
- 77
- Citations by year
- 2026: 12 · 2025: 8
- FWCI
- 4.11
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Joint Conference on Biometrics (IJCB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- cross-modal attention, multimodal fusion, recurrent neural networks, contextual information, modality gap, synthetic media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization. https://app.cochat.ai/papers/10.1109/ijcb62174.2024.10744480
- The paper
- Katamneni, V. S., & Rattani, A. (2024). Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization. 2024 IEEE International Joint Conference on Biometrics (IJCB). https://doi.org/10.1109/ijcb62174.2024.10744480
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.