Glitch in the matrix: A large scale benchmark for content driven audio–visual forgery detection and localization
Open accessAbstract
Most deepfake detection methods focus on detecting spatial and/or spatio-temporal changes in facial attributes and are centered around the binary classification task of detecting whether a video is real or fake. This is because available benchmark datasets contain mostly visual-only modifications present in the entirety of the video. However, a sophisticated deepfake may include small segments of audio or audio-visual manipulations that can completely change the meaning of the video content. To addresses this gap, we propose and benchmark a new dataset, Localized Audio Visual DeepFake (LAV-DF), consisting of strategic content-driven audio, visual and audio-visual manipulations. The proposed baseline method, Boundary Aware Temporal Forgery Detection (BA-TFD), is a 3D Convolutional Neural Network-based architecture which effectively captures multimodal manipulations. We further improve (i.e. BA-TFD+) the baseline method by replacing the backbone with a Multiscale Vision Transformer and guide the training process with contrastive, frame classification, boundary matching and multimodal boundary matching loss functions. The quantitative analysis demonstrates the superiority of BA-TFD+ on temporal forgery localization and deepfake detection tasks using several benchmark datasets including our newly proposed dataset. The dataset, models and code are available at https://github.com/ControlNet/LAV-DF.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing temporal forgery localization methods have limited generalization capabilities, especially across different languages.
“the generalization capabilities of current TFL methods are limited, especially across different languages”
From CLFormer: a cross-lingual transformer framework for temporal forgery localization (2025) · cites this paper as [4]
Zero-shot voice cloning methods such as SV2TTS utilized by previous datasets have low signal-to-noise ratio resulting in low-quality audio manipulations that are easily localized by BA-TFD and UMMAFormer.
“Zero-shot voice cloning methods such as SV2TTS [27] utilized by previous datasets [6, 31] have low signal-to-noise ratio resulting in lowquality audio manipulations that are easily localized by BA-TFD [4] and UMMAFormer [75].”
From AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset (2024) · cites this paper as [4]
BA-TFD+ AP@0.5 drops from 96.30 on LAV-DF to 44.42 on AV-Deepfake1M.
“A similar pattern is also observed for BA-TFD+ [4] (AP@0.5 96.30 → 44.42)”
From AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset (2024) · cites this paper as [4]
The quality and scale of LAV-DF are limited, and state-of-the-art methods designed for temporal localization are already achieving very strong performance.
“However, the quality and scale of LAV-DF are limited, and the state-of-the-art methods designed for temporal localization [4, 75] are already achieving very strong performance.”
From AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset (2024) · cites this paper as [4]
Deepfakes have become more content-driven, generating partially fake media.
“Given that deepfakes have become more content-driven generating partially fake media [9, 8, 64], integrated deepfake detection as well as localization is crucial.”
From Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization (2024) · cites this paper as [9]
Deepfake localization approaches are either anchor-based or boundary-prediction based.
“Deepfake localization approaches are either anchor-based [21, 22] that utilize a sliding window approach to detect deepfake segments or boundary prediction based [6, 10, 9, 64] that focuses on predicting the boundary of fake segments in a video using boundary matching loss.”
From Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization (2024) · cites this paper as [9]
BA-TFD+ obtained the second-best localization performance on LAV-DF with AP@0.5 of 96.30 and AR@50 of 80.48.
“BA-TFD+ has obtained the second-best performance with an AP@0.5 and AR@50 of 96 . 30 and 80 . 48 , respectively.”
From Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization (2024) · cites this paper as [9]
BA-TFD+ showed a smaller performance difference compared to MMMS-BA at lower threshold levels but a larger performance difference at higher thresholds, indicating less precise localization.
“Although BA-TFD+ showed a smaller performance difference compared to MMMS-BA at lower threshold levels, it exhibited a larger performance difference at higher thresholds, indicating a less precise localization.”
From Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization (2024) · cites this paper as [9]
The citation lists haven’t been fetched yet
References 119 · Cited by 63
- Authors
- Cai, Z., Ghosh, S., Dhall, A., Gedeon, T., Stefanov, K., & Hayat, M.
- Affiliations
- Monash University (AU)
- Curtin University (AU)
- Indian Institute of Technology Ropar (IN)
- Corresponding author
- Cai, Zhixi
- Venue
- Computer Vision and Image Understanding
- Publisher
- Elsevier BV
- Published
- 2023-11
- Volume
- 236
- Pages
- 103818
- ISSN
- 1077-3142 (ISSN-L)
- OpenAlex
- W4386102876
- Cited by
- 63
- References
- 119
- Citations by year
- 2026: 23 · 2025: 26 · 2024: 13 · 2023: 1
- FWCI
- 5.73
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Computer Vision and Image Understanding (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Monash University Research Portal (Monash University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- deepfake detection, 3D convolutional neural networks, contrastive learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
