UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization
Open accessAbstract
The emergence of artificial intelligence-generated content (AIGC) has raised concerns about the authenticity of multimedia content in various fields. However, existing research for forgery content detection has focused mainly on binary classification tasks of complete videos, which has limited applicability in industrial settings. To address this gap, we propose UMMAFormer, a novel universal transformer framework for temporal forgery localization (TFL) that predicts forgery segments with multimodal adaptation. Our approach introduces a Temporal Feature Abnormal Attention (TFAA) module based on temporal feature reconstruction to enhance the detection of temporal differences. We also design a Parallel Cross-Attention Feature Pyramid Network (PCA-FPN) to optimize the Feature Pyramid Network (FPN) for subtle feature enhancement. To evaluate the proposed method, we contribute a novel Temporal Video Inpainting Localization (TVIL) dataset specifically tailored for video inpainting scenes. Our experiments show that our approach achieves state-of-the-art performance on benchmark datasets, including Lav-DF, TVIL, and Psynd, significantly outperforming previous methods. The code and data are available at https://github.com/ymhzyj/UMMAFormer/.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
On AV-Deepfake1M, CLFormer improves over UMMAFormer by 14.42% in AP at tIoU of 0.95 and by 6.71% in average AP.
“on AV-Deepfake1M, CLFormer demonstrates a significant improvement over the second-best method, UMMAFormer [6], with a 14.42% increase in AP at tIoU of 0.95 and a 6.71% increase in the average AP.”
From CLFormer: a cross-lingual transformer framework for temporal forgery localization (2025) · cites this paper as [6]
Existing temporal forgery localization methods have limited generalization capabilities, especially across different languages.
“the generalization capabilities of current TFL methods are limited, especially across different languages”
From CLFormer: a cross-lingual transformer framework for temporal forgery localization (2025) · cites this paper as [6]
Zero-shot voice cloning methods such as SV2TTS utilized by previous datasets have low signal-to-noise ratio resulting in low-quality audio manipulations that are easily localized by BA-TFD and UMMAFormer.
“Zero-shot voice cloning methods such as SV2TTS [27] utilized by previous datasets [6, 31] have low signal-to-noise ratio resulting in lowquality audio manipulations that are easily localized by BA-TFD [4] and UMMAFormer [75].”
From AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset (2024) · cites this paper as [75]
The quality and scale of LAV-DF are limited, and state-of-the-art methods designed for temporal localization are already achieving very strong performance.
“However, the quality and scale of LAV-DF are limited, and the state-of-the-art methods designed for temporal localization [4, 75] are already achieving very strong performance.”
From AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset (2024) · cites this paper as [75]
UMMAFormer AP@0.5 drops from 98.83 on LAV-DF to 51.64 on AV-Deepfake1M.
“and UMMAFormer [75] (AP@0.5 98.83 → 51.64).”
From AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset (2024) · cites this paper as [75]
Cites
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · see citing works
Deep High-Resolution Representation Learning for Visual Recognition
Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2983686 · Cited by 4,894 · see citing works
SlowFast Networks for Video Recognition
Feichtenhofer, C., Fan, H., Malik, J., & He, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00630 · Cited by 4,007 · see citing works
ActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578 · see citing works
Soft-NMS — Improving Object Detection with One Line of Code
Bodla, N., Singh, B., Chellappa, R., & Davis, L. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.593 · Cited by 2,274 · cited 1× · see citing works
“For the inference stage, we applied Soft-NMS [4] to post-process the results and remove a large number of redundant predictions.”
MethodDiffusion Models: A Comprehensive Survey of Methods and Applications
Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., & Yang, M.-H. (2024) · ACM Computing Surveys · doi:10.1145/3626235 · Cited by 1,704 · see citing works
End-to-End Reconstruction-Classification Learning for Face Forgery Detection
Cao, J., Ma, C., Yao, T., Chen, S., Ding, S., & Yang, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00408 · Cited by 428 · see citing works
The deepfake detection challenge (dfdc) dataset
Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., & Ferrer, C. C. (2020) · arXiv · doi:10.48550/arxiv.2006.07397 · Cited by 427 · see citing works
The Deepfake Detection Challenge (DFDC) Preview Dataset
Dolhansky, B., Howes, R., Pflaum, B., Baram, N., & Ferrer, C. C. (2019) · arXiv · doi:10.48550/arxiv.1910.08854 · Cited by 413 · see citing works
Combining EfficientNet and Vision Transformers for Video Deepfake Detection
Coccomini, D. A., Messina, N., Gennaro, C., & Falchi, F. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-06433-3_19 · Cited by 296 · see citing works
Joint Audio-Visual Deepfake Detection
Zhou, Y., & Lim, S.-N. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01453 · Cited by 217 · see citing works
Self-citationCai, Z., Stefanov, K., Dhall, A., & Hayat, M. (2022) · 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta56598.2022.10034605 · Cited by 99 · see citing works
Noise Based Deepfake Detection via Multi-Head Relative-Interaction
Wang, T., & Chow, K. P. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26701 · Cited by 93 · cited 1× · see citing works
“These approaches explore various features, including noise features [9, 64], local texture characteristics [7, 63], and frequency domain anomalies [28, 44], to enhance detection capabilities.”
BackgroundWaveFake: A Data Set to Facilitate Audio Deepfake Detection
Frank, J., & Schönherr, L. (2021) · arXiv · doi:10.48550/arxiv.2111.02813 · Cited by 76 · see citing works
Deepfake Video Detection with Spatiotemporal Dropout Transformer
Zhang, D., Lin, F., Hua, Y., Wang, P., Zeng, D., & Ge, S. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3547913 · Cited by 50 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset
Cai, Z., Ghosh, S., Adatia, A. P., Hayat, M., Dhall, A., Gedeon, T., & Stefanov, K. (2024) · Proceedings of the 32nd ACM International Conference on Multimedia · doi:10.1145/3664647.3680795 · Cited by 59 · see citing works
Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization
Katamneni, V. S., & Rattani, A. (2024) · 2024 IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb62174.2024.10744480 · Cited by 20 · see citing works
Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges
Liu, P., Tao, Q., & Zhou, J. T. (2024) · arXiv · doi:10.48550/arxiv.2406.06965 · Cited by 6 · see citing works
Vigo: Audiovisual Fake Detection and Segment Localization
Pérez-Vieites, D., Moreira-Pérez, J. J., Aragón-Kifute, Á., Román-Sarmiento, R., & Castro-González, R. (2024) · Proceedings of the 32nd ACM International Conference on Multimedia · doi:10.1145/3664647.3688983 · Cited by 5 · see citing works
Liu, P., Tao, Q., & Zhou, J. (2024) · Proceedings of the 1st ACM Multimedia Workshop on Multi-modal Misinformation Governance in the Era of Foundation Models · doi:10.1145/3689090.3689386 · Cited by 3 · see citing works
CLFormer: a cross-lingual transformer framework for temporal forgery localization
Cheng, H., Liu, H., Cai, J., & Ye, L. (2025) · Visual Intelligence · doi:10.1007/s44267-025-00084-z · Cited by 2 · see citing works
Self-citationAnshul, A., Gopal, S., Rajan, D., & Chng, E. S. (2025) · 2025 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01283 · Cited by 1 · see citing works
- Authors
- Zhang, R., Wang, H., Du, M., Liu, H., Zhou, Y., & Zeng, Q.
- Affiliations
- Sichuan University (CN)
- Venue
- Proceedings of the 31st ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2023-10-26
- Pages
- 8749-8759
- OpenAlex
- W4386272941
- Cited by
- 42
- References
- 15
- Citations by year
- 2026: 11 · 2025: 18 · 2024: 13
- FWCI
- 4.49
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Sichuan Science and Technology Program (2022YFG0320)
- National Natural Science Foundation of China (62272331,61972269)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Keywords
- video forgery detection, transformer models, AI-generated content, video inpainting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
