M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection
Open accessThe DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
Abstract
The widespread dissemination of Deepfakes demands effective approaches that can detect perceptually convincing forged images. In this paper, we aim to capture the subtle manipulation artifacts at different scales using transformer models. In particular, we introduce a Multi-modal Multi-scale TRansformer (M2TR), which operates on patches of different sizes to detect local inconsistencies in images at different spatial levels. M2TR further learns to detect forgery artifacts in the frequency domain to complement RGB information through a carefully designed cross modality fusion block. In addition, to stimulate Deepfake detection research, we introduce a high-quality Deepfake dataset, SR-DF, which consists of 4,000 DeepFake videos generated by state-of-the-art face swapping and facial reenactment methods. We conduct extensive experiments to verify the effectiveness of the proposed method, which outperforms state-of-the-art Deepfake detection methods by clear margins.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“These two evaluation metrics commonly have been used in existing deepfake detection tasks [17,61,62].”
From DFDT: An End-to-End DeepFake Detection Framework Using Vision Transformer (2022) · cites this paper as [61]
The cited source says what the claim attributes to it.
In the citing paper
“Therefore, by combining the audio and visual modalities, the proposed model can learn richer representations than others that use either unimodal approaches or multimodal models whose modalities are exclusively from the visual stream [23, 27].”
From Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection (2023) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Also, we see that there are several works that rely on mask data for the detection, e.g. , Multi-task [28], Face X-ray [20], M2TR [47], etc .”
From DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection (2023) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
(2008) · Journal of Machine Learning Research · Cited by 33,051
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Research works printed but not linked: 11
- François Chollet . 2017 . Xception: Deep learning with depthwise separable convolutions. In CVPR. François Chollet. 2017. Xception: Deep learning with depthwise separable convolutions. In CVPR.
- Jiankang Deng , Jia Guo , Evangelos Ververas , Irene Kotsia , and Stefanos Zafeiriou . 2020 . Retinaface: Single-shot multi-level face localisation in the wild. In CVPR. Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020. Retinaface: Single-shot multi-level face localisation in the wild. In CVPR.
- Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.
- Yinan He Bei Gan Siyu Chen Yichun Zhou Guojun Yin Luchuan Song Lu Sheng Jing Shao and Ziwei Liu. 2021. ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis. In CVPR. Yinan He Bei Gan Siyu Chen Yichun Zhou Guojun Yin Luchuan Song Lu Sheng Jing Shao and Ziwei Liu. 2021. ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis. In CVPR.
- Andrew Howard , Mark Sandler , Grace Chu , Liang-Chieh Chen , Bo Chen , Mingxing Tan , Weijun Wang , Yukun Zhu , Ruoming Pang , Vijay Vasudevan , et al . 2019 . Searching for mobilenetv3. In ICCV. Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al . 2019. Searching for mobilenetv3. In ICCV.
- Ira Kemelmacher-Shlizerman . 2016. Transfiguring portraits. ACM TOG ( 2016 ). Ira Kemelmacher-Shlizerman. 2016. Transfiguring portraits. ACM TOG (2016).
- Zhengzhe Liu Xiaojuan Qi and Philip HS Torr. 2020. Global texture enhancement for fake face detection in the wild. In CVPR. Zhengzhe Liu Xiaojuan Qi and Philip HS Torr. 2020. Global texture enhancement for fake face detection in the wild. In CVPR.
- Karen Simonyan and Andrew Zisserman. [n.d.]. Very deep convolutional networks for large-scale image recognition. In ICLR. Karen Simonyan and Andrew Zisserman. [n.d.]. Very deep convolutional networks for large-scale image recognition. In ICLR.
- Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In ICML. Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In ICML.
- Tete Xiao , Piotr Dollar , Mannat Singh, Eric Mintun, Trevor Darrell, and Ross Girshick. 2021 . Early Convolutions Help Transformers See Better. In NIPS. Tete Xiao, Piotr Dollar, Mannat Singh, Eric Mintun, Trevor Darrell, and Ross Girshick. 2021. Early Convolutions Help Transformers See Better. In NIPS.
- Bojia Zi Minghao Chang Jingjing Chen Xingjun Ma and Yu-Gang Jiang. 2020. WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection. In ACM MM. Bojia Zi Minghao Chang Jingjing Chen Xingjun Ma and Yu-Gang Jiang. 2020. WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection. In ACM MM.
Cited by
DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation
Lin, A., Chen, B., Xu, J., Zhang, Z., Lu, G., & Zhang, D. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3178991 · Cited by 882
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543
Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting
Chen, P., Zhang, Y., Cheng, Y., Shu, Y., Wang, Y., Wen, Q., Yang, B., & Guo, C. (2024) · VBN Forskningsportal (Aalborg Universitet) · doi:10.48550/arxiv.2402.05956 · Cited by 504
CIFAKE: Image Classification and Explainable Identification of AI-Generated Synthetic Images
Bird, J. J., & Lotfi, A. (2024) · IEEE Access · doi:10.1109/access.2024.3356122 · Cited by 265
UCF: Uncovering Common Features for Generalizable Deepfake Detection
Yan, Z., Zhang, Y., Fan, Y., & Wu, B. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02048 · Cited by 241
“Other notable directions are focusing on some specific representations, i.e., forgery region location [33], neuron behaviors [49], optical flow [4], landmark geometric features [42], 3D decomposition [59], erasing technology [47], and attentional networks [10, 55, 48].”
Background
Metadata
- Authors
- Wang, J., Wu, Z., Ouyang, W., Han, X., Chen, J., Jiang, Y.-G., & Li, S.-N.
- Affiliations
- Fudan University (CN)
- Huya Inc, shenzhen, China
- Meta AI, Newyork, NY, USA
- Venue
- Proceedings of the 2022 International Conference on Multimedia Retrieval
- Publisher
- ACM
- Published
- 2022-06-27
- Pages
- 615-623
- OpenAlex
- W3154326567
- Cited by
- 368
- References
- 64
- Citations by year
- 2026: 66 · 2025: 110 · 2024: 90 · 2023: 67 · 2022: 31
- FWCI
- 27.02
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference on Multimedia Retrieval (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation (62072116)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Keywords
- deepfake detection, multi-scale transformer, multimodal fusion, frequency-domain features, cross-modal fusion, face swapping, video forgery detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection. https://app.cochat.ai/papers/10.1145/3512527.3531415
- The paper
- Wang, J., Wu, Z., Ouyang, W., Han, X., Chen, J., Jiang, Y.-G., & Li, S.-N. (2022). M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection. Proceedings of the 2022 International Conference on Multimedia Retrieval. https://doi.org/10.1145/3512527.3531415
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.