CoChat

M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection

Open access
Wang, Junke, Wu, Zuxuan, Ouyang, Wenhao, Han, Xintong, Chen, Jingjing, Jiang, Yu-Gang, Li, Ser-Nam · 2022 · Proceedings of the 2022 International Conference on Multimedia Retrieval

The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.

Abstract

The widespread dissemination of Deepfakes demands effective approaches that can detect perceptually convincing forged images. In this paper, we aim to capture the subtle manipulation artifacts at different scales using transformer models. In particular, we introduce a Multi-modal Multi-scale TRansformer (M2TR), which operates on patches of different sizes to detect local inconsistencies in images at different spatial levels. M2TR further learns to detect forgery artifacts in the frequency domain to complement RGB information through a carefully designed cross modality fusion block. In addition, to stimulate Deepfake detection research, we introduce a high-quality Deepfake dataset, SR-DF, which consists of 4,000 DeepFake videos generated by state-of-the-art face swapping and facial reenactment methods. We conduct extensive experiments to verify the effectiveness of the proposed method, which outperforms state-of-the-art Deepfake detection methods by clear margins.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3512527.3531415

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 72 references

Research works printed but not linked: 11

  1. François Chollet . 2017 . Xception: Deep learning with depthwise separable convolutions. In CVPR. François Chollet. 2017. Xception: Deep learning with depthwise separable convolutions. In CVPR.
  2. Jiankang Deng , Jia Guo , Evangelos Ververas , Irene Kotsia , and Stefanos Zafeiriou . 2020 . Retinaface: Single-shot multi-level face localisation in the wild. In CVPR. Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020. Retinaface: Single-shot multi-level face localisation in the wild. In CVPR.
  3. Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.
  4. Yinan He Bei Gan Siyu Chen Yichun Zhou Guojun Yin Luchuan Song Lu Sheng Jing Shao and Ziwei Liu. 2021. ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis. In CVPR. Yinan He Bei Gan Siyu Chen Yichun Zhou Guojun Yin Luchuan Song Lu Sheng Jing Shao and Ziwei Liu. 2021. ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis. In CVPR.
  5. Andrew Howard , Mark Sandler , Grace Chu , Liang-Chieh Chen , Bo Chen , Mingxing Tan , Weijun Wang , Yukun Zhu , Ruoming Pang , Vijay Vasudevan , et al . 2019 . Searching for mobilenetv3. In ICCV. Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al . 2019. Searching for mobilenetv3. In ICCV.
  6. Ira Kemelmacher-Shlizerman . 2016. Transfiguring portraits. ACM TOG ( 2016 ). Ira Kemelmacher-Shlizerman. 2016. Transfiguring portraits. ACM TOG (2016).
  7. Zhengzhe Liu Xiaojuan Qi and Philip HS Torr. 2020. Global texture enhancement for fake face detection in the wild. In CVPR. Zhengzhe Liu Xiaojuan Qi and Philip HS Torr. 2020. Global texture enhancement for fake face detection in the wild. In CVPR.
  8. Karen Simonyan and Andrew Zisserman. [n.d.]. Very deep convolutional networks for large-scale image recognition. In ICLR. Karen Simonyan and Andrew Zisserman. [n.d.]. Very deep convolutional networks for large-scale image recognition. In ICLR.
  9. Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In ICML. Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In ICML.
  10. Tete Xiao , Piotr Dollar , Mannat Singh, Eric Mintun, Trevor Darrell, and Ross Girshick. 2021 . Early Convolutions Help Transformers See Better. In NIPS. Tete Xiao, Piotr Dollar, Mannat Singh, Eric Mintun, Trevor Darrell, and Ross Girshick. 2021. Early Convolutions Help Transformers See Better. In NIPS.
  11. Bojia Zi Minghao Chang Jingjing Chen Xingjun Ma and Yu-Gang Jiang. 2020. WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection. In ACM MM. Bojia Zi Minghao Chang Jingjing Chen Xingjun Ma and Yu-Gang Jiang. 2020. WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection. In ACM MM.

Cited by

See all 529

Metadata

Authors
Wang, J., Wu, Z., Ouyang, W., Han, X., Chen, J., Jiang, Y.-G., & Li, S.-N.
Affiliations
  • Fudan University (CN)
  • Huya Inc, shenzhen, China
  • Meta AI, Newyork, NY, USA
Venue
Proceedings of the 2022 International Conference on Multimedia Retrieval
Publisher
ACM
Published
2022-06-27
Pages
615-623
OpenAlex
W3154326567
Cited by
368
References
64
Citations by year
2026: 66 · 2025: 110 · 2024: 90 · 2023: 67 · 2022: 31
FWCI
27.02
Type
conference paper
Language
en
Open access
Locations
  • International Conference on Multimedia Retrieval (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Natural Science Foundation (62072116)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
Keywords
deepfake detection, multi-scale transformer, multimodal fusion, frequency-domain features, cross-modal fusion, face swapping, video forgery detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection. https://app.cochat.ai/papers/10.1145/3512527.3531415
The paper
Wang, J., Wu, Z., Ouyang, W., Han, X., Chen, J., Jiang, Y.-G., & Li, S.-N. (2022). M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection. Proceedings of the 2022 International Conference on Multimedia Retrieval. https://doi.org/10.1145/3512527.3531415

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.