M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection
Open accessThe DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
Abstract
The widespread dissemination of Deepfakes demands effective approaches that can detect perceptually convincing forged images. In this paper, we aim to capture the subtle manipulation artifacts at different scales using transformer models. In particular, we introduce a Multi-modal Multi-scale TRansformer (M2TR), which operates on patches of different sizes to detect local inconsistencies in images at different spatial levels. M2TR further learns to detect forgery artifacts in the frequency domain to complement RGB information through a carefully designed cross modality fusion block. In addition, to stimulate Deepfake detection research, we introduce a high-quality Deepfake dataset, SR-DF, which consists of 4,000 DeepFake videos generated by state-of-the-art face swapping and facial reenactment methods. We conduct extensive experiments to verify the effectiveness of the proposed method, which outperforms state-of-the-art Deepfake detection methods by clear margins.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“These two evaluation metrics commonly have been used in existing deepfake detection tasks [17,61,62].”
From DFDT: An End-to-End DeepFake Detection Framework Using Vision Transformer (2022) · cites this paper as [61]
The cited source says what the claim attributes to it.
In the citing paper
“Therefore, by combining the audio and visual modalities, the proposed model can learn richer representations than others that use either unimodal approaches or multimodal models whose modalities are exclusively from the visual stream [23, 27].”
From Attention-based Multimodal learning framework for Generalized Audio-Visual Deepfake Detection (2023) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Also, we see that there are several works that rely on mask data for the detection, e.g. , Multi-task [28], Face X-ray [20], M2TR [47], etc .”
From DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection (2023) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“To address the above limitations, we explore transformers to model the relationships of pixels due to their strong capability of long-term dependency modeling for both natural language processing tasks [12, 46, 58] and computer vision tasks [2, 14, 68].”
BackgroundMethodBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 1×
“To address the above limitations, we explore transformers to model the relationships of pixels due to their strong capability of long-term dependency modeling for both natural language processing tasks [12, 46, 58] and computer vision tasks [2, 14, 68].”
MethodImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“Inspired by this, we use the relu1_1, relu2_1, relu3_1, relu4_1 and relu5_1 of the pretrained VGG-19 network on ImageNet [11] to calculate the perceptual loss between the feature maps of forged faces and that of corresponding real faces.”
MethodAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472 · cited 3×
“To address the above limitations, we explore transformers to model the relationships of pixels due to their strong capability of long-term dependency modeling for both natural language processing tasks [12, 46, 58] and computer vision tasks [2, 14, 68].”
(2008) · Journal of Machine Learning Research · Cited by 33,051
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253 · cited 2×
“To address the above limitations, we explore transformers to model the relationships of pixels due to their strong capability of long-term dependency modeling for both natural language processing tasks [12, 46, 58] and computer vision tasks [2, 14, 68].”
MethodEfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · cited 2×
“For backbone network, we use Efficient-b4 [55] pretrained on ImageNet [12].”
MethodExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 1×
“To address the above limitations, we explore transformers to model the relationships of pixels due to their strong capability of long-term dependency modeling for both natural language processing tasks [14, 48, 61] and computer vision tasks [2, 16, 62, 63, 73].”
MethodQuo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Deformable DETR: Deformable Transformers for End-to-End Object Detection
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., & Dai, J. (2020) · arXiv · doi:10.48550/arxiv.2010.04159 · Cited by 6,027 · cited 1×
“To address the above limitations, we explore transformers to model the relationships of pixels due to their strong capability of long-term dependency modeling for both natural language processing tasks [14, 48, 61] and computer vision tasks [2, 16, 62, 63, 73].”
MethodFaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 10×
“DatasetsWe conduct experiments on FaceForensics++ (FF++) [48], Celeb-DF [35], and the newly proposed SR-DF dataset.”
MethodInfluentialReinhard, E., Adhikhmin, M., Gooch, B., & Shirley, P. (2001) · IEEE Computer Graphics and Applications · doi:10.1109/38.946629 · Cited by 2,937 · cited 1×
“We compare the blending results with naive stitching and a color transfer algorithm [47] adopted by [35], and show an example of synthesized image in Figure 5.”
ResultDlib-ml: A Machine Learning Toolkit
King, D. E. (2009) · doi:10.5555/1577069.1755843 · Cited by 2,777 · cited 3×
“Implementation Details For all real and forgery images, we use dlib [27] to crop the face regions as inputs with a size of 320 × 320.”
MethodInfluentialPWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
Sun, D., Yang, X., Liu, M.-Y., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00931 · Cited by 2,648
VoxCeleb: A Large-Scale Speaker Identification Dataset
Nagrani, A., Chung, J. S., & Zisserman, A. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-950 · Cited by 2,210 · cited 2×
“To process our dataset, we use the pretrained model on VoxCeleb dataset [40], which contains speech videos from speakers spanning a wide range of different ethnic groups, accents, professions and ages, and reenact the faces in our real videos.”
MethodRich Models for Steganalysis of Digital Images
Fridrich, J., & Kodovsky, J. (2012) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2012.2190402 · Cited by 2,144 · cited 1×
“FF++ [48] is a widely used dataset in various Deepfake detection approaches [1, 7, 21, 32, 44, 66].”
MethodLearning Spatio-Temporal Representation with Pseudo-3D Residual Networks
Qiu, Z., Yao, T., & Mei, T. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.590 · Cited by 1,874 · cited 1×
“P3D [45] 80.”
MesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00327 · Cited by 1,846 · cited 8×
“Given that Celeb-DF [37] contains high-quality samples (as discussed in 4.2, Celeb-DF achieves the best results on Mask-SSIM , Perceptual loss and Ewarp metrics in the available Deepfake dataset.)”
BackgroundMethodInfluentialResNeSt: Split-Attention Networks
Zhang, H., Wu, C., Zhang, Z., Zhu, Y., Lin, H., Zhang, Z., Sun, Y., He, T., Mueller, J., Manmatha, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.08955 · Cited by 1,809 · cited 1×
“These methods capture artifacts from the face regions in a single scale with stacked convolutional operations [55, 70].”
MethodFace2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Deferred neural rendering: image synthesis using neural textures
Thies, J., Zollhöfer, M., & Nießner, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323035 · Cited by 1,356 · cited 3×
“The FaceForensics++ (FF++) dataset [48] has 1,000 real videos from YouTube and 4,000 corresponding Deefake videos that are generated with 4 face manipulation methods: Deepfakes [10], FaceSwap [18], Face2Face [55], and NeuralTextures [54].”
BackgroundMethodFace X-Ray for More General Face Forgery Detection
Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., & Guo, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00505 · Cited by 1,211
CNN-Generated Images Are Surprisingly Easy to Spot… for Now
Wang, S.-Y., Wang, O., Zhang, R., Owens, A., & Efros, A. A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00872 · Cited by 1,210
Exposing Deep Fakes Using Inconsistent Head Poses
Yang, X., Li, Y., & Lyu, S. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683164 · Cited by 1,196
Thinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues
Qian, Y., Yin, G., Sheng, L., Chen, Z., & Shao, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58610-2_6 · Cited by 1,058
Synthesizing Obama: learning lip sync from audio
Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073640 · Cited by 1,057 · cited 1×
“Recent years have witnessed the rapid development of Deepfake techniques [26, 29, 43, 52], which enable attackers to manipulate the facial area of an image and generate a forged image.”
BackgroundMulti-attentional Deepfake Detection
Zhao, H., Wei, T., Zhou, W., Zhang, W., Chen, D., & Yu, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00222 · Cited by 1,019 · cited 7×
“Training Set Testing Set Xception [48] Multi-task [40] Capsule [41] DSW-FPA [33] Two-Branch [36] F3-Net [44] MaDD [66] DCViT [60] Ours”
BackgroundMethodInfluentialExposing DeepFake Videos By Detecting Face Warping Artifacts
Li, Y., & Lyu, S. (2018) · arXiv · doi:10.48550/arxiv.1811.00656 · Cited by 785
R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
Xu, H., Das, A., & Saenko, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.617 · Cited by 783 · cited 1×
“%) and compare with (1) P3D [47], which simplifies 3D convolutions with 2D filters on spatial dimension and 1D temporal connections; (2)R3D [67], which encodes the video sequences using a 3D fully convolutional networks and then generates candidate temporal fragments for classification; (3)I3D [3],…”
MethodFSGAN: Subject Agnostic Face Swapping and Reenactment
Nirkin, Y., Keller, Y., & Hassner, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00728 · Cited by 706 · cited 1×
“Synthesis Approaches To guarantee the diversity of synthesized images, we use four facial manipulation methods, including two face swapping methods: FSGAN [42] and FaceShifter [31], and two facial reenactment methods: First-order-motion [50] and IcFace [56].”
MethodTwo-Stream Neural Networks for Tampered Face Detection
Zhou, P., Han, X., Morariu, V. I., & Davis, L. S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.229 · Cited by 700 · cited 2×
“[67] uses a two-stream architecture to capture facial manipulation clues and patch inconsistency separately, while [40] simultaneously identifies forged faces and locate the manipulated regions with multi-task learning.”
BackgroundMethodSelf-citationGANimation: Anatomically-Aware Facial Animation from a Single Image
Pumarola, A., Agudo, A., Martinez, A. M., Sanfeliu, A., & Moreno-Noguer, F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_50 · Cited by 621
First Order Motion Model for Image Animation
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., & Sebe, N. (2020) · arXiv · doi:10.48550/arxiv.2003.00196 · Cited by 585 · cited 1×
“Synthesis Approaches To guarantee the diversity of synthesized images, we use four facial manipulation methods, including two face swapping methods: FSGAN [42] and FaceShifter [31], and two facial reenactment methods: First-order-motion [50] and IcFace [56].”
MethodEdgeConnect: Structure Guided Image Inpainting using Edge Prediction
Nazeri, K., Ng, E., Joseph, T., Qureshi, F., & Ebrahimi, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00408 · Cited by 562 · cited 1×
“Perceptual Loss Perceptual loss is usually used in face inpainting approaches [41, 69] to measure the similarity between the restored faces and corresponding complete faces.”
MethodMulti-task Learning For Detecting and Segmenting Manipulated Facial Images and Videos
Nguyen, H. H., Fang, F., Yamagishi, J., & Echizen, I. (2019) · arXiv · doi:10.48550/arxiv.1906.06876 · Cited by 551
Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos
Nguyen, H. H., Fang, F., Yamagishi, J., & Echizen, I. (2019) · 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS) · doi:10.1109/btas46853.2019.9185974 · Cited by 549 · cited 2×
“[72] uses a two-stream architecture to capture facial manipulation clues and patch inconsistency separately, while [42] simultaneously identifies forged faces and locates the manipulated regions with multi-task learning.”
BackgroundMethodEdgeConnect: Generative Image Inpainting with Adversarial Edge Learning
Nazeri, K., Ng, E., Joseph, T., Qureshi, F. Z., & Ebrahimi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00212 · Cited by 538 · cited 1×
“Perceptual Loss Perceptual loss is usually used in face inpainting approaches [39, 64] to measure the similarity between the restored”
MethodDeepFakes: a New Threat to Face Recognition? Assessment and Detection
Korshunov, P., & Marcel, S. (2018) · arXiv · doi:10.48550/arxiv.1812.08685 · Cited by 497 · cited 1×
“However, the quality of visual samples in current Deepfake datasets [15, 30, 37, 49, 68] is limited, containing clear artifacts (see Figure 1) like color mismatch, shape distortion, visible boundaries, and facial blurring.”
BackgroundTwo-Branch Recurrent Network for Isolating Deepfakes in Videos
Masi, I., Killekar, A., Mascarenhas, R. M., Gurudatt, S. P., & AbdAlmageed, W. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58571-6_39 · Cited by 471
Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints
Yu, N., Davis, L., & Fritz, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00765 · Cited by 437 · cited 2×
“It has been shown that artifacts in manipulated images and videos are no longer perceptible with compression approaches like JPEG compression [15, 24, 59, 65].”
BackgroundThe deepfake detection challenge (dfdc) dataset
Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., & Ferrer, C. C. (2020) · arXiv · doi:10.48550/arxiv.2006.07397 · Cited by 427
Cozzolino, D., Poggi, G., & Verdoliva, L. (2017) · Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/3082031.3083247 · Cited by 349
Local Relation Learning for Face Forgery Detection
Chen, S., Yao, T., Chen, Y., Ding, S., Li, J., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16193 · Cited by 334
Coherent Online Video Style Transfer
Chen, D., Liao, J., Yuan, L., Yu, N., & Hua, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.126 · Cited by 281 · cited 1×
“Ewarp The warping error 𝐸 𝑤𝑎𝑟𝑝 is used by [4, 25, 32] to measure the temporal inconsistency for video style transfer.”
MethodLearning Spatio-Temporal Representation with Pseudo-3D Residual NetworksPreprint
Qiu, Z., Yao, T., & Mei, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.10305 · Cited by 250 · cited 1×
“P3D [45] 80.”
Face2Face: real-time face capture and reenactment of RGB videos
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2018) · Communications of the ACM · doi:10.1145/3292039 · Cited by 244 · cited 1×
“Each video is manipulated by four Deep-fake methods, i.e. , Deepfakes [11], FaceSwap [20], Face2Face [57], and NeuralTextures [56].”
MethodFaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping
Li, L., Bao, J., Yang, H., Chen, D., & Wen, F. (2019) · arXiv · doi:10.48550/arxiv.1912.13457 · Cited by 236 · cited 3×
“FaceShifter [33] consists of two networks for full pipeline: AEI-Net for face swapping, and HEAR-Net for occlusion handling.”
BackgroundMethodInfluentialObjectFormer for Image Manipulation Detection and Localization
Wang, J., Wu, Z., Chen, J., Han, X., Shrivastava, A., Lim, S.-N., & Jiang, Y.-G. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00240 · Cited by 231 · cited 1×
“To address the above limitations, we explore transformers to model the relationships of pixels due to their strong capability of long-term dependency modeling for both natural language processing tasks [14, 48, 61] and computer vision tasks [2, 16, 62, 63, 73].”
MethodSelf-citationReal-Time Neural Style Transfer for Videos
Huang, H., Wang, H., Luo, W., Ma, L., Jiang, W., Zhu, X., Li, Z., & Liu, W. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.745 · Cited by 217 · cited 1×
“Ewarp The warping error Ewarp is used by [4, 22, 30] to measure the temporal inconsistency for video style transfer.”
MethodDoveNet: Deep Image Harmonization via Domain Verification
Cong, W., Zhang, J., Niu, L., Liu, L., Ling, Z., Li, W., & Zhang, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00842 · Cited by 204
Artistic Style Transfer for Videos
Ruder, M., Dosovitskiy, A., & Brox, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-45886-1_3 · Cited by 193
Unmasking DeepFakes with simple Features
Durall, R., Keuper, M., Pfreundt, F.-J., & Keuper, J. (2019) · arXiv · doi:10.48550/arxiv.1911.00686 · Cited by 184 · cited 2×
“Furthermore, [17, 26, 46, 64] suggest that the artifacts of forged images will be destroyed by perturbations such as JPEG compression, making them imper-ceptible in the RGB domain but can still be detected in the frequency domain.”
BackgroundJPEG-Phase-Aware Convolutional Neural Network for Steganalysis of JPEG Images
Chen, M., Sedighi, V., Boroumand, M., & Fridrich, J. (2017) · Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/3082031.3083248 · Cited by 176
Dale, K., Sunkavalli, K., Johnson, M. K., Vlasic, D., Matusik, W., & Pfister, H. (2011) · Proceedings of the 2011 SIGGRAPH Asia Conference · doi:10.1145/2024156.2024164 · Cited by 166 · cited 1×
“Face swapping methods [8, 10, 18, 38] replace the face of a source person in an image with the face of a target person.”
BackgroundDeepFake video detection using convolutional vision transformer
Wodajo, D., & Atnafu, S. (2021) · arXiv · doi:10.48550/arxiv.2102.11126 · Cited by 147 · cited 1×
“DCViT [65] extracts features from the face image using a CNN model, which are then fed to a traditional single-scale transformer for forgery detection.”
MethodR-C3D: Region Convolutional 3D Network for Temporal Activity DetectionPreprint
Xu, H. J., Das, A., & Saenko, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07814 · Cited by 128 · cited 1×
“%) and compare with (1) P3D [47], which simplifies 3D convolutions with 2D filters on spatial dimension and 1D temporal connections; (2)R3D [67], which encodes the video sequences using a 3D fully convolutional networks and then generates candidate temporal fragments for classification; (3)I3D [3],…”
MethodUse of a Capsule Network to Detect Fake Images and Videos
Nguyen, H. H., Yamagishi, J., & Echizen, I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12467 · Cited by 121 · cited 3×
“7 Capsule [41] 93.”
BackgroundMethodICface: Interpretable and Controllable Face Reenactment Using GANs
Tripathy, S., Kannala, J., & Rahtu, E. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093474 · Cited by 88
Blind Video Temporal Consistency via Deep Video PriorPreprint
Lei, C., Xing, Y., & Chen, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11838 · Cited by 76 · cited 2×
“We use it to compute the 𝐸 𝑤𝑎𝑟𝑝 of consecutive forged frames in different datasets to quantitatively measure the short-term consistency.”
MethodGenerative Landmark Guided Face Inpainting
Yang, Y., & Guo, X. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60633-6_2 · Cited by 67 · cited 1×
“Perceptual Loss Perceptual loss is usually used in face inpainting approaches [41, 69] to measure the similarity between the restored faces and corresponding complete faces.”
MethodFdftnet: facing off fake images using fake detection fine-tuning network
Jeon, H., Bang, Y., & Woo, S. S. (2020) · IFIP Advances in Information and Communication Technology · doi:10.1007/978-3-030-58201-2_28 · Cited by 65 · cited 1×
“The above challenges have driven the development of Deepfake forensics using deep neural networks [1, 5, 25, 32, 34, 40, 67].”
BackgroundHead2Head: Video-based Neural Head Synthesis
Koujan, M. R., Doukas, M. C., Roussos, A., & Zafeiriou, S. (2020) · 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) · doi:10.1109/fg47880.2020.00048 · Cited by 61 · cited 1×
“Recent years have witnessed the rapid development of Deepfake techniques [28, 31, 45, 54], which enable attackers to manipulate”
BackgroundLearning Identity-Invariant Motion Representations for Cross-ID Face Reenactment
Huang, P.-H., Yang, F.-E., & Wang, Y.-C. F. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00711 · Cited by 48 · cited 1×
“Specially, [23] use a landmark decoder to model expression motions and a content encoder to extract identity information, thus handling the task by feature disentanglement.”
MethodThinking in Frequency: Face Forgery Detection by Mining Frequency-aware CluesPreprint
Qian, Y.-Y., Yin, G., Sheng, L., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.09355 · Cited by 42
RSGAN: Face Swapping and Editing using Face and Hair Representation in Latent Spaces
Natsume, R., Yatagawa, T., & Morishima, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.03447 · Cited by 39 · cited 1×
“Face swapping methods [8, 10, 18, 38] replace the face of a source person in an image with the face of a target person.”
BackgroundCoherent Online Video Style TransferPreprint
Chen, D., Liao, J., Yuan, L., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.09211 · Cited by 39 · cited 1×
“Ewarp The warping error 𝐸 𝑤𝑎𝑟𝑝 is used by [4, 25, 32] to measure the temporal inconsistency for video style transfer.”
MethodLaFIn: Generative Landmark Guided Face InpaintingPreprint
Yang, Y., Guo, X., Ma, J., Ma, L., & Ling, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.11394 · Cited by 32 · cited 1×
“Perceptual Loss Perceptual loss is usually used in face inpainting approaches [37, 62] to measure the similarity between the restored”
MethodDeep Frequent Spatial Temporal Learning for Face Anti-SpoofingPreprint
Huang, Y., Zhang, W., & Wang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.03723 · Cited by 20 · cited 2×
“Furthermore, [17, 26, 46, 64] suggest that the artifacts of forged images will be destroyed by perturbations such as JPEG compression, making them imper-ceptible in the RGB domain but can still be detected in the frequency domain.”
BackgroundSelf-citationReenactGAN: Learning to Reenact Faces via Boundary TransferPreprint
Wu, W., Zhang, Y., Li, C., Qian, C., & Loy, C. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11079 · Cited by 19 · cited 1×
“Facial reenactment methods [54, 55, 61], on the other hand, transfer the face expressions of the source person to the target person while preserving the identity information of the source person.”
BackgroundMulti-attentional Deepfake DetectionPreprint
Zhao, H., Zhou, W., Chen, D., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.02406 · Cited by 15 · cited 7×
“Training Set Testing Set Xception [48] Multi-task [40] Capsule [41] DSW-FPA [33] Two-Branch [36] F3-Net [44] MaDD [66] DCViT [60] Ours”
BackgroundMethodInfluentialEfficient Video Transformers with Spatial-Temporal Token SelectionPreprint
Wang, J., Yang, X., Li, H., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.11591 · Cited by 2 · cited 1×
“To address the above limitations, we explore transformers to model the relationships of pixels due to their strong capability of long-term dependency modeling for both natural language processing tasks [14, 48, 61] and computer vision tasks [2, 16, 62, 63, 73].”
MethodSelf-citation
Research works printed but not linked: 11
- [7] François Chollet. 2017. Xception: Deep learning with depthwise separable convolutions. In CVPR . -
- [13] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020. Retinaface: Single-shot multi-level face localisation in the wild. In CVPR . -
- [22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR . -
- [23] Yinan He, Bei Gan, Siyu Chen, Yichun Zhou, Guojun Yin, Luchuan Song, Lu Sheng, Jing Shao, and Ziwei Liu. 2021. ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysis. In CVPR . -
- [24] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. 2019. Searching for mobilenetv3. In ICCV . -
- [28] Ira Kemelmacher-Shlizerman. 2016. Transfiguring portraits. ACM TOG (2016). -
- [38] Zhengzhe Liu, Xiaojuan Qi, and Philip HS Torr. 2020. Global texture enhancement for fake face detection in the wild. In CVPR . -
- [52] Karen Simonyan and Andrew Zisserman. [n.d.]. Very deep convolutional networks for large-scale image recognition. In ICLR . -
- [58] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. 2021. Training data-efficient image transformers & distillation through attention. In ICML . -
- [66] Tete Xiao, Piotr Dollar, Mannat Singh, Eric Mintun, Trevor Darrell, and Ross Girshick. 2021. Early Convolutions Help Transformers See Better. In NIPS . -
- [74] Bojia Zi, Minghao Chang, Jingjing Chen, Xingjun Ma, and Yu-Gang Jiang. 2020. WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection. In ACM MM . -
Cited by
DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation
Lin, A., Chen, B., Xu, J., Zhang, Z., Lu, G., & Zhang, D. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3178991 · Cited by 882
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., & Malik, H. (2023) · Applied Intelligence · doi:10.1007/s10489-022-03766-z · Cited by 543
Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting
Chen, P., Zhang, Y., Cheng, Y., Shu, Y., Wang, Y., Wen, Q., Yang, B., & Guo, C. (2024) · VBN Forskningsportal (Aalborg Universitet) · doi:10.48550/arxiv.2402.05956 · Cited by 504
CIFAKE: Image Classification and Explainable Identification of AI-Generated Synthetic Images
Bird, J. J., & Lotfi, A. (2024) · IEEE Access · doi:10.1109/access.2024.3356122 · Cited by 265
“In Wang et al. [20], EfficientNets and Vision Transformers are proposed within a system that can detect forged images by adversarial models at an F1 score of 0.88 and AUC of 0.95, competing with the state of the art on the DeepFake Detection Challenge dataset while remaining efficient.”
MethodInfluentialUCF: Uncovering Common Features for Generalizable Deepfake Detection
Yan, Z., Zhang, Y., Fan, Y., & Wu, B. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02048 · Cited by 241
“Other notable directions are focusing on some specific representations, i.e., forgery region location [33], neuron behaviors [49], optical flow [4], landmark geometric features [42], 3D decomposition [59], erasing technology [47], and attentional networks [10, 55, 48].”
Background
Metadata
- Authors
- Wang, J., Wu, Z., Ouyang, W., Han, X., Chen, J., Jiang, Y.-G., & Li, S.-N.
- Affiliations
- Fudan University (CN)
- Huya Inc, shenzhen, China
- Meta AI, Newyork, NY, USA
- Venue
- Proceedings of the 2022 International Conference on Multimedia Retrieval
- Publisher
- ACM
- Published
- 2022-06-27
- Pages
- 615-623
- OpenAlex
- W3154326567
- Cited by
- 368
- References
- 64
- Citations by year
- 2026: 66 · 2025: 110 · 2024: 90 · 2023: 67 · 2022: 31
- FWCI
- 27.02
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference on Multimedia Retrieval (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation (62072116)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Keywords
- deepfake detection, multi-scale transformer, multimodal fusion, frequency-domain features, cross-modal fusion, face swapping, video forgery detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.