Video Transformer for Deepfake Detection with Incremental Learning
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
Face forgery by deepfake is widely spread over the internet and this raises severe societal concerns. In this paper, we propose a novel video transformer with incremental learning for detecting deepfake videos. To better align the input face images, we use a 3D face reconstruction method to generate UV texture from a single input face image. The aligned face image can also provide pose, eyes blink and mouth movement information that cannot be perceived in the UV texture image, so we use both face images and their UV texture maps to extract the image features. We present an incremental learning strategy to fine-tune the proposed model on a smaller amount of data and achieve better deepfake detection performance. The comprehensive experiments on various public deepfake datasets demonstrate that the proposed video transformer model with incremental learning achieves state-of-the-art performance in the deepfake video detection task with enhanced feature learning from the sequenced data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · cited 3×
“They employ a simple Xception [12] network pre-trained on imagenet dataset, and � ne-tune it on FaceForensics++ dataset.”
MethodEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595 · cited 1×
“The transformer models can naturally accommodate the video sequences for the feature learning [28].”
BackgroundSelf-citationTransformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
Self-citationFaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 22×
“They evaluate their networks on a public dataset [46] and a dataset they generate from videos available online.”
BackgroundMethodInfluentialLongformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · cited 1×
“A large number of transformer based models are used to deal with the vision tasks, such as image classification[22], object detection [8], image segmentation [49], image captioning [34], video classification [24, 48], and visual question answering [11, 34, 42].”
MethodMesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00327 · Cited by 1,846 · cited 2×
“After we extract image features through XceptionNet, we get a feature vector of dimension [2048, 10, 10], we then pass this feature vector to a 2D convolutional layer and a linear layer, which gives us a feature vector of dimension [1, 32, 768].”
MethodFace X-Ray for More General Face Forgery Detection
Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., & Guo, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00505 · Cited by 1,211
Exposing Deep Fakes Using Inconsistent Head Poses
Yang, X., Li, Y., & Lyu, S. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683164 · Cited by 1,196
End-to-End Incremental Learning
Castro, F. M., Marín-Jiménez, M. J., Guil, N., Schmid, C., & Alahari, K. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01258-8_15 · Cited by 1,184
VideoBERT: A Joint Model for Video and Language Representation Learning
Sun, C., Myers, A., Vondrick, C., Murphy, K., & Schmid, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,172
Media Forensics and DeepFakes: An Overview
Verdoliva, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3002101 · Cited by 979
Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos
Nguyen, H. H., Yamagishi, J., & Echizen, I. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682602 · Cited by 951
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · cited 1×
“A number of more e � cient transformer architectures are used to address this issue [4, 15, 35, 49, 52].”
MethodThe Creation and Detection of Deepfakes: A Survey
Mirsky, Y., & Lee, W. (2022) · ACM Computing Surveys · doi:10.1145/3425780 · Cited by 863 · cited 1×
“Deep-fake can be grouped into four categories: face replacement, facial re-enactment, face editing, and complete face synthesis [37].”
BackgroundThe Creation and Detection of Deepfakes: A Survey
Mirsky, Y., & Lee, W. (2020) · arXiv · doi:10.48550/arxiv.2004.11138 · Cited by 863
A Deep Learning Approach to Universal Image Manipulation Detection Using a New Convolutional Layer
Bayar, B., & Stamm, M. C. (2016) · Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/2909827.2930786 · Cited by 838
Exposing DeepFake Videos By Detecting Face Warping Artifacts
Li, Y., & Lyu, S. (2018) · arXiv · doi:10.48550/arxiv.1811.00656 · Cited by 785 · cited 1×
“Li et al. [31] employ CNNs to detect face warping artifacts in images from the deepfake datasets [29, 55].”
BackgroundMulti-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos
Nguyen, H. H., Fang, F., Yamagishi, J., & Echizen, I. (2019) · 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS) · doi:10.1109/btas46853.2019.9185974 · Cited by 549
Scaling Neural Machine Translation
Ott, M., Edunov, S., Grangier, D., & Auli, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6301 · Cited by 544
FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals
Ciftci, U. A., Demir, I., & Yin, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3009287 · Cited by 499
DeepFakes: a New Threat to Face Recognition? Assessment and Detection
Korshunov, P., & Marcel, S. (2018) · arXiv · doi:10.48550/arxiv.1812.08685 · Cited by 497 · cited 1×
“[31] employ CNNs to detect face warping artifacts in images from the deepfake datasets [29, 55].”
MethodTowards Fast, Accurate and Stable 3D Dense Face Alignment
Guo, J., Zhu, X., Yang, Y., Yang, F., Lei, Z., & Li, S. Z. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58529-7_10 · Cited by 462
Recurrent convolutional strategies for face manipulation detection in videos
Sabir, E., Cheng, J., Jaiswal, A., AbdAlmageed, W., Masi, I., & Natarajan, P. (2019) · arXiv · doi:10.48550/arxiv.1905.00582 · Cited by 443 · cited 4×
“The recurrent networks can learn to detect the intra-frame discrepancies, such as flickering, blurry frames, and mouth movement [27, 47].”
BackgroundMethodInfluentialCozzolino, D., Poggi, G., & Verdoliva, L. (2017) · Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/3082031.3083247 · Cited by 349
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413570 · Cited by 319 · cited 1×
“propose a multi-modal deepfake detection method in [38].”
Methodrwightman/pytorch-image-models: 3rd Party Vision Transformer Weights
Wightman, R., Raw, N., Soare, A., Arora, A., Ha, C., Reich, C., Guan, F., Kaczmarzyk, J., MrT23, M., et al. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4414861 · Cited by 250
Detecting Deep-Fake Videos from Appearance and Behavior
Agarwal, S., Farid, H., El-Gaaly, T., & Lim, S.-N. (2020) · 2020 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs49906.2020.9360904 · Cited by 242
On the Generalization of GAN Image Forensics
Xuan, X., Peng, B., Wang, W., & Dong, J. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-31456-9_15 · Cited by 184
Statistical Modeling of Craniofacial Shape and Texture
Dai, H., Pears, N. E., Smith, W. A. P., & Duncan, C. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-019-01260-7 · Cited by 155
Self-citationVideoBERT: A Joint Model for Video and Language Representation LearningPreprint
Sun, C., Myers, A., Vondrick, C., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.01766 · Cited by 148
A 3D Morphable Model of Craniofacial Shape and Texture Variation
Dai, H., Pears, N. E., Smith, W. A. P., & Duncan, C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.335 · Cited by 130
Self-citationTowards Fast, Accurate and Stable 3D Dense Face AlignmentPreprint
Guo, J., Zhu, X., Yang, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09960 · Cited by 31
Video Face Manipulation Detection Through Ensemble of CNNs
Bonettini, N., Cannas, E. D., Mandelli, S., Bondi, L., Bestagini, P., & Tubaro, S. (2021) · 2020 25th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr48806.2021.9412711 · Cited by 27
C4AV: Learning Cross-Modal Representations from Transformers
Luo, S., Dai, H., Shao, L., & Ding, Y. S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66096-3_3 · Cited by 16
Self-citationCommands for Autonomous Vehicles by Progressively Stacking Visual-Linguistic Representations
Dai, H., Luo, S., Ding, Y. S., & Shao, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66096-3_2 · Cited by 14
Self-citation3D Morphable Models: The Face, Ear and Head
Dai, H., Pears, N. E., Huber, P., & Smith, W. A. P. (2020) · Springer eBooks · doi:10.1007/978-3-030-44070-1_10 · Cited by 13
Self-citation(2000) · Medical Entomology and Zoology
Research works printed but not linked: 18
- Bonettini Nicolò, 2020, Sara Mandelli, Luca Bondi, Paolo Bestagini, and Stefano Tubaro.
- Brown Tom B., 2020, Lin (Eds.), 33
- Charitidis Polychronis, 2020, Investigating the Impact of Pre-processing and Prediction Aggregation on the DeepFake Detection Task. arXiv: Computer Vision and Pattern Recognition
- de Lima Oscar, 2020, Deepfake Detection using Spatiotemporal Convolutional Networks. arXiv: Computer Vision and Pattern Recognition
- Dolhansky Brian, 2020, The DeepFake Detection Challenge (DFDC) Dataset. arXiv: Computer Vision and Pattern Recognition
- Dosovitskiy Alexey, 2010, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arxiv
- Fedus William, 2021, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. ArXiv
- Girdhar Rohit, 2019, Video Action Transformer Network. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Guo Jianzhu, Proceedings of the European Conference on Computer Vision (ECCV).
- Güera David, 2018, Delp
- Khan Salman, 2021, Fahad Shahbaz Khan, and Mubarak Shah.
- Li Yuezun, 2019, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. IEEE. https://arxiv.org/abs/1811
- 10.5555/3454287.3454289 · doi:10.5555/3454287.3454289
- Qi Di, 2020, ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data. ArXiv
- Sabir Ekraam, 2019, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. IEEE, 80--87
- Touvron Hugo, 2020, Training data-efficient image transformers & distillation through attention. ArXiv
- 10.5555/3295222.3295349 · doi:10.5555/3295222.3295349
- Zhu Xiangyu, 2020, Li
Cited by
ISTVT: Interpretable Spatial-Temporal Video Transformer for Deepfake Detection
Zhao, C., Wang, C., Hu, G., Chen, H., Liu, C., & Tang, J. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3239223 · Cited by 184
Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection
Haliassos, A., Mira, R., Petridis, S., & Pantic, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01453 · Cited by 180
Zhuang, W., Chu, Q., Tan, Z., Liu, Q., Yuan, H., Miao, C., Luo, Z., & Yu, N. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20065-6_23 · Cited by 224
Deepfakes: current and future trends
Gambín, Á. F., Yazidi, A., Vasilakos, A., Haugerud, H., & Djenouri, Y. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-023-10679-x · Cited by 145
Audio-Visual Person-of-Interest DeepFake Detection
Cozzolino, D., Pianese, A., Nießner, M., & Verdoliva, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00101 · Cited by 104
Metadata
- Authors
- Khan, S. A., & Dai, H.
- Affiliations
- Mohamed bin Zayed University of Artificial Intelligence (AE)
- Venue
- Proceedings of the 29th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2021-10-17
- Pages
- 1821-1828
- OpenAlex
- W3206817819
- Cited by
- 79
- References
- 40
- Citations by year
- 2026: 13 · 2025: 21 · 2024: 22 · 2023: 16 · 2022: 7
- FWCI
- 3.68
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- deepfake detection, video transformer, incremental learning, face forgery, 3D face reconstruction, fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.