CoChat
Sign Up
PreprintCited by 147Digital Media Forensic DetectionGenerative Adversarial Networks and Image SynthesisFace recognition and analysis

DeepFake video detection using convolutional vision transformer

Open access
Wodajo, Deressa, Atnafu, Solomon·2021·arXiv
DOI 10.48550/arxiv.2102.11126arXiv 2102.11126
Open this paper in CoChat — free accountOpen PDF

Abstract

The rapid advancement of deep learning models that can generate and synthesis hyper-realistic videos known as Deepfakes and their ease of access to the general public have raised concern from all concerned bodies to their possible malicious intent use. Deep learning techniques can now generate faces, swap faces between two subjects in a video, alter facial expressions, change gender, and alter facial features, to list a few. These powerful video manipulation methods have potential use in many fields. However, they also pose a looming threat to everyone if used for harmful purposes such as identity theft, phishing, and scam. In this work, we propose a Convolutional Vision Transformer for the detection of Deepfakes. The Convolutional Vision Transformer has two components: Convolutional Neural Network (CNN) and Vision Transformer (ViT). The CNN extracts learnable features while the ViT takes in the learned features as input and categorizes them using an attention mechanism. We trained our model on the DeepFake Detection Challenge Dataset (DFDC) and have achieved 91.5 percent accuracy, an AUC value of 0.91, and a loss value of 0.32. Our contribution is that we have added a CNN module to the ViT architecture and have achieved a competitive result on the DFDC dataset.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2102.11126

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Creating patches by using characteristics gathered in an image increases the quality of the training process.

    “Additionally, several researches in [57] and [58] have demonstrated that creating patches by using characteristics gathered in an image also increases the quality of the training process.”

    From Generalization of Forgery Detection With Meta Deepfake Detection Model (2023) · cites this paper as [58]

    The Detection section (pp. 4–5) says the model extracts features from face images and splits the resulting feature maps into patches. It does not show that creating those patches improves the quality of training; the Evaluation section (pp. 5–6) reports model performance without isolating the effect of patch creation. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Wodajo et al. (2020) achieved a 0.91 AUROC on the DFDC dataset.

    “This model achieved 91.5% accuracy and a 0.91 AUROC on the DFDC dataset by analyzing facial regions in detail.”

    From Enhancing Deepfake Detection Through Quantum Transfer Learning and Class-Attention Vision Transformer Architecture (2025) · cites this paper as [12]

  • Wodajo et al. (2020) achieved 91.5% accuracy on the DFDC dataset.

    “This model achieved 91.5% accuracy and a 0.91 AUROC on the DFDC dataset by analyzing facial regions in detail.”

    From Enhancing Deepfake Detection Through Quantum Transfer Learning and Class-Attention Vision Transformer Architecture (2025) · cites this paper as [12]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. [1][1] Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. MesoNet: a Compact Facial Video Forgery Detection Network. pages 1-7, 2018. -
  2. [2][2] Shruti Agarwal, Hany Farid, Yuming Gu, Mingming He, Koki Nagano, and Hao Li. Protecting World Leaders Against Deep Fakes. In CVPR Workshops , 2019. -
  3. [3][3] Charu C. Aggarwal. Neural Networks and Deep Learning: A Textbook . Springer International Publishing, Switzerland, 2020. -
  4. [4][4] Md Zahangir Alom, Tarek M. Taha, Chris Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Mahmudul Hasan, Brian C. Van Essen, Abdul A. S. Awwal, and Vijayan K. Asari. A State-of-the-Art Survey on Deep Learning Theory and Architectures. Electronics , 8(3):292, 2019. -
  5. [6][6] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V. Le. Attention augmented convolutional networks. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 3285-3294, 2019. -
  6. [7][7] Avishek Joey Bose and Parham Aarabi. Virtual Fakes: DeepFakes for Virtual Reality. In 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP) , pages 1-1. IEEE, 2019. -
  7. [8][8] Andrew P. Bradley. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition , 30(7):1145-1159, 1997. -
  8. [10][10] Joshua Brockschmidt, Jiacheng Shang, and Jie Wu. On the Generality of Facial Forgery Detection. In 2019 IEEE 16th International Conference on Mobile Ad Hoc and Sensor Systems Workshops (MASSW) , pages 43-47. IEEE, 2019. -
  9. [11][11] Polychronis Charitidis, Giorgos Kordopatis-Zilos, Symeon Papadopoulos, and Ioannis Kompatsiaris. Investigating the Impact of Pre-processing and Prediction Aggregation on the DeepFake Detection Task. arXiv preprint arXiv:2006.07084v1 , 2020. - · arXiv
  10. [12][12] Bobby Chesney and Danielle Citron. Deep Fakes A Looming Challenge for Privacy, Democracy, and National Security, 2019. Available at https://ssrn.com/abstract=3213954. -
  11. [13][13] Umur Aybars Ciftci, Ilke Demir, and Lijun Yin. FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals. arXiv preprint arXiv:1901.02212v2 , 2019. - · arXiv
  12. [14][14] Sourabh Dhere, Suresh B. Rathod, Sanket Aarankalle, Yash Lad, and Megh Gandhi. A Review on Face Reenactment Techniques. In 2020 International Conference on Industry 4.0 Technology (I4Tech) , pages 191-194, Pune, India, 2020. IEEE. -
  13. [15][15] Chris Donahue, Julian J. McAuley, and Miller S. Puckette. Adversarial Audio Synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , New York, NY, USA, 2019. OpenReview.net. -
  14. [18][18] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozairy, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 , page 2672-2680, Cambridge, MA, USA, 2014. MIT Press. -
  15. [19][19] Arushi Handa, Prerna Garg, and Vijay Khare. Masked Neural Style Transfer using Convolutional Neural Networks. In 2018 International Conference on Recent Innovations in Electrical, Electronics Communication Engineering (ICRIEECE) , pages 2099-2104, 2018. -
  16. [20][20] Rahul Haridas and Jyothi R L. Convolutional Neural Networks: A Comprehensive Survey. International Journal of Applied Engineering Research (IJAER) , 14(03):780-789, 2019. -
  17. [23][23] He Huang, Phillip S. Yu, and Changhu Wang. An Introduction to Image Synthesis with Generative Adversarial Nets. arXiv preprints arXiv:1803.04469v1 , 2018. - · arXiv
  18. [24][24] Xun Huang, Ming-Yu Liu, Serge Belongie, and Ming-Yu Liu. Multimodal Unsupervised Image-to-Image Translation. In Computer Vision - ECCV 2018 , pages 179-196, Cham, 2018. Springer International Publishing. -
  19. [25][25] TackHyun Jung, SangWon Kim, and KeeCheon Kim. DeepVision: Deepfakes Detection Using Human Eye Blinking Pattern. IEEE Access , 8:83144-83154, 2020. -
  20. [26][26] Tero Karras, Samuli Laine, and Timo Aila. A StyleBased Generator Architecture for Generative Adversarial Networks. arXiv preprints arXiv:1812.04948 , 2018. - · arXiv
  21. [28][28] Ali Khodabakhsh, Raghavendra Ramachandra, Kiran Raja, and Pankaj Wasnik. Fake face detection methods: Can they be generalized? In 2018 International Conference of the Biometrics Special Interest Group (BIOSIG) , pages 1-6. IEEE, 2018. -
  22. [29][29] Junyaup Kim, Siho Han, and Simon S. Woo. Classifying Genuine Face images from Disguised Face Images. In 2019 IEEE International Conference on Big Data (Big Data) , pages 6248-6250, 2019. -
  23. [33][33] Yuezun Li, Ming-Ching Chang, and Siwei Lyu. In Ictu Oculi: Exposing AI Generated Fake Face Videos by Detecting Eye Blinking. arXiv preprint arXiv:1806.02877v2 , 2018. - · arXiv
  24. [34][34] Yuezun Li and Siwei Lyu. Exposing DeepFake Videos By Detecting Face Warping Artifacts. arXiv preprint arXiv:1811.00656v3 , 2019. - · arXiv
  25. [35][35] Arun Mallya, Ting-Chun Wang, Karan Sapra, and Ming-Yu Liu. World-Consistent Video-to-Video Synthesis. In Computer Vision - ECCV 2020 , pages 359-378, Cham, 2020. Springer International Publishing. -
  26. [36][36] Brais Martinez, Michel F. Valstar, Bihan Jiang, and Maja Pantic. Automatic Analysis of Facial Actions: A Survey. IEEE Transactions on Affective Computing , 10(3):325-347, 2019. -
  27. [39][39] Ryota Natsume, Tatsuya Yatagawa, and Shigeo Morishima. RSGAN: Face Swapping and Editing Using Face and Hair Representation in Latent Spaces. In ACM SIGGRAPH 2018 Posters , SIGGRAPH '18, New York, NY, USA, 2018. Association for Computing Machinery. -
  28. [40][40] Huy H. Nguyen, Ngoc-Dung T. Tieu, Hoang-Quoc NguyenSon, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. Modular Convolutional Neural Network for Discriminating between Computer-Generated Images and Photographic Images. In Proceedings of the 13th International Conference on Availability, Reliability and Security , New York, NY, USA, 2018. Association for Computing Machinery. -
  29. [41][41] Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2307-2311, 2019. -
  30. [42][42] Thanh Thi Nguyen, Cuong M. Nguyen, Dung Tien Nguyen, Duc Thanh Nguyen, and Saeid Nahavandi. Deep Learning for Deepfakes Creation and Detection. arXiv preprint arXiv:1909.11573v1 , 2019. - · arXiv
  31. [43][43] Yuval Nirkin, Yosi Keller, and Tal Hassner. FSGAN: Subject Agnostic Face Swapping and Reenactment. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019 , pages 7183-7192. IEEE, 2019. -
  32. [45][45] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic Image Synthesis With Spatially-Adaptive Normalization. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2332-2341. IEEE, 2019. -
  33. [46][46] Hai X. Pham, Yuting Wang, and Vladimir Pavlovic. Generative Adversarial Talking Head: Bringing Portraits to Life with a Weakly Supervised Neural Network. arXiv preprint arXiv:1803.07716 , 2018. - · arXiv
  34. [49][49] Prajwal K R, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, and C V Jawahar. Towards Automatic Face-to-Face Translation. In the 27th ACM International Conference on Multimedia (MM '19) , page 1428-1436, New York, NY, USA, 2019. Association for Computing Machinery. -
  35. [51][51] Kuniaki Saito, Kate Saenko, and Ming-Yu Liu. COCOFUNIT: Few-Shot Unsupervised Image Translation with a Content Conditioned Style Encoder. In Computer Vision ECCV 2020 , pages 382-398, Cham, 2020. Springer International Publishing. -
  36. [52][52] Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. -
  37. [53][53] Supasorn Suwajanakorn, Steven M. Seitz, and Ira Kemelmacher-Shlizerman. Synthesizing Obama: Learning Lip Sync from Audio. ACM Trans. Graph. , 36(4):780-789, 2017. -
  38. [54][54] Justus Thies, Michael Zollh¨ ofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. Face2Face: Real-Time Face Capture and Reenactment of RGB Videos. Commun. ACM , 62(1):96-104, 2018. -
  39. [57][57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is All You Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 6000-6010. Curran Associates Inc., 2017. -
  40. [58][58] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Realistic Speech-Driven Facial Animation with GANs. International Journal of Computer Vision , 128:1398-1413, 2020. -
  41. [59][59] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Videoto-Video Synthesis. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, page 1152-1164, Red Hook, NY, USA, 2018. Curran Associates Inc. -
  42. [60][60] M. Arif Wani, Farooq Ahmad Bhat, Saduf Afzal, and Asif Iqbal Khan. Advances in Deep Learning , volume 57 of Studies in Big Data . Springer Nature, Singapore, 2020. -
  43. [61][61] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-Shot Adversarial Learning of Realistic Neural Talking Head Models. arXiv preprint arXiv:1905.08233v2 , 2019. - · arXiv
  44. [62][62] Lilei Zheng, Ying Zhang, and Vrizlynn L.L. Thing. A survey on image tampering and its detection in real-world photos. Elsevier , 58:380-399, 2018. -
  45. [63][63] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired Image-to-Image Translation Using CycleConsistent Adversarial Networks. In 2017 IEEE International Conference on Computer Vision (ICCV) , pages 22422251, 2017.

Cited by

Sorted by citations

Authors
Wodajo, D., & Atnafu, S.
Affiliations
Jimma University (ET)
Corresponding author
Wodajo, Deressa
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2102.11126
arXiv
2102.11126
OpenAlex
W3128609017
Cited by
147
References
11
Citations by year
2025: 21 · 2024: 42 · 2023: 49 · 2022: 28 · 2021: 7
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
Keywords
deepfake detection, video manipulation, convolutional vision transformer, face swapping, attention mechanism, DFDC, identity theft

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review