Combining EfficientNet and Vision Transformers for Video Deepfake Detection
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Deepfakes are the result of digital manipulation to forge realistic yet fake imagery. With the astonishing advances in deep generative models, fake images or videos are nowadays obtained using variational autoencoders (VAEs) or Generative Adversarial Networks (GANs). These technologies are becoming more accessible and accurate, resulting in fake videos that are very difficult to be detected. Traditionally, Convolutional Neural Networks (CNNs) have been used to perform video deepfake detection, with the best results obtained using methods based on EfficientNet B7. In this study, we focus on video deep fake detection on faces, given that most methods are becoming extremely accurate in the generation of realistic human faces. Specifically, we combine various types of Vision Transformers with a convolutional EfficientNet B0 used as a feature extractor, obtaining comparable results with some very recent methods that use Vision Transformers. Differently from the state-of-the-art approaches, we use neither distillation nor ensemble methods. Furthermore, we present a straightforward inference procedure based on a simple voting scheme for handling multiple faces in the same video shot. The best model achieved an AUC of 0.951 and an F1 score of 88.0%, very close to the state-of-the-art on the DeepFake Detection Challenge (DFDC).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, several researches in [57] and [58] have demonstrated that creating patches by using characteristics gathered in an image also increases the quality of the training process.”
From Generalization of Forgery Detection With Meta Deepfake Detection Model (2023) · cites this paper as [57]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Generative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
YOLOv3: An Incremental Improvement
Redmon, J., & Farhadi, A. (2018) · arXiv · doi:10.48550/arxiv.1804.02767 · Cited by 17,141
A Multi-Modal Distributed Real-Time IoT System for Urban Traffic Control (Invited Paper)
Khanam, Z., Achari, V. P. S., Boukhennoufa, I., Jindal, A., & Singh, A. K. (2024) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/oasics.ng-res.2024.2 · Cited by 14,040
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00813 · Cited by 6,332
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2019) · arXiv · doi:10.48550/arxiv.1912.04958 · Cited by 6,332
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
Generative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., & Choo, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00916 · Cited by 4,163
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595 · cited 1×
“…same time, more and more models based on Transformers are gaining ground in the field of Computer Vision, showing excellent results in image processing [21,17], document retrieval [27], and efficient visual-textual matching [30,32], mainly for use in large-scale multi-modal retrieval systems [1,31].”
BackgroundTransformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 1×
“These datasets are grouped into three generations, the first generation consisting of DF-TIMIT [24], UADFC [41] and FaceForensics++ [36], the second generation datasets such as Google Deepfake Detection Dataset [11], Celeb-DF [25], and finally the third generation datasets, with the DFDC dataset [9]…”
MethodAlbumentations: fast and flexible image augmentations
Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., & Kalinin, A. A. (2020) · Information · doi:10.3390/info11020125 · Cited by 2,714 · cited 1×
“We used the Albumentations library [4], and we applied common transformations such as the introduction of blur, Gaussian noise, transposition, rotation, and various isotropic resizes during training.”
MethodCrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
Chen, C.-F. R., Fan, Q., & Panda, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00041 · Cited by 2,074
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00327 · Cited by 1,846 · cited 1×
“…into three generations, the first generation consisting of DF-TIMIT [24], UADFC [41] and FaceForensics++ [36], the second generation datasets such as Google Deepfake Detection Dataset [11], Celeb-DF [25], and finally the third generation datasets, with the DFDC dataset [9] and DeepForensics [19].”
MethodLearning to Discover Cross-Domain Relations with Generative Adversarial Networks
Kim, T., Cha, M., Kim, H., Lee, J. K., & Kim, J. (2017) · arXiv · doi:10.48550/arxiv.1703.05192 · Cited by 1,716 · cited 1×
“With GANs, very credible and realistic results have been obtained, and over time, numerous approaches have been introduced such as StarGAN [7] and DiscoGAN [22]; the best results in this field have been obtained with StyleGAN-V2 [20].”
MethodDeepfakes and beyond: A Survey of face manipulation and fake detection
Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., & Ortega-Garcia, J. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.014 · Cited by 1,237
Exposing Deep Fakes Using Inconsistent Head Poses
Yang, X., Li, Y., & Lyu, S. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683164 · Cited by 1,196
Jointly Embedding Multiple Single-Cell Omics Measurements
Liu, J., Huang, Y., Singh, R., Vert, J., & Noble, W. S. (2019) · Leibniz international proceedings in informatics · doi:10.4230/lipics.wabi.2019.10 · Cited by 1,141
Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security
Chesney, R., & Citron, D. K. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3213954 · Cited by 880 · cited 1×
“These methods have generated numerous fake news or revenge porn videos, becoming a severe problem in modern society [6].”
BackgroundDeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection
Jiang, L., Li, R., Wu, W., Qian, C., & Loy, C. C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00296 · Cited by 599
Sharpness-Aware Minimization for Efficiently Improving Generalization
Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. (2020) · arXiv · doi:10.48550/arxiv.2010.01412 · Cited by 521 · cited 1×
“CNNs, in fact, have a long-established success on many tasks, ranging from image classification [13,40] and object detection [35,2,8] to abstract visual reasoning [28,29].”
BackgroundDeepFakes: a New Threat to Face Recognition? Assessment and Detection
Korshunov, P., & Marcel, S. (2018) · arXiv · doi:10.48550/arxiv.1812.08685 · Cited by 497 · cited 1×
“These datasets are grouped into three generations, the first generation consisting of DF-TIMIT [24], UADFC [41] and FaceForensics++ [36], the second generation datasets such as Google Deepfake Detection Dataset [11], Celeb-DF [25], and finally the third generation datasets, with the DFDC dataset [9]…”
MethodThe deepfake detection challenge (dfdc) dataset
Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., & Ferrer, C. C. (2020) · arXiv · doi:10.48550/arxiv.2006.07397 · Cited by 427
Deepfake Video Detection through Optical Flow Based CNN
Amerini, I., Galteri, L., Caldelli, R., & Del Bimbo, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00152 · Cited by 424 · cited 1×
“Some efforts to capture spatio-temporal inconsistencies were made in [26] using 3DCNN networks and in [3], which presented a method that exploits optical flow to detect video glitches.”
MethodDeepFake Detection by Analyzing Convolutional Traces
Guarnera, L., Giudice, O., & Battiato, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00341 · Cited by 374 · cited 1×
“In particular, the authors in [16] proposed a method based on K-nearest neighbors, while the work in [41] exploited SVMs.”
MethodA Survey on Visual TransformerPreprint
Han, K., Wang, Y., Chen, H., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.12556 · Cited by 214
TweepFake: about detecting deepfake tweets
Fagni, T., Falchi, F., Gambini, M., Martella, A., & Tesconi, M. (2021) · PLOS ONE · doi:10.1371/journal.pone.0251415 · Cited by 185 · cited 1×
“For example, the recent work in [12] analyzes deepfakes in tweets for finding and defeating false content in social networks.”
BackgroundSelf-citationDeepfakes Detection with Automatic Face Weighting
Montserrat, D. M., Hao, H., Yarlagadda, S. K., Baireddy, S., Shao, R., Horvath, J., Bartusiak, E., Yang, J., Guera, D., Zhu, F., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00342 · Cited by 181 · cited 1×
“Other noteworthy methods include those conducted in [33], who attempted to identify spatio-temporal anomalies by combining an EfficientNet with a Gated Recurrent Unit (GRU).”
MethodEfficient Document Re-Ranking for Transformers by Precomputing Term Representations
MacAvaney, S., Nardini, F. M., Perego, R., Tonellotto, N., Goharian, N., & Frieder, O. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401093 · Cited by 176
Fine-Grained Visual Textual Alignment for Cross-Modal Retrieval Using Transformer Encoders
Messina, N., Amato, G., Esuli, A., Falchi, F., Gennaro, C., & Marchand-Maillet, S. (2021) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3451390 · Cited by 149
Self-citationDeepFake video detection using convolutional vision transformer
Wodajo, D., & Atnafu, S. (2021) · arXiv · doi:10.48550/arxiv.2102.11126 · Cited by 147
Fighting Deepfakes by Detecting GAN DCT Anomalies
Giudice, O., Guarnera, L., & Battiato, S. (2021) · Journal of Imaging · doi:10.3390/jimaging7080128 · Cited by 118
DeepFaceLab: A simple, flexible and extensible face swapping framework.
Perov, I., Gao, D., Chervoniy, N., Liu, K., Marangonda, S., Umé, C., Dpfks, Facenheim, C. S., Rp, L., Jiang, J., et al. (2020) · Cited by 106
Transformer Reasoning Network for Image- Text Matching and Retrieval
Messina, N., Falchi, F., Esuli, A., & Amato, G. M. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9413172 · Cited by 88
Self-citationDeepFaceLab: Integrated, flexible and extensible face-swapping framework
Perov, I., Gao, D., Chervoniy, N., Liu, K., Marangonda, S., Umé, C., Dpfks, Facenheim, C. S., RP, L., Jiang, J., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.05535 · Cited by 86
DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery DetectionPreprint
Jiang, L., Li, R., Wu, W., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.03024 · Cited by 65
Deepfake Detection using Spatiotemporal Convolutional Networks
de Lima, O., Franklin, S., Basu, S., Karwoski, B., & George, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14749 · Cited by 62
Virtual to Real Adaptation of Pedestrian Detectors
Ciampi, L., Messina, N., Falchi, F., Gennaro, C., & Amato, G. M. (2020) · Sensors · doi:10.3390/s20185250 · Cited by 50
Self-citationDeepfake Detection Scheme Based on Vision Transformer and Distillation
Heo, Y. J., Choi, Y. J., Lee, Y., & Kim, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01353 · Cited by 38
Learning Pedestrian Detection from Virtual Worlds
Amato, G. M., Ciampi, L., Falchi, F., Gennaro, C., & Messina, N. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30642-7_27 · Cited by 35
Self-citationCrossViT: Cross-Attention Multi-Scale Vision Transformer for Image ClassificationPreprint
Chen, C.-F., Fan, Q., & Panda, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.14899 · Cited by 23
VISIONE at Video Browser Showdown 2021
Amato, G. M., Bolettieri, P., Falchi, F., Gennaro, C., Messina, N., Vadicamo, L., & Vairo, C. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-67835-7_47 · Cited by 20
Self-citationSolving the same-different task with convolutional neural networks
Messina, N., Amato, G. M., Carrara, F., Gennaro, C., & Falchi, F. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.12.019 · Cited by 18
Self-citationTesting Deep Neural Networks on the Same-Different Task
Messina, N., Amato, G. M., Carrara, F., Falchi, F., & Gennaro, C. (2019) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi.2019.8877412 · Cited by 9
Self-citationTowards Efficient Cross-Modal Visual Textual Retrieval using Transformer-Encoder Deep Features
Messina, N., Amato, G. M., Falchi, F., Gennaro, C., & Marchand‐Maillet, S. (2021) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi50038.2021.9461890 · Cited by 9
Self-citation
Research works printed but not linked: 1
- Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 28, 91–99 (2015)
Cited by
A Survey on the Detection and Impacts of Deepfakes in Visual, Audio, and Textual Formats
Mubarak, R., Alsboui, T., Alshaikh, O., Inuwa-Dutse, I., Khan, S., & Parkinson, S. (2023) · IEEE Access · doi:10.1109/access.2023.3344653 · Cited by 132
“VAEs are often used for signal analysis tasks in deepfake generation [40], [41].”
MethodA review of deep learning‐based approaches for deepfake content detection
(2024) · doi:10.1111/exsy.13570 · Cited by 116
“The same authors also found that coupling higher-order wavelets and CNN attained an improved and competitive performance compared to a Discrete Cosine Transformer (DCT) approach or working directly on the raw images, where combined architectures show the best performance.”
ResultInfluentialAbbas, F., & Taeihagh, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124260 · Cited by 186
“Gener - ally, a face-swapping method comprises three stages (Ding et al., 2022; Coccomini et al., 2022; Sun et al., 2023).”
BackgroundMethodComparison of Deepfake Detection Techniques through Deep Learning
Taeb, M., & Chi, H. (2022) · Journal of Cybersecurity and Privacy · doi:10.3390/jcp2010007 · Cited by 109
“The main difference between deepfake video- and image-detection methods is that video-detection methods can leverage spatial features [14], spatio-temporal anomalies [15] and supervised domain [16] to draw a conclusion on the whole video by aggregating the inferred output both in time and across multiple faces.”
BackgroundDeep convolutional pooling transformer for deepfake detection
Wang, T., Cheng, H., Chow, K. P., & Nie, L. (2023) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3588574 · Cited by 111
“The three experiments are conducted upon the model with depths [4, 4, 6], [8, 8, 8], and [10, 10, 12] for the three phases and having the total depths of 14, 24, and 32 from shallow to deep, respectively.”
Method
Metadata
- Authors
- Coccomini, D. A., Messina, N., Gennaro, C., & Falchi, F.
- Affiliations
- Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" (IT)
- Corresponding author
- Messina, Nicola
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2022
- Pages
- 219-229
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W3179508259
- Cited by
- 296
- References
- 43
- Citations by year
- 2026: 54 · 2025: 85 · 2024: 80 · 2023: 56 · 2022: 21
- FWCI
- 21.94
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- ISTI Open Portal (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- face manipulation, EfficientNet, vision transformer, CNN-Transformer hybrid, feature extraction, DFDC, generative adversarial networks, variational autoencoder
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.