Cited by 49Digital Media Forensic DetectionGenerative Adversarial Networks and Image SynthesisMisinformation and Its Impacts

Cross-Forgery Analysis of Vision Transformers and CNNs for Deepfake Image Detection

Open access
Coccomini, Davide Alessandro, Caldelli, Roberto, Falchi, Fabrizio, Gennaro, Claudio, Amato, Giuseppe·2022·Proceedings of the 1st International Workshop on Multimedia AI against Disinformation

Abstract

Deepfake Generation Techniques are evolving at a rapid pace, making it possible to create realistic manipulated images and videos and endangering the serenity of modern society. The continual emergence of new and varied techniques brings with it a further problem to be faced, namely the ability of deepfake detection models to update themselves promptly in order to be able to identify manipulations carried out using even the most recent methods. This is an extremely complex problem to solve, as training a model requires large amounts of data, which are difficult to obtain if the deepfake generation method is too recent. Moreover, continuously retraining a network would be unfeasible. In this paper, we ask ourselves if, among the various deep learning techniques, there is one that is able to generalise the concept of deepfake to such an extent that it does not remain tied to one or more specific deepfake generation methods used in the training set. We compared a Vision Transformer with an EfficientNetV2 on a cross-forgery context based on the ForgeryNet dataset. From our experiments, It emerges that EfficientNetV2 has a greater tendency to specialize often obtaining better results on training methods while Vision Transformers exhibit a superior generalization ability that makes them more competent even on images generated with new methodologies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3512732.3533582

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The attention mechanism enables the model to abstract better the concept of deepfake, but only with the constraint of availability of a large quantity and diversity of data used during training.

    “This result probably stems from the fact that the training set consists of significantly more images than in the previous setup and it is strongly in line with what is presented in [39].”

    From On the Generalization of Deep Learning Models in Video Deepfake Detection (2023) · cites this paper as [39]

    The Conclusions report that Vision Transformers better generalise the concept of deepfake than convolutional networks. Section 4.2 explores whether training on varied generation methods improves generalisation, but does not establish that an attention mechanism causes the advantage or that a large, diverse training set is required. Section 4.1 also reports generalisation when training used only one generation method at a time. Page numbers were not available in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • The Vision Transformer's superior generalization stems from the larger quantity and diversity of training data, which enables it to abstract the concept of deepfake better.

    “This result probably stems from the fact that the training set consists of significantly more images than in the previous setup and it is strongly in line with what presented in [39].”

    From On the Generalization of Deep Learning Models in Video Deepfake Detection (2023) · cites this paper as [39]

    Sections 4.2 and 4.3 report that the Vision Transformer generalizes better across forgery methods and suggest that training on varied methods might improve generalization. They do not establish that its advantage stems from more or more diverse training data: both architectures were trained on the same sub-datasets, and Section 4.3 reports the Transformer's lower variance regardless of the methods used to build the training set. Page numbers are unavailable in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • The Vision Transformer's superior generalization in the multiple methods training setup probably stems from the training set consisting of significantly more images than in the previous setup.

    “This result probably stems from the fact that the training set consists of significantly more images than in the previous setup and it is strongly in line with what is presented in [39].”
  • EfficientNetV2 performs better in training techniques, but ViTs are more proficient in generalisation, making them superior in detecting deepfakes.

    “EfficientNetV2 performs better in training techniques, but ViTs are more proficient in generalisation, making them superior in detecting deepfakes.”

    From Deepfake video detection: challenges and opportunities (2024) · cites this paper as Coccomini et al. 2022

  • Coccomini et al. (2022) compared CNNs and Vision Transformers (ViTs) for deepfake image detection using the ForgeryNet dataset to assess cross-forgery performance.

    “Coccomini etal. (2022) compare CNNs and Vision Transformers (ViTs) in the context of deepfake image detection. They used the ForgeryNet dataset to assess the efficiency of their cross-forgery performance.”

    From Deepfake video detection: challenges and opportunities (2024) · cites this paper as Coccomini et al. 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. [18]18. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8110-8119.
  2. [33]33. Mingxing Tan and Quoc V. Le. 2021. EfficientNetV2: Smaller Models and Faster Training. (2021). https://doi.org/10.48550/ARXIV.2104.00298 · DOI

Cited by

Sorted by citations

Authors
Coccomini, D. A., Caldelli, R., Falchi, F., Gennaro, C., & Amato, G.
Affiliations
  • Istituto di Scienza e Tecnologie dell'Informazione "Alessandro Faedo" (IT)
  • Consorzio Nazionale Interuniversitario per le Telecomunicazioni (IT)
Venue
Proceedings of the 1st International Workshop on Multimedia AI against Disinformation
Publisher
ACM
Published
2022-06-27
Pages
52-58
OpenAlex
W4283452332
Cited by
47
References
33
Citations by year
2026: 13 · 2025: 11 · 2024: 16 · 2023: 6 · 2022: 1
FWCI
3.08
Type
conference paper
Language
en
Open access
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, cc-by)
  • ISTI Open Portal (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
AI4Media (951911)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
deepfake detection, vision transformer, EfficientNetV2, convolutional neural networks, image forgery detection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review