CoChat

Comprehensive Evaluation of Deepfake Detection Models: Accuracy, Generalization, and Resilience to Adversarial Attacks

Open access
Abbasi, Maryam, Váz, Paulo, Silva, José, Martins, Pedro · 2025 · Applied Sciences

The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index; 3 of the 18 sources its claims cite were checked for support.

Abstract

The rise of deepfakes—synthetic media generated using artificial intelligence—threatens digital content authenticity, facilitating misinformation and manipulation. However, deepfakes can also depict real or entirely fictitious individuals, leveraging state-of-the-art techniques such as generative adversarial networks (GANs) and emerging diffusion-based models. Existing detection methods face challenges with generalization across datasets and vulnerability to adversarial attacks. This study focuses on subsets of frames extracted from the DeepFake Detection Challenge (DFDC) and FaceForensics++ videos to evaluate three convolutional neural network architectures—XCeption, ResNet, and VGG16—for deepfake detection. Performance metrics include accuracy, precision, F1-score, AUC-ROC, and Matthews Correlation Coefficient (MCC), combined with an assessment of resilience to adversarial perturbations via the Fast Gradient Sign Method (FGSM). Among the tested models, XCeption achieves the highest accuracy (89.2% on DFDC), strong generalization, and real-time suitability, while VGG16 excels in precision and ResNet provides faster inference. However, all models exhibit reduced performance under adversarial conditions, underscoring the need for enhanced resilience. These findings indicate that robust detection systems must consider advanced generative approaches, adversarial defenses, and cross-dataset adaptation to effectively counter evolving deepfake threats.

Overview

CoChat Summary

TL;DR

This paper compares XCeption, ResNet-50, VGG16, and MobileNet V2 for deepfake detection on DFDC and FaceForensics++, with a focus on accuracy, cross-dataset generalization, speed, and robustness to FGSM attacks. XCeption is the strongest overall performer on DFDC, while all models lose performance under adversarial perturbations and when transferred across datasets.

Key findings

  • On DFDC, XCeption achieved the best per-frame performance: 89.2% accuracy, 88.9% precision, 90.5% recall, 89.7% F1, and 0.94 AUC-ROC; its inference time was 85 ms per frame.
  • VGG16 was competitive on DFDC with 85.5% accuracy, 87.0% F1, and 0.90 AUC-ROC, but it was much slower at 1020 ms per frame.
  • ResNet-50 had lower DFDC accuracy at 72.8% and 0.81 AUC-ROC, but was faster than VGG16 at 270 ms per frame; MobileNet V2 reached 80.2% accuracy and the lowest inference time at 60 ms per frame.
  • Under FGSM attack with ϵ = 0.01, accuracy dropped for all models: XCeption from 89.2% to 79.1%, ResNet-50 from 72.8% to 64.2%, VGG16 from 85.5% to 74.3%, and MobileNet V2 from 80.2% to 69.4%.
  • Cross-dataset testing on FaceForensics++ without fine-tuning reduced accuracy for all models; XCeption declined from 89.2% to 85.7%, and fine-tuning on 10% of FaceForensics++ raised it to 88.6%.
  • Statistical tests on accuracy found all pairwise model comparisons significant, with p < 0.01 for every comparison in Table 7.

Methods

  • The study evaluated XCeption, ResNet-50, VGG16, and MobileNet V2 using frame-level deepfake detection on two datasets: DFDC and FaceForensics++.
  • Videos were sampled at 30 fps with up to 50 frames extracted per video; faces were detected and cropped using MTCNN with a confidence threshold of 0.9, and images were resized and normalized before training.
  • Models were trained on DFDC with ImageNet initialization; performance was measured using accuracy, precision, recall, F1, AUC-ROC, and MCC, and adversarial robustness was tested with FGSM (ϵ = 0.01).

Limitations

  • The paper states that cross-dataset generalization remains challenging, as accuracy drops when models trained on DFDC are tested on FaceForensics++ without fine-tuning.
  • The paper states that none of the evaluated models is fully resilient to adversarial attacks, and all degrade under FGSM perturbations.
  • Evident from the text: the evaluation is frame-level and limited to two datasets and three main CNN architectures plus MobileNet V2, so the findings may not cover other modalities, architectures, or video-level/end-to-end settings.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/app15031225

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

3 Valid · 0 Not valid · 12 Not checked · 83 Own results and interpretations

50 more claims

Create a free account to see all 98 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 24
  • It may state two separate claims as one. 26
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

4 of 4

Citations

Cites

Research works printed but not linked: 2

  1. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Adv. Neural Inf. Process. Syst., 27.
  2. Dang, 2020, Adversarial attacks on deep learning-based face recognition: A comprehensive review, IEEE Trans. Artif. Intell., 1, 168

Cited by

See all 32

Metadata

Authors
Abbasi, M., Váz, P., Silva, J., & Martins, P.
Affiliations
  • Universidade Politécnica de Coimbra (PT)
  • Universidade Politécnica de Viseu (PT)
Corresponding author
Martins, Pedro
Venue
Applied Sciences
Publisher
MDPI AG
Published
2025-01-25
Volume
15
Issue
3
Pages
1225
ISSN
2076-3417 (ISSN-L)
OpenAlex
W4406845293
Cited by
32
References
19
Citations by year
2026: 21 · 2025: 11
FWCI
11.12
Type
journal article
Language
en
Open access
Locations
  • Applied Sciences (publishedVersion, cc-by)
  • Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • FCT—Foundation for Science and Technology, I.P. (UIDB/05583/2020, CEECINST/00077/2021)
  • Research Center in Digital Services (CISeD) (UIDB/05583/2020, CEECINST/00077/2021)
  • Instituto Politécnico de Viseu (UIDB/05583/2020, CEECINST/00077/2021)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
deepfake detection, synthetic media, adversarial robustness, FGSM, cross-dataset generalization, Xception, ResNet, VGG16, DFDC, FaceForensics++, video forensics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.