Comprehensive Evaluation of Deepfake Detection Models: Accuracy, Generalization, and Resilience to Adversarial Attacks
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index; 3 of the 18 sources its claims cite were checked for support.
Abstract
The rise of deepfakes—synthetic media generated using artificial intelligence—threatens digital content authenticity, facilitating misinformation and manipulation. However, deepfakes can also depict real or entirely fictitious individuals, leveraging state-of-the-art techniques such as generative adversarial networks (GANs) and emerging diffusion-based models. Existing detection methods face challenges with generalization across datasets and vulnerability to adversarial attacks. This study focuses on subsets of frames extracted from the DeepFake Detection Challenge (DFDC) and FaceForensics++ videos to evaluate three convolutional neural network architectures—XCeption, ResNet, and VGG16—for deepfake detection. Performance metrics include accuracy, precision, F1-score, AUC-ROC, and Matthews Correlation Coefficient (MCC), combined with an assessment of resilience to adversarial perturbations via the Fast Gradient Sign Method (FGSM). Among the tested models, XCeption achieves the highest accuracy (89.2% on DFDC), strong generalization, and real-time suitability, while VGG16 excels in precision and ResNet provides faster inference. However, all models exhibit reduced performance under adversarial conditions, underscoring the need for enhanced resilience. These findings indicate that robust detection systems must consider advanced generative approaches, adversarial defenses, and cross-dataset adaptation to effectively counter evolving deepfake threats.
Overview
CoChat Summary
TL;DR
This paper compares XCeption, ResNet-50, VGG16, and MobileNet V2 for deepfake detection on DFDC and FaceForensics++, with a focus on accuracy, cross-dataset generalization, speed, and robustness to FGSM attacks. XCeption is the strongest overall performer on DFDC, while all models lose performance under adversarial perturbations and when transferred across datasets.
Key findings
- On DFDC, XCeption achieved the best per-frame performance: 89.2% accuracy, 88.9% precision, 90.5% recall, 89.7% F1, and 0.94 AUC-ROC; its inference time was 85 ms per frame.
- VGG16 was competitive on DFDC with 85.5% accuracy, 87.0% F1, and 0.90 AUC-ROC, but it was much slower at 1020 ms per frame.
- ResNet-50 had lower DFDC accuracy at 72.8% and 0.81 AUC-ROC, but was faster than VGG16 at 270 ms per frame; MobileNet V2 reached 80.2% accuracy and the lowest inference time at 60 ms per frame.
- Under FGSM attack with ϵ = 0.01, accuracy dropped for all models: XCeption from 89.2% to 79.1%, ResNet-50 from 72.8% to 64.2%, VGG16 from 85.5% to 74.3%, and MobileNet V2 from 80.2% to 69.4%.
- Cross-dataset testing on FaceForensics++ without fine-tuning reduced accuracy for all models; XCeption declined from 89.2% to 85.7%, and fine-tuning on 10% of FaceForensics++ raised it to 88.6%.
- Statistical tests on accuracy found all pairwise model comparisons significant, with p < 0.01 for every comparison in Table 7.
Methods
- The study evaluated XCeption, ResNet-50, VGG16, and MobileNet V2 using frame-level deepfake detection on two datasets: DFDC and FaceForensics++.
- Videos were sampled at 30 fps with up to 50 frames extracted per video; faces were detected and cropped using MTCNN with a confidence threshold of 0.9, and images were resized and normalized before training.
- Models were trained on DFDC with ImageNet initialization; performance was measured using accuracy, precision, recall, F1, AUC-ROC, and MCC, and adversarial robustness was tested with FGSM (ϵ = 0.01).
Limitations
- The paper states that cross-dataset generalization remains challenging, as accuracy drops when models trained on DFDC are tested on FaceForensics++ without fine-tuning.
- The paper states that none of the evaluated models is fully resilient to adversarial attacks, and all degrade under FGSM perturbations.
- Evident from the text: the evaluation is frame-level and limited to two datasets and three main CNN architectures plus MobileNet V2, so the findings may not cover other modalities, architectures, or video-level/end-to-end settings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 0 Not valid · 12 Not checked · 83 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 2 compares accuracy under various distortions with XCeption showing higher robustness.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The metrics highlight significant differences in how these models localize and interpret manipulation artifacts.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the tested models, XCeption achieves the highest accuracy (89.2% on DFDC)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It achieves a low false positive rate (10.9%) and high recall (90.5%).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Expected Calibration Error (ECE) scores (Table 4) reveal that XCeption and VGG16 are better calibrated compared to ResNet-50.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
50 more claims
Create a free account to see all 98 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 24
- It may state two separate claims as one. 26
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.7. Visualization of Model Decisions
“| VGG16 | 76.7 | 0.74 | 7.9 | 88.1 |”
§ 4.7. Visualization of Model Decisions
“- VGG16: VGG16 strikes a balance between attention coverage (78.7%) and FRA (7.9%).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.7. Visualization of Model Decisions
“| XCeption | 82.5 | 0.77 | 5.1 | 91.6 |”
§ 4.7. Visualization of Model Decisions
“- XCeption: With the highest attention coverage (82.3%) and the lowest FRA (5.4%), XCeption demonstrates superior focus on manipulation-prone regions.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.7. Visualization of Model Decisions
“| XCeption | 82.5 | 0.77 | 5.1 | 91.6 |”
§ 4.7. Visualization of Model Decisions
“Its high activation intensity (0.78) and MDR (91.6%) indicate confidence and precision in detecting deepfake artifacts.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.7. Visualization of Model Decisions
“Table 8. Quantitative analysis of Grad-CAM activations across models. | Model | Attention Coverage (%) | Avg. Activation Intensity | FRA (%) | MDR(%) | |-----------|--------------------------|-----------------------------|-----------|----------| | XCeption | 82.5 | 0.77 | 5.1 | 91.6 |”
§ 4.7. Visualization of Model Decisions
“- XCeption: With the highest attention coverage (82.3%) and the lowest FRA (5.4%), XCeption demonstrates superior focus on manipulation-prone regions.”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380
Towards Evaluating the Robustness of Neural Networks
Carlini, N., & Wagner, D. (2017) · 2017 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2017.49 · Cited by 8,859
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
MesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Deepfakes and beyond: A Survey of face manipulation and fake detection
Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., & Ortega-Garcia, J. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.014 · Cited by 1,237
Face X-Ray for More General Face Forgery Detection
Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., & Guo, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00505 · Cited by 1,211
In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking
Li, Y., Chang, M.-C., & Lyu, S. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630787 · Cited by 1,089
Media Forensics and DeepFakes: An Overview
Verdoliva, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3002101 · Cited by 979
Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security
Chesney, R., & Citron, D. K. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3213954 · Cited by 880
Bayar, B., & Stamm, M. C. (2018) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2018.2825953 · Cited by 599
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2016) · arXiv · doi:10.48550/arxiv.1610.02357 · Cited by 538
The deepfake detection challenge (dfdc) dataset
Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., & Ferrer, C. C. (2020) · arXiv · doi:10.48550/arxiv.2006.07397 · Cited by 427
The Deepfake Detection Challenge (DFDC) Preview Dataset
Dolhansky, B., Howes, R., Pflaum, B., Baram, N., & Ferrer, C. C. (2019) · arXiv · doi:10.48550/arxiv.1910.08854 · Cited by 413
In Ictu Oculi: Exposing AI Generated Fake Face Videos by Detecting Eye Blinking
Li, Y., Chang, M., & Lyu, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.02877 · Cited by 169
Peng, C., Miao, Z., Liu, D., Wang, N., Hu, R., & Gao, X. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3381823 · Cited by 93
A Comparative Study: Deepfake Detection Using Deep-learning
Khatri, N., Borar, V., & Garg, R. (2023) · 2023 13th International Conference on Cloud Computing, Data Science & Engineering (Confluence) · doi:10.1109/confluence56041.2023.10048888 · Cited by 29
LARNet: Towards Lightweight, Accurate and Real-Time Salient Object Detection
Wang, Z., Zhang, Y., Liu, Y., Qin, C., Coleman, S., & Kerr, D. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3330082 · Cited by 21
PM-DETR: Domain Adaptive Prompt Memory for Object Detection with TransformersPreprint
Jia, P., Liu, J., Yang, S., Wu, J., Xie, X., & Zhang, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.00313 · Cited by 15
Progressive Source-Aware Transformer for Generalized Source-Free Domain Adaptation
Tang, S. Y., Shi, Y., Song, Z., Ye, M., Zhang, C., & Zhang, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3321421 · Cited by 14
Kumar, K., Majumdar, A., Kumar, A. A., & Chandra, M. G. (2023) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco58844.2023.10289755 · Cited by 4
Bhattacharyya, C., Wang, H., Zhang, F., Kim, S., & Zhu, X. (2024) · arXiv · doi:10.48550/arxiv.2404.01579 · Cited by 4
Research works printed but not linked: 2
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. Adv. Neural Inf. Process. Syst., 27.
- Dang, 2020, Adversarial attacks on deep learning-based face recognition: A comprehensive review, IEEE Trans. Artif. Intell., 1, 168
Cited by
Alkhatib, M. (2025) · Computers · doi:10.3390/computers14110488 · Cited by 10
Exploring the Advancements and Challenges of Deepfake Face-swap: A Survey
Bakar, S. A. R. B. S. A., Waseem, S., Omar, Z., & Bilalashfaqahmed (2026) · Multimedia Tools and Applications · doi:10.1007/s11042-026-21285-8 · Cited by 15
Explainability-driven adversarial robustness assessment for generalized deepfake detectors
Cirillo, L., Gervasio, A., & Amerini, I. (2025) · EURASIP Journal on Information Security · doi:10.1186/s13635-025-00211-9 · Cited by 11
“Such attacks, which have already been tested on deepfake detectors [38, 39], except GNP, try to fool the model by maximizing its loss function.”
Real-Time Deepfake Video Detection using Machine Learning: A CNN-based Authentication Framework
Babitha, S., Yadavamuthiah, K., Dhanush Harsha., J., V, & Hariharan, J. (2025) · International Conference on Augmented Intelligence and Sustainable Systems (ICAISS) · doi:10.1109/icaiss61471.2025.11041947 · Cited by 2
“This work is very important since it improves the robustness to high-resolution manipulation by adversarial attacks, making for a scalable and effective solution in the field of digital security [13].”
Adversarial Attacks on Deepfake Detectors: A Challenge in the Era of AI-Generated Media (AADD-2025)
Battiato, S., Casu, M., Guarnera, F., Guarnera, L., Puglisi, G., Pontorno, O., Ragaglia, C. V., & Akhtar, Z. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3761983 · Cited by 12
“The domain of deepfake detection has become a dynamic field of study, with a primary focus on developing classifiers that can identify sophisticated forgeries [1].”
Influential
Metadata
- Authors
- Abbasi, M., Váz, P., Silva, J., & Martins, P.
- Affiliations
- Universidade Politécnica de Coimbra (PT)
- Universidade Politécnica de Viseu (PT)
- Corresponding author
- Martins, Pedro
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2025-01-25
- Volume
- 15
- Issue
- 3
- Pages
- 1225
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4406845293
- Cited by
- 32
- References
- 19
- Citations by year
- 2026: 21 · 2025: 11
- FWCI
- 11.12
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- FCT—Foundation for Science and Technology, I.P. (UIDB/05583/2020, CEECINST/00077/2021)
- Research Center in Digital Services (CISeD) (UIDB/05583/2020, CEECINST/00077/2021)
- Instituto Politécnico de Viseu (UIDB/05583/2020, CEECINST/00077/2021)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- deepfake detection, synthetic media, adversarial robustness, FGSM, cross-dataset generalization, Xception, ResNet, VGG16, DFDC, FaceForensics++, video forensics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.