Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 90 of the 96 references it cites were found in a registry or an index; 29 of the 34 sources its claims cite were checked for support.

DEEPFAKER: A Unified Evaluation Platform for Facial Deepfake and Detection Models

Open access
Wang, Li, Meng, Xiangtao, Li, Dan, Zhang, Xuhong, Ji, Shouling, Guo, Shanqing · 2024 · ACM Transactions on Privacy and Security

Abstract

Deepfake data contains realistically manipulated faces—its abuses pose a huge threat to the security and privacy-critical applications. Intensive research from academia and industry has produced many deepfake/detection models, leading to a constant race of attack and defense. However, due to the lack of a unified evaluation platform, many critical questions on this subject remain largely unexplored. How is the anti-detection ability of the existing deepfake models? How generalizable are existing detection models against different deepfake samples? How effective are the detection APIs provided by the cloud-based vendors? How evasive and transferable are adversarial deepfakes in the lab and real-world environment? How do various factors impact the performance of deepfake and detection models? To bridge the gap, we design and implement DEEPFAKER 1 a unified and comprehensive deepfake detection evaluation platform. Specifically, DEEPFAKER has integrated 10 state-of-the-art deepfake methods and 9 representative detection methods, while providing a user-friendly interface and modular design that allows for easy integration of new methods. Leveraging DEEPFAKER , we conduct a large-scale empirical study of facial deepfake/detection models and draw a set of key findings: (i) the detection methods have poor generalization on samples generated by different deepfake methods; (ii) there is no significant correlation between anti-detection ability and visual quality of deepfake samples; (iii) the current detection APIs have poor detection performance and adversarial deepfakes can achieve about 70% attack success rate on all cloud-based vendors, calling for an urgent need to deploy effective and robust detection APIs; (iv) the detection methods in the lab are more robust against transfer attacks than the detection APIs in the real-world environment; and (v) deepfake videos may not always be more difficult to detect after video compression. We envision that DEEPFAKER will benefit future research on facial deepfake and detection.

Overview

CoChat Summary

TL;DR

DEEPFAKER is a unified, modular platform integrating 10 deepfake methods, 9 detection methods, and 3 cloud detection APIs to evaluate facial deepfake and detection models in lab and real-world settings. Using it, the authors find poor detection generalization, weak commercial API performance, and substantial vulnerability to adversarial deepfakes.

Key findings

  • Detection methods generalize poorly to unseen deepfake samples: average performance across all detection methods is 50%–70% on all deepfake samples, and CViT scores 96.73% AUC on ICface but only 41.73% on FaceShifter.
  • Anti-detection ability does not correlate significantly with visual quality: FaceShifter has a slightly stronger average anti-detection score than FOMM (51.12% vs. 53.64% AUC), despite FaceShifter videos showing more obvious visual artifacts.
  • Cloud detection APIs perform poorly: FNRs exceed 60% for all APIs, Baidu misclassifies all fake data as real (FNR 100%), and adversarial deepfakes achieve about 70% ASR on all cloud-based vendors; on Tencent, original deepfake video FNR is 61.5%, while CViT-based adversarial video ASR is 93.2%.
  • Lab detection methods are more robust against transfer attacks than real-world detection APIs, and compressed deepfake videos are not always harder to detect: FOMM videos become easier to detect under strong compression, while FSGAN CViT ACC drops from 89.43% at C0 to 65.97% at C40.

Methods

  • DEEPFAKER integrates 10 SOTA deepfake methods (3 face swapping, 4 face reenactment, 3 attribute editing), 9 representative detection methods, and 3 commercial detection APIs (Baidu Cloud, Tencent Cloud, Deepware), using Docker containers and a modular Flask-based architecture.
  • Evaluation data use CelebA-HQ source images and driving videos from FaceForensics++, Celeb-DF, and SiW-M; each swapping/reenactment method generates 3,000 fake videos (50 source images × 60 driving videos), and each attribute-editing method generates 1,000 fake images (200 identities × 5 attributes), yielding a dataset of 21,000 deepfake videos and 24,000 deepfake images.
  • Main analyses use AUC for detection/deepfake performance, FNR and FPR for detection APIs, and ASR for adversarial robustness; adversarial deepfakes are generated with iterative gradient sign, FGSM, and PGD under white-box and black-box transfer settings, plus case studies on facial similarity, input quality, and video compression.

Limitations

  • The paper states that DEEPFAKER integrates 10 deepfake methods and 9 detection models, which cannot cover all models due to evolving technology and incomplete open-source code, though its modular design supports new methods.
  • The paper states that because many methods do not release training code, it uses open-source pre-trained models and cannot uniformly explore training-phase factors such as training dataset or model architecture.
  • The paper states that it focuses on visual deepfake images and videos, not audio or text modalities, and plans future extension; an evident limitation is that the real-world API evaluation covers only three vendors and depends on their current returned results and recommended thresholds.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3634914

Cites

90 of 90 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

17 Valid · 6 Not valid · 5 Not checked · 116 Own results and interpretations

81 more claims · 4 with their cited sources checked

Create a free account to see all 144 claims and to verify the 3 references still open.

See all claims — free account
  • It may state two separate claims as one. 41
  • Probably not a substantial claim. 25
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 96 references

Research works printed but not linked: 3

  1. Ivana Chingovska, André Anjos, and Sébastien Marcel. 2012. On the effectiveness of local binary patterns in face anti-spoofing. In Proceedings of the International Conference of the Biometrics Special Interest Group (BIOSIG’12). IEEE, Los Alamitos, CA, 1–7.
  2. Paul Ekman and Wallace V. Friesen. 1978. FacialActionCodingSystem:A Technique for the Measurement of FacialMovement. Consulting Psychologists Press Palo Alto CA.
  3. 10.5555/1324787 · doi:10.5555/1324787

Cited by

See all 11

Metadata

Authors
Wang, L., Meng, X., Li, D., Zhang, X., Ji, S., & Guo, S.
Affiliations
  • Shandong University (CN)
  • China Electronic Product Reliability and Environmental Test Institute (CN)
  • Zhejiang University (CN)
Venue
ACM Transactions on Privacy and Security
Published
2024-02-05
Volume
27
Issue
1
Pages
1-34
ISSN
  • 2471-2566 (ISSN-L)
  • 2471-2574
OpenAlex
W4389140876
Cited by
11
References
96
Citations by year
2026: 2 · 2025: 5 · 2024: 3
FWCI
1.00
Type
journal article
Language
en
Open access
Locations
ACM Transactions on Privacy and Security (publishedVersion)
Funders
  • National Natural Science Foundation of China (61902148)
  • Shandong Provincial Natural Science Foundation (ZR2020MF055, ZR2021LZH007, ZR2020LZH002, ZR2020QF045)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
Keywords
deepfake generation, evaluation platform, transfer attacks, video compression

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: DEEPFAKER: A Unified Evaluation Platform for Facial Deepfake and Detection Models. https://app.cochat.ai/papers/10.1145/3634914
The paper
Wang, L., Meng, X., Li, D., Zhang, X., Ji, S., & Guo, S. (2024). DEEPFAKER: A Unified Evaluation Platform for Facial Deepfake and Detection Models. ACM Transactions on Privacy and Security. https://doi.org/10.1145/3634914

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.