The DOI matches its Crossref record; 90 of the 96 references it cites were found in a registry or an index; 29 of the 34 sources its claims cite were checked for support.
DEEPFAKER: A Unified Evaluation Platform for Facial Deepfake and Detection Models
Open accessAbstract
Deepfake data contains realistically manipulated faces—its abuses pose a huge threat to the security and privacy-critical applications. Intensive research from academia and industry has produced many deepfake/detection models, leading to a constant race of attack and defense. However, due to the lack of a unified evaluation platform, many critical questions on this subject remain largely unexplored. How is the anti-detection ability of the existing deepfake models? How generalizable are existing detection models against different deepfake samples? How effective are the detection APIs provided by the cloud-based vendors? How evasive and transferable are adversarial deepfakes in the lab and real-world environment? How do various factors impact the performance of deepfake and detection models? To bridge the gap, we design and implement DEEPFAKER 1 a unified and comprehensive deepfake detection evaluation platform. Specifically, DEEPFAKER has integrated 10 state-of-the-art deepfake methods and 9 representative detection methods, while providing a user-friendly interface and modular design that allows for easy integration of new methods. Leveraging DEEPFAKER , we conduct a large-scale empirical study of facial deepfake/detection models and draw a set of key findings: (i) the detection methods have poor generalization on samples generated by different deepfake methods; (ii) there is no significant correlation between anti-detection ability and visual quality of deepfake samples; (iii) the current detection APIs have poor detection performance and adversarial deepfakes can achieve about 70% attack success rate on all cloud-based vendors, calling for an urgent need to deploy effective and robust detection APIs; (iv) the detection methods in the lab are more robust against transfer attacks than the detection APIs in the real-world environment; and (v) deepfake videos may not always be more difficult to detect after video compression. We envision that DEEPFAKER will benefit future research on facial deepfake and detection.
Overview
CoChat Summary
TL;DR
DEEPFAKER is a unified, modular platform integrating 10 deepfake methods, 9 detection methods, and 3 cloud detection APIs to evaluate facial deepfake and detection models in lab and real-world settings. Using it, the authors find poor detection generalization, weak commercial API performance, and substantial vulnerability to adversarial deepfakes.
Key findings
- Detection methods generalize poorly to unseen deepfake samples: average performance across all detection methods is 50%–70% on all deepfake samples, and CViT scores 96.73% AUC on ICface but only 41.73% on FaceShifter.
- Anti-detection ability does not correlate significantly with visual quality: FaceShifter has a slightly stronger average anti-detection score than FOMM (51.12% vs. 53.64% AUC), despite FaceShifter videos showing more obvious visual artifacts.
- Cloud detection APIs perform poorly: FNRs exceed 60% for all APIs, Baidu misclassifies all fake data as real (FNR 100%), and adversarial deepfakes achieve about 70% ASR on all cloud-based vendors; on Tencent, original deepfake video FNR is 61.5%, while CViT-based adversarial video ASR is 93.2%.
- Lab detection methods are more robust against transfer attacks than real-world detection APIs, and compressed deepfake videos are not always harder to detect: FOMM videos become easier to detect under strong compression, while FSGAN CViT ACC drops from 89.43% at C0 to 65.97% at C40.
Methods
- DEEPFAKER integrates 10 SOTA deepfake methods (3 face swapping, 4 face reenactment, 3 attribute editing), 9 representative detection methods, and 3 commercial detection APIs (Baidu Cloud, Tencent Cloud, Deepware), using Docker containers and a modular Flask-based architecture.
- Evaluation data use CelebA-HQ source images and driving videos from FaceForensics++, Celeb-DF, and SiW-M; each swapping/reenactment method generates 3,000 fake videos (50 source images × 60 driving videos), and each attribute-editing method generates 1,000 fake images (200 identities × 5 attributes), yielding a dataset of 21,000 deepfake videos and 24,000 deepfake images.
- Main analyses use AUC for detection/deepfake performance, FNR and FPR for detection APIs, and ASR for adversarial robustness; adversarial deepfakes are generated with iterative gradient sign, FGSM, and PGD under white-box and black-box transfer settings, plus case studies on facial similarity, input quality, and video compression.
Limitations
- The paper states that DEEPFAKER integrates 10 deepfake methods and 9 detection models, which cannot cover all models due to evolving technology and incomplete open-source code, though its modular design supports new methods.
- The paper states that because many methods do not release training code, it uses open-source pre-trained models and cannot uniformly explore training-phase factors such as training dataset or model architecture.
- The paper states that it focuses on visual deepfake images and videos, not audio or text modalities, and plans future extension; an evident limitation is that the real-world API evaluation covers only three vendors and depends on their current returned results and recommended thresholds.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
17 Valid · 6 Not valid · 5 Not checked · 116 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“these videos generated by two subjects with the same identity are generally more challenging for detection [47].”
What the sources say
- Supportedcites [47] [47] Huy H Nguyen, Fuming Fang, Junichi Yamagishi, and Isao Echizen. 2019. Multi-task learning for detecting and segmenting manipulated facial images and videos. arXiv preprint arXiv:1906.06876 (2019). - source
Section 4.1 (p. 4) describes self-reenactment as using the same source and target video and calls it a more challenging benchmark than source-to-target reenactment. Section 4.2 (p. 5) says its images were harder to detect. The paper's comparison is specifically between these Face2Face datasets, rather than all videos involving subjects with the same identity. (LLM, conf=0.88)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“existing works have investigated the evasion and transferability of adversarial deepfakes in the lab setting”
What the sources say
- Supportedcites [6] [6] Nicholas Carlini and Hany Farid. 2020. Evading deepfake-image detectors with white-and black-box attacks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops . IEEE, 658ś659. - source
Section 4, “Black-Box Attack” (page not provided), describes testing whether adversarial images transfer between deepfake-image detectors in a controlled experiment. Section 6, “Conclusions” (page not provided), reports that both white-box and black-box attacks can make fake images evade detection. (LLM, conf=0.91)
- Supportedcites [46] [46] Paarth Neekhara, Brian Dolhansky, Joanna Bitton, and Cristian Canton Ferrer. 2021. Adversarial threats to deepfake detection: A practical perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . IEEE, 923ś932. - source
The Introduction says the paper studies whether adversarial examples can fool DeepFake detectors and transfer across detection methods. Section 4.3 reports experimental transfer attacks against unseen detectors, while Section 2.3 discusses earlier work on evading them. These passages support the claim about investigating evasion and transferability in an experimental setting. (LLM, conf=0.89)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“They lack a comprehensive experimental evaluation to demonstrate the practical eicacy of state-of-the-art methods.”
What the sources say
- Partly supportedcites [68] [68] Ruben Tolosana, Ruben Vera-Rodriguez, Julian Fierrez, Aythami Morales, and Javier Ortega-Garcia. 2020. Deepfakes and beyond: A survey of face manipulation and fake detection. Information Fusion 64 (2020), 131ś148. - sourceExperimental — accuracy not yet measured
In the introductory discussion of other surveys (page not available), the paper says its review adds a more detailed account of detection benchmarks and their results. That supports a narrower claim about the detail of earlier surveys, but it does not establish that existing surveys generally lack a comprehensive experimental evaluation of state-of-the-art methods. (LLM, conf=0.76)
- Could not verifycites [70] [70] Luisa Verdoliva. 2020. Media forensics and deepfakes: an overview. IEEE Journal of Selected Topics in Signal Processing 14, 5 (2020), 910ś932. - sourceExperimental — accuracy not yet measured
The Datasets section (p. 12) calls for comparing methods across multiple datasets, and the Deep Learning Considerations section (p. 12) stresses the importance of experimental protocols. Neither says that existing surveys lack a comprehensive experimental evaluation. Searches for that assertion and paraphrases of it found no supporting passage in our copy. (LLM, conf=0.76)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“current benchmarks [37], such as FaceForensics++ [57] and Celeb-DF [36], primarily focus on their own datasets and evaluate various detection methods on these public datasets.”
What the sources say
- Supportedcites [37] [37] Chenhao Lin, Jingyi Deng, Pengbin Hu, Chao Shen, Qian Wang, and Qi Li. 2022. Towards Benchmarking and Evaluating Deepfake Detection. arXiv preprint arXiv:2203.02115 (2022). - source
Section 2.3 (Evaluation Metrics and Forensic Benchmarks) says FaceForensics++ evaluates six detection methods on its dataset and Celeb-DF evaluates ten methods using publicly released pretrained models. This supports the citing sentence’s description of dataset-focused benchmarks evaluating detection methods. (LLM, conf=0.90)
- Supportedcites [57] [57] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. 2019. Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF International Conference on Computer Vision . IEEE, 1ś11. - source
Section 5 (Benchmark) describes a public benchmark built from the paper’s own videos and reports results for multiple detection methods. The Additional information section likewise says FaceForensics++ evaluates detection approaches on its generated dataset. This supports the claim as an example of such a benchmark, without establishing the broader trend across all benchmarks. (LLM, conf=0.92)
- Supportedcites [36] [36] Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. 2020. Celeb-df: A large-scale challenging dataset for deepfake forensics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . IEEE, 3207ś3216. - source
The Introduction (p. 1) presents Celeb-DF as a dataset for evaluating DeepFake detection methods. Section 4 (p. 6) says the paper evaluates multiple detection methods using Celeb-DF and other existing datasets. Celeb-DF is a central focus, though the evaluation is not limited to its own dataset. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the existing detection methods generally only evaluate their overall performance on publicly available datasets”
What the sources say
- Could not verifycites [57] [57] Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. 2019. Faceforensics++: Learning to detect manipulated facial images. In Proceedings of the IEEE/CVF International Conference on Computer Vision . IEEE, 1ś11. - sourceExperimental — accuracy not yet measured
Section 4.2.2 reports the authors' evaluation of detection methods on their dataset, and Section 5 describes their public benchmark. Neither passage says that existing methods generally evaluate only overall performance on publicly available datasets. Searches for prior evaluation practices and that exclusivity claim found no supporting passage in our copy. (LLM, conf=0.70)
- Partly supportedcites [36] [36] Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. 2020. Celeb-df: A large-scale challenging dataset for deepfake forensics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . IEEE, 3207ś3216. - sourceExperimental — accuracy not yet measured
Section 4 (p. 6) evaluates existing DeepFake detection methods across several datasets, and Section 4.2 (p. 6) measures their overall performance using frame-level AUC. These passages support evaluation on datasets, but do not establish that detection methods generally evaluate themselves only on publicly available datasets. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
81 more claims · 4 with their cited sources checked
Create a free account to see all 144 claims and to verify the 3 references still open.
- It may state two separate claims as one. 41
- Probably not a substantial claim. 25
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.1 Case Study 1: Facial Similarity · p. 18
“we choose two representative deepfake methods (FOMM and SimSwap) from the face swapping and reenactment methods, respectively.”
§ 4.1 Overall Experimental Seting · p. 9
“Face Swapping : FSGAN [49], FaceShifter [31], SimSwap [8]; Face Reenactment : FSGAN [49], ICface [69], FOMM [62], MRAA [63]”
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
FaceNet: A unified embedding for face recognition and clustering
Schroff, F., Kalenichenko, D., & Philbin, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298682 · Cited by 15,267
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602
Research works printed but not linked: 3
- Ivana Chingovska, André Anjos, and Sébastien Marcel. 2012. On the effectiveness of local binary patterns in face anti-spoofing. In Proceedings of the International Conference of the Biometrics Special Interest Group (BIOSIG’12). IEEE, Los Alamitos, CA, 1–7.
- Paul Ekman and Wallace V. Friesen. 1978. FacialActionCodingSystem:A Technique for the Measurement of FacialMovement. Consulting Psychologists Press Palo Alto CA.
- 10.5555/1324787 · doi:10.5555/1324787
Cited by
A Comprehensive Review of DeepFake Detection Using Advanced Machine Learning and Fusion Methods
Gupta, G., Raja, K., Gupta, M., Jan, T., Whiteside, S. T., & Prasad, M. (2023) · Electronics · doi:10.3390/electronics13010095 · Cited by 99
Video and Audio Deepfake Datasets and Open Issues in Deepfake Technology: Being Ahead of the Curve
Akhtar, Z., Pendyala, T. L., & Athmakuri, V. S. (2024) · Forensic Sciences · doi:10.3390/forensicsci4030021 · Cited by 60
Khimi, W., Albarqi, K., Saif, K., & Elhag, S. (2024) · International Journal of Computers and Informatics · doi:10.59992/ijci.2024.v3n8p3 · Cited by 11
Dong, Y., Wang, L., Li, Z., Li, H., Tang, P., Hu, C., & Guo, S. (2024) · ACM Transactions on Privacy and Security · doi:10.1145/3705611 · Cited by 14
“And AEs have been demonstrated the potential to pose significant security risks to deep learning models operating in various domains [12, 13, 23, 24, 33, 43, 52, 63, 65, 70, 77, 79].”
Self-citationAdaptive Network Intrusion Detection Using Reinforcement Learning with Proximal Policy Optimization
Suresh, A., & Jose, A. C. (2025) · ACM Transactions on Privacy and Security · doi:10.1145/3764586 · Cited by 19
“Stacked ensembles, in particular, enhance decision-making by using meta-learners [24], often MLP [66], to aggregate predictions from base models, creating a more accurate and adaptable framework.”
Metadata
- Authors
- Wang, L., Meng, X., Li, D., Zhang, X., Ji, S., & Guo, S.
- Affiliations
- Shandong University (CN)
- China Electronic Product Reliability and Environmental Test Institute (CN)
- Zhejiang University (CN)
- Venue
- ACM Transactions on Privacy and Security
- Published
- 2024-02-05
- Volume
- 27
- Issue
- 1
- Pages
- 1-34
- ISSN
- 2471-2566 (ISSN-L)
- 2471-2574
- OpenAlex
- W4389140876
- Cited by
- 11
- References
- 96
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 3
- FWCI
- 1.00
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Privacy and Security (publishedVersion)
- Funders
- National Natural Science Foundation of China (61902148)
- Shandong Provincial Natural Science Foundation (ZR2020MF055, ZR2021LZH007, ZR2020LZH002, ZR2020QF045)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- deepfake generation, evaluation platform, transfer attacks, video compression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: DEEPFAKER: A Unified Evaluation Platform for Facial Deepfake and Detection Models. https://app.cochat.ai/papers/10.1145/3634914
- The paper
- Wang, L., Meng, X., Li, D., Zhang, X., Ji, S., & Guo, S. (2024). DEEPFAKER: A Unified Evaluation Platform for Facial Deepfake and Detection Models. ACM Transactions on Privacy and Security. https://doi.org/10.1145/3634914
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.