CoChat
Sign Up
PreprintCited by 5Digital Media Forensic DetectionMisinformation and Its ImpactsGenerative Adversarial Networks and Image Synthesis

Towards Benchmarking and Evaluating Deepfake Detection

Open access
Lin, Chenhao, Deng, Jingyi, Hu, Pengbin, Shen, Chao, Wang, Qian, Li, Qi·2022·arXiv
DOI 10.48550/arxiv.2203.02115arXiv 2203.02115
Open this paper in CoChat — free accountOpen PDF

Abstract

Deepfake detection automatically recognizes the manipulated medias through the analysis of the difference between manipulated and non-altered videos. It is natural to ask which are the top performers among the existing deepfake detection approaches to identify promising research directions and provide practical guidance. Unfortunately, it's difficult to conduct a sound benchmarking comparison of existing detection approaches using the results in the literature because evaluation conditions are inconsistent across studies. Our objective is to establish a comprehensive and consistent benchmark, to develop a repeatable evaluation procedure, and to measure the performance of a range of detection approaches so that the results can be compared soundly. A challenging dataset consisting of the manipulated samples generated by more than 13 different methods has been collected, and 11 popular detection approaches (9 algorithms) from the existing literature have been implemented and evaluated with 6 fair-minded and practical evaluation metrics. Finally, 92 models have been trained and 644 experiments have been performed for the evaluation. The results along with the shared data and evaluation methodology constitute a benchmark for comparing deepfake detection approaches and measuring progress.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2203.02115

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Lin et al. conduct a comprehensive benchmarking comparison of existing detection approaches, but their experiments are based on collected public datasets and do not support dynamic generation of different types of deepfake data for fine-grained evaluation.

    “However, their extensive experiments are also based on the collected public datasets for overall evaluation, and do not support the dynamic generation of diferent types of deepfake data for the ine-grained evaluation.”

    From DEEPFAKER: A Unified Evaluation Platform for Facial Deepfake and Detection Models (2024) · cites this paper as [37]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Existing benchmarks [25] and [18] lack a modular, extensible, and comprehensive codebase that includes data preprocessing, unified settings, training modules, evaluations, and a series of analytical tools.

    “However, these two benchmarks still lack a modular, extensible, and comprehensive codebase that includes data preprocessing, unified settings, training modules, evaluations, and a series of analytical tools.”

    From DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection (2023) · cites this paper as [25]

    Section 2.3, ‘Evaluation Metrics and Forensic Benchmarks,’ says most existing benchmarks do not re-implement methods with the same training data, creating evaluation bias. The Introduction likewise identifies inconsistent training data as a problem. Our copy does not establish that the two benchmarks lack a modular, extensible codebase with all the listed components. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Current benchmarks primarily focus on their own datasets and evaluate various detection methods on these public datasets.

    “current benchmarks [37], such as FaceForensics++ [57] and Celeb-DF [36], primarily focus on their own datasets and evaluate various detection methods on these public datasets.”

    From DEEPFAKER: A Unified Evaluation Platform for Facial Deepfake and Detection Models (2024) · cites this paper as [37]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 5

Research works printed but not linked: 32

  1. Darius Afchar, Vincent Nozick, Junichi Yamagishi, and Isao Echizen. Mesonet: a compact facial video forgery detection network. In 2018 IEEE International Workshop on Informa- tion Forensics and Security (WIFS), pages 1-7. IEEE, 2018. 4, 5, 10, 12
  2. Jeonghun Baek, Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Seong Joon Oh, and Hwal- suk Lee. What is wrong with scene text recognition model comparisons? dataset and model analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 4715-4723, 2019. 2
  3. G. Bradski. The OpenCV Library. Dr. Dobb's Journal of Software Tools, 2000. 11
  4. Lorenzo Brigato, Björn Barz, Luca Iocchi, and Joachim Den- zler. Tune it or don't use it: Benchmarking data-efficient image classification. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 1071-1080, 2021. 2
  5. Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola. What makes fake images detectable? understanding prop- erties that generalize. In European Conference on Computer Vision, pages 103-120. Springer, 2020. 2, 4, 5, 10
  6. Haonan Chen, Guosheng Hu, Zhen Lei, Yaowu Chen, Neil M Robertson, and Stan Z Li. Attention-based two- stream convolutional networks for face spoofing detection. IEEE Transactions on Information Forensics and Security, 15:578-593, 2019. 4
  7. Franc ¸ois Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 1251-1258, 2017. 11
  8. Davide Cozzolino, Justus Thies, Andreas Rössler, Christian Riess, Matthias Nießner, and Luisa Verdoliva. Forensictrans- fer: Weakly-supervised domain adaptation for forgery detec- tion. arXiv preprint arXiv:1812.02510, 2018. 4 · arXiv
  9. Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K Jain. On the detection of digital face manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition, pages 5781-5790, 2020. 1, 5, 11
  10. David Güera and Edward J Delp. Deepfake video detection using recurrent neural networks. In 2018 15th IEEE Inter- national Conference on Advanced Video and Signal Based Surveillance (AVSS), pages 1-6. IEEE, 2018. 4, 5, 11, 12
  11. Dong Huang and Fernando De La Torre. Facial action trans- fer with personalized bilinear regression. In European Con- ference on Computer Vision, pages 144-158. Springer, 2012.
  12. Liming Jiang, Ren Li, Wayne Wu, Chen Qian, and Chen Change Loy. Deeperforensics-1.0: A large-scale dataset for real-world face forgery detection. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2886-2895. IEEE, 2020. 2, 4
  13. Davis E King. Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research, 10:1755-1758, 2009. 5, 11
  14. Pavel Korshunov and Sébastien Marcel. Deepfakes: a new threat to face recognition? assessment and detection. arXiv preprint arXiv:1812.08685, 2018. 2 · arXiv
  15. Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen. Faceshifter: Towards high fidelity and occlusion aware face swapping. arXiv preprint arXiv:1912.13457, 2019. 2 · arXiv
  16. Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and Baining Guo. Face x-ray for more gen- eral face forgery detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5001-5010, 2020. 2, 3, 4, 5, 10, 12
  17. Yuezun Li, Ming-Ching Chang, and Siwei Lyu. In ictu oculi: Exposing ai generated fake face videos by detecting eye blinking. arXiv preprint arXiv:1806.02877, 2018. 2 · arXiv
  18. Yuezun Li and Siwei Lyu. Exposing deepfake videos by detecting face warping artifacts. arXiv preprint arXiv:1811.00656, 2018. 3, 4, 5, 10 · arXiv
  19. Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. Celeb-df: A large-scale challenging dataset for deep- fake forensics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3207- 3216, 2020. 3
  20. Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015. 5
  21. Daniel Mas Montserrat, Hanxiang Hao, Sri Kalyan Yarla- gadda, Sriram Baireddy, Ruiting Shao, János Horváth, Emily Bartusiak, Justin Yang, David Güera, Fengqing Zhu, et al. Deepfakes detection with automatic face weighting. arXiv preprint arXiv:2004.12027, 2020. 4 · arXiv
  22. Yuval Nirkin, Yosi Keller, and Tal Hassner. Fsgan: Subject agnostic face swapping and reenactment. In Proceedings of the IEEE international conference on computer vision, pages 7184-7193, 2019. 2, 4
  23. Ivan Perov, Daiheng Gao, Nikolay Chervoniy, Kunlin Liu, Sugasa Marangonda, Chris Umé, Mr Dpfks, Carl Shift Facenheim, Luis RP, Jian Jiang, et al. Deepfacelab: A sim- ple, flexible and extensible face swapping framework. arXiv preprint arXiv:2005.05535, 2020. 2 · arXiv
  24. Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, and Jing Shao. Thinking in frequency: Face forgery detection by mining frequency-aware clues. In European Conference on Computer Vision, pages 86-103. Springer, 2020. 3
  25. Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Chris- tian Riess, Justus Thies, and Matthias Nießner. Faceforen- sics++: Learning to detect manipulated facial images. In Proceedings of the IEEE International Conference on Com- puter Vision, pages 1-11, 2019. 2, 4, 5, 11, 12
  26. Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAl- mageed, Iacopo Masi, and Prem Natarajan. Recurrent convo- lutional strategies for face manipulation detection in videos. Interfaces (GUI), 3(1), 2019. 4
  27. Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose esti- mation. In CVPR, 2019. 10
  28. Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114. PMLR, 2019. 11
  29. Xin Yang, Yuezun Li, and Siwei Lyu. Exposing deep fakes using inconsistent head poses. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8261-8265. IEEE, 2019. 3, 4, 5, 10
  30. Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi-attentional deep- fake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2185- 2194, 2021. 1, 5, 11, 12
  31. Peng Zhou, Xintong Han, Vlad I Morariu, and Larry S Davis. Two-stream neural networks for tampered face detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1831-1839. IEEE, 2017. 4
  32. Yuhao Zhu, Qi Li, Jian Wang, Cheng-Zhong Xu, and Zhenan Sun. One shot face swapping on megapixels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 4834-4844, 2021.

Authors
Lin, C., Deng, J., Hu, P., Shen, C., Wang, Q., & Li, Q.
Affiliations
  • Xi'an Jiaotong University (CN)
  • Wuhan University (CN)
  • Tsinghua University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2203.02115
arXiv
2203.02115
OpenAlex
W4221165930
Cited by
5
References
0
Citations by year
2026: 1 · 2024: 2 · 2023: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
deepfake detection, benchmarking, manipulated video, evaluation metrics, video manipulation, detection performance comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review