CoChat

Assessment framework for deepfake detection in real-world situations

Open access
Lu, Yuhang, Ebrahimi, Touradj · 2024 · EURASIP Journal on Image and Video Processing

The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index; 5 of the 9 sources its claims cite were checked for support.

Abstract

Abstract Detecting digital face manipulation in images and video has attracted extensive attention due to the potential risk to public trust. To counteract the malicious usage of such techniques, deep learning-based deepfake detection methods have been employed and have exhibited remarkable performance. However, the performance of such detectors is often assessed on related benchmarks that hardly reflect real-world situations. For example, the impact of various image and video processing operations and typical workflow distortions on detection accuracy has not been systematically measured. In this paper, a more reliable assessment framework is proposed to evaluate the performance of learning-based deepfake detectors in more realistic settings. To the best of our acknowledgment, it is the first systematic assessment approach for deepfake detectors that not only reports the general performance under real-world conditions but also quantitatively measures their robustness toward different processing operations. To demonstrate the effectiveness and usage of the framework, extensive experiments and detailed analysis of four popular deepfake detection methods are further presented in this paper. In addition, a stochastic degradation-based data augmentation method driven by realistic processing operations is designed, which significantly improves the robustness of deepfake detectors.

Overview

CoChat Summary

## TL;DR

The paper proposes a systematic assessment framework to measure deepfake detector robustness in realistic image and video conditions, rather than only on standard benchmark data. It also introduces stochastic degradation-based augmentation (SDAug), which the paper reports improves robustness against many real-world distortions.

## Key findings

  • On realistic image deepfakes, even mild processing operations caused noticeable drops in detection accuracy; the paper says Capsule-Forensics and XceptionNet showed “severe performance deterioration” on modified data, and that UIA-VIT also degraded under processing operations.
  • On realistic video deepfakes, Capsule-Forensics and XceptionNet performed poorly on pre-processed video deepfakes, while SBIs and UIA-VIT were more stable; however, none of the three methods could properly classify heavily compressed, low-resolution, or noisy video deepfakes.
  • In cross-manipulation evaluation on Celeb-DFv1 and Celeb-DFv2 after training on FFpp, Capsule rose from AUC 43.36/54.39 with no augmentation to 74.84/71.86 with SDAug; XceptionNet rose from 39.35/50.00 to 80.67/73.88 with SDAug.
  • The paper reports that SDAug improved robustness while maintaining high performance on original data, and that it outperformed standard Augmix and the non-stochastic degradation-based variant (DAug) in the reported experiments.

## Methods

  • The paper proposes an assessment framework that trains deepfake detectors on their original datasets, then creates multiple distorted copies of the test set by applying one processing operation at a time at different severity levels; performance is measured with AUC from ROC curves.
  • Data used: FaceForensics++ (FFpp; 1,000 pristine and 4,000 manipulated videos, with C23 and C40 compressed versions) and Celeb-DFv2 (590 pristine celebrity videos and 5,639 fake videos); evaluation focuses on unaltered and distorted variants.
  • Models evaluated include Capsule-Forensics, XceptionNet, UIA-VIT, and SBIs; for augmentation experiments, the paper compares No Aug, Augmix, DAug, and SDAug.

## Limitations

  • Stated by the paper: experiments are mainly limited to video deepfakes or standard-quality image deepfakes; HD single-image deepfakes, including those created by GANs and diffusion models, were not evaluated with the framework.
  • Stated by the paper: the SDAug chain is hand-designed and its hyperparameters may not be optimal; the authors suggest AutoML-based parameter search.
  • Stated by the paper: SDAug appears to help SBIs less than the other models because SBIs relies entirely on synthetic data and augmentation may corrupt its manually designed forgery traces.
  • Evident from the text: several tables in the excerpt are partially garbled/truncated, so some detailed numeric results are not fully recoverable from the provided text.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13640-024-00621-8

Cites

73 of 73 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

2 Valid · 10 Not valid · 3 Not checked · 73 Own results and interpretations

50 more claims

Create a free account to see all 88 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 13
  • The quote only partly states this claim. 2
  • Probably not a substantial claim. 20

Create a free account to see these claims.

Citations

Cites

Show all 73 references

Research works printed but not linked: 2

  1. B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, C.C. Ferrer, The deepfake detection challenge dataset 2006, 07397 (2020)
  2. S. Agarwal, H. Farid, Y. Gu, M. He, K. Nagano, H. Li, Protecting world leaders against deep fakes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)

Cited by

See all 60

Metadata

Authors
Lu, Y., & Ebrahimi, T.
Affiliations
École Polytechnique Fédérale de Lausanne (CH)
Corresponding author
Lu, Yuhang
Venue
EURASIP Journal on Image and Video Processing
Published
2024-02-13
Volume
2024
Issue
1
ISSN
  • 1687-5176 (ISSN-L)
  • 1687-5281
OpenAlex
W4391778999
Cited by
60
References
63
Citations by year
2026: 18 · 2025: 25 · 2024: 15
FWCI
8.08
Type
journal article
Language
en
Open access
Licence
Locations
  • EURASIP Journal on Image and Video Processing (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
Swiss National Foundation for Scientific Research (20CH21 195532)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
Keywords
deepfake detection, real-world robustness, evaluation framework, data augmentation, stochastic deterioration, benchmark limitations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Assessment framework for deepfake detection in real-world situations. https://app.cochat.ai/papers/10.1186/s13640-024-00621-8
The paper
Lu, Y., & Ebrahimi, T. (2024). Assessment framework for deepfake detection in real-world situations. EURASIP Journal on Image and Video Processing. https://doi.org/10.1186/s13640-024-00621-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.