Assessment framework for deepfake detection in real-world situations
Open accessThe DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index; 5 of the 9 sources its claims cite were checked for support.
Abstract
Abstract Detecting digital face manipulation in images and video has attracted extensive attention due to the potential risk to public trust. To counteract the malicious usage of such techniques, deep learning-based deepfake detection methods have been employed and have exhibited remarkable performance. However, the performance of such detectors is often assessed on related benchmarks that hardly reflect real-world situations. For example, the impact of various image and video processing operations and typical workflow distortions on detection accuracy has not been systematically measured. In this paper, a more reliable assessment framework is proposed to evaluate the performance of learning-based deepfake detectors in more realistic settings. To the best of our acknowledgment, it is the first systematic assessment approach for deepfake detectors that not only reports the general performance under real-world conditions but also quantitatively measures their robustness toward different processing operations. To demonstrate the effectiveness and usage of the framework, extensive experiments and detailed analysis of four popular deepfake detection methods are further presented in this paper. In addition, a stochastic degradation-based data augmentation method driven by realistic processing operations is designed, which significantly improves the robustness of deepfake detectors.
Overview
CoChat Summary
## TL;DR
The paper proposes a systematic assessment framework to measure deepfake detector robustness in realistic image and video conditions, rather than only on standard benchmark data. It also introduces stochastic degradation-based augmentation (SDAug), which the paper reports improves robustness against many real-world distortions.
## Key findings
- On realistic image deepfakes, even mild processing operations caused noticeable drops in detection accuracy; the paper says Capsule-Forensics and XceptionNet showed “severe performance deterioration” on modified data, and that UIA-VIT also degraded under processing operations.
- On realistic video deepfakes, Capsule-Forensics and XceptionNet performed poorly on pre-processed video deepfakes, while SBIs and UIA-VIT were more stable; however, none of the three methods could properly classify heavily compressed, low-resolution, or noisy video deepfakes.
- In cross-manipulation evaluation on Celeb-DFv1 and Celeb-DFv2 after training on FFpp, Capsule rose from AUC 43.36/54.39 with no augmentation to 74.84/71.86 with SDAug; XceptionNet rose from 39.35/50.00 to 80.67/73.88 with SDAug.
- The paper reports that SDAug improved robustness while maintaining high performance on original data, and that it outperformed standard Augmix and the non-stochastic degradation-based variant (DAug) in the reported experiments.
## Methods
- The paper proposes an assessment framework that trains deepfake detectors on their original datasets, then creates multiple distorted copies of the test set by applying one processing operation at a time at different severity levels; performance is measured with AUC from ROC curves.
- Data used: FaceForensics++ (FFpp; 1,000 pristine and 4,000 manipulated videos, with C23 and C40 compressed versions) and Celeb-DFv2 (590 pristine celebrity videos and 5,639 fake videos); evaluation focuses on unaltered and distorted variants.
- Models evaluated include Capsule-Forensics, XceptionNet, UIA-VIT, and SBIs; for augmentation experiments, the paper compares No Aug, Augmix, DAug, and SDAug.
## Limitations
- Stated by the paper: experiments are mainly limited to video deepfakes or standard-quality image deepfakes; HD single-image deepfakes, including those created by GANs and diffusion models, were not evaluated with the framework.
- Stated by the paper: the SDAug chain is hand-designed and its hyperparameters may not be optimal; the authors suggest AutoML-based parameter search.
- Stated by the paper: SDAug appears to help SBIs less than the other models because SBIs relies entirely on synthetic data and augmentation may corrupt its manually designed forgery traces.
- Evident from the text: several tables in the excerpt are partially garbled/truncated, so some detailed numeric results are not fully recoverable from the provided text.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 Valid · 10 Not valid · 3 Not checked · 73 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“On the other hand, the resizing operation, more specifically the down-sampling operation, can significantly reduce the performance of modern deep learningbased detectors [58, 59] due to a loss of discriminative information.”
What the sources say
- Partly supportedcites [58] T. Marciniak, A. Chmielewska, R. Weychan, M. Parzych, A. Dabrowski, Influence of low resolution of images on reliability of face detection and recognition. Multimed. Tools Appl. 74 (2013). https://doi.org/10.1007/s11042-013-1568-8 sourceExperimental — accuracy not yet measured
Section 5.2 (p. 9) reports that eightfold down-sampling reduced face-detection efficiency by about 10–20% for certain head positions. But the detector tested was Haar-like, not a modern deep-learning detector (section 5.1, p. 6), and the cited passages do not establish loss of discriminative information as the cause. (LLM, conf=0.88)
- Partly supportedcites [59] P. Li, L. Prieto, D. Mery, P.J. Flynn, On low-resolution face recognition in the wild: comparisons and new techniques. IEEE Trans. Inf. Forensics Secur. 14, 2000–2012 (2019) sourceExperimental — accuracy not yet measured
In “A. Super-resolution Techniques” (p. 4), the paper reports that face-matching performance decreases as downsampled resolution decreases. “A. Low-Resolution Face Recognition” (p. 2) links low-resolution images to a lack of information for robust feature representation. These passages support the general effect of information loss on recognition, but the paper does not establish the claimed performance reduction for deep learning-based detectors. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“although the SBIs method demonstrates a good generalization ability in cross-dataset experiments in their paper [21], our assessment framework shows that it is susceptible to some common perturbations in the real world, such as video compression, video noise, and low resolution.”
What the sources say
- Supportedcites [21] K. Shiohara, T. Yamasaki, Detecting deepfakes with self-blended images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18720–18729 (2022) source
The citation is attached to the claim that SBIs generalized well in the source paper’s cross-dataset experiments. Table 1 (Implementation Details, p. 4) reports that SBIs outperformed prior methods on four of five test datasets and ranked second on the fifth. The statement about susceptibility to compression, noise, and low resolution is explicitly attributed to the citing authors’ own assessment, not to this source. (LLM, conf=0.92)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“On the contrary, both tested detectors are more robust to test data compressed using hific codec than using JPEG operation or bmshj codec, even with extremely low bitrates.”
What the sources say
- Could not verifycites [61] F. Mentzer, G.D. Toderici, M. Tschannen, E. Agustsson, High-fidelity generative image compression. Adv. Neural. Inf. Process. Syst. 33, 11913–11924 (2020) sourceExperimental — accuracy not yet measured
Our copy of the HiFiC paper evaluates only perceptual and distortion quality: the user study and rate-distortion-perception curves (Sec. 5.1, Fig. 3; Metrics section; Appendix A.3, A.6, B.2) compare HiFiC against BPG, M&S Hyperprior and a no-GAN baseline. No passage addresses 'detectors' or their robustness to compressed test data, and no codec named 'bmshj' appears anywhere (searched: detector robustness/downstream classifier accuracy, JPEG vs. codec robustness comparisons, BMShJ, extremely low bitrates). The nearest text is the comparison to Agustsson et al. at 'extremely low' bitrates (A.3), which concerns reconstruction quality, not detector performance, so the claim cannot be checked against this source. (LLM, conf=0.70)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when the previous two methods are trained directly on compressed data, they maintain higher robustness toward multiple processing operations and even outperform the SBIs method, whose overall score even decreases by 0.66% instead.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, both models trained with compressed data ( FFpp-C23 , FFpp-Full ) show a higher AUC score on our realistic benchmark, but their performance on original unaltered data decreases by 0.5-1%.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
50 more claims
Create a free account to see all 88 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 13
- The quote only partly states this claim. 2
- Probably not a substantial claim. 20
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 2×
“In [47], Hendrycks et al. presented a corrupted version of ImageNet [48] to benchmark the robustness of image recognition models against common image manipulations.”
BackgroundMethodXception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · cited 2×
“The two detectors present exceptional performance on unaltered FFpp and CelebDFv2 testing data as expected, but then show severe performance deterioration on all kinds of modified data from the assessment framework, which indicates a lack of robustness.”
BackgroundEfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714 · cited 1×
“For example, the recent advancement of generative adversarial networks (GANs) [1, 2, 3] has made it possible to generate realistic forged contents that are difficult for humans to distinguish from their authentic counterparts.”
BackgroundBeyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2662206 · Cited by 8,983 · cited 1×
“The assessment framework takes this situation into consideration and measures the performance of a detector under different brightness conditions.”
BackgroundAnalyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00813 · Cited by 6,332 · cited 1×
“For example, the recent advancement of generative adversarial networks (GANs) [1, 2, 3] has made it possible to generate realistic forged contents that are difficult for humans to distinguish from their authentic counterparts.”
BackgroundAnalyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2019) · arXiv · doi:10.48550/arxiv.1912.04958 · Cited by 6,332
Progressive Growing of GANs for Improved Quality, Stability, and Variation
Karras, T., Aila, T., Laine, S., & Lehtinen, J. (2017) · arXiv · doi:10.48550/arxiv.1710.10196 · Cited by 4,727
Dynamic Routing Between Capsules
Sabour, S., Frosst, N., & Hinton, G. E. (2017) · arXiv · doi:10.48550/arxiv.1710.09829 · Cited by 4,167
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 8×
“For both methods, 100 frames are randomly sampled from each video for training purposes and 32 frames are extracted for validation and testing.”
BackgroundMethodInfluential
Research works printed but not linked: 2
- B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, C.C. Ferrer, The deepfake detection challenge dataset 2006, 07397 (2020)
- S. Agarwal, H. Farid, Y. Gu, M. He, K. Nagano, H. Li, Protecting world leaders against deep fakes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019)
Cited by
Khan, A. A., Laghari, A. A., Inam, S. A., Ullah, S., Shahzad, M., & Syed, D. (2025) · Discover Computing · doi:10.1007/s10791-025-09550-0 · Cited by 99
Video and Audio Deepfake Datasets and Open Issues in Deepfake Technology: Being Ahead of the Curve
Akhtar, Z., Pendyala, T. L., & Athmakuri, V. S. (2024) · Forensic Sciences · doi:10.3390/forensicsci4030021 · Cited by 60
Parents and Children: Distinguishing Multimodal Deepfakes from Natural Images
Amoroso, R., Morelli, D., Cornia, M., Baraldi, L., Bimbo, A. D., & Cucchiara, R. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3665497 · Cited by 63
Deep Fake Image Detection using Xception Architecture
Joshi, P., & Nivethitha, V. (2024) · International Conference on Recent Trends in Computer Science and Technology (ICRTCST) · doi:10.1109/icrtcst61793.2024.10578398 · Cited by 37
Continuous fake media detection: Adapting deepfake detectors to new generative techniques
Tassone, F., Maiano, L., & Amerini, I. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104143 · Cited by 27
Metadata
- Authors
- Lu, Y., & Ebrahimi, T.
- Affiliations
- École Polytechnique Fédérale de Lausanne (CH)
- Corresponding author
- Lu, Yuhang
- Venue
- EURASIP Journal on Image and Video Processing
- Published
- 2024-02-13
- Volume
- 2024
- Issue
- 1
- ISSN
- 1687-5176 (ISSN-L)
- 1687-5281
- OpenAlex
- W4391778999
- Cited by
- 60
- References
- 63
- Citations by year
- 2026: 18 · 2025: 25 · 2024: 15
- FWCI
- 8.08
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- EURASIP Journal on Image and Video Processing (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Swiss National Foundation for Scientific Research (20CH21 195532)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
- Keywords
- deepfake detection, real-world robustness, evaluation framework, data augmentation, stochastic deterioration, benchmark limitations
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessment framework for deepfake detection in real-world situations. https://app.cochat.ai/papers/10.1186/s13640-024-00621-8
- The paper
- Lu, Y., & Ebrahimi, T. (2024). Assessment framework for deepfake detection in real-world situations. EURASIP Journal on Image and Video Processing. https://doi.org/10.1186/s13640-024-00621-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.