The DOI matches its Crossref record; 130 of the 140 references it cites were found in a registry or an index; 35 of the 90 sources its claims cite were checked for support.
Deepfake: definitions, performance metrics and standards, datasets, and a meta-review
Open accessAbstract
Meta-review covering definitions, performance metrics, standards, datasets, and challenges across deepfake research.
Overview
CoChat Summary
## TL;DR
This paper is a comprehensive review of deepfake definitions, performance metrics and standards, datasets, and a meta-review of 15 deepfake survey papers published since 2020. It argues that deepfake research needs broader, more consistent definitions, better standardised evaluation, and more complete datasets.
## Key findings
- The authors identify a lack of consensus on what “deepfake” means and propose a more inclusive definition that covers manipulated or synthetic media created with sophisticated automated methods, often AI-based.
- They survey commonly used evaluation metrics for deepfake generation and detection, including accuracy, AUC, EER, log loss, precision, recall, F1-score, MOS, and multimodal alignment metrics; they also discuss standards such as ISO/IEC 24029-1:2021 and biometric PAD standards like ISO/IEC 30107-1:2016 and ISO/IEC 30107-3:2017.
- The dataset review covers image, video, audio/speech, text, and hybrid datasets, and the paper states that it is “the most comprehensive review of deepfake-related datasets so far.”
- In the meta-review, the authors found that only 2 of 15 survey papers explicitly covered performance metrics, none covered text datasets, and only 3 mentioned audio datasets; they also note that 15 survey papers were selected from 117 survey papers identified in Scopus searches.
## Methods
- Systematic literature search on Scopus using the query: `(deepfake* OR deep-fake* OR “deep fake*”) AND (review OR survey OR overview OR systemati* OR SoK)`.
- From 117 English survey papers published between 2020 and 2024, the authors selected 15 for the meta-review based on having a substantial comparative angle and coverage of items such as performance comparison, datasets, tools, challenges, competitions, and metrics.
- They compiled datasets from selected survey papers, deepfake-related challenges/competitions/benchmarks, GitHub collections, and the co-authors’ personal collections; relevant standards were identified from surveyed papers, co-author knowledge, and Google Web searches.
## Limitations
- The paper states that its scope is intentionally narrower for some parts: for the meta-review it includes only survey papers whose metadata explicitly refer to “deepfakes,” and for datasets it includes those with an explicit mention of “deepfake” or a comparable term.
- The paper notes that many deepfake-related surveys have limited coverage due to page constraints, and that performance comparisons are difficult because different papers use different metrics and inconsistent datasets.
- Evident from the text: some dataset and standard coverage depends on the authors’ selection criteria and available online sources, so the review may not be exhaustive beyond the stated scope.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
24 Valid · 12 Not valid · 51 Not checked · 61 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“only three of them (Heidari et al., 2023; Khanjani et al., 2023; Masood et al., 2023) mentioned audio datasets.”
§ Deepfake: definitions, performance metrics and standards, datasets, and a meta-review > 5 A meta-review of deepfake-related surveys > 5.3 Datasets
What the sources say
- Supportedcites Heidari et al., 2023 source
The paper identifies audio datasets, including the ASVSpoof 2019 logical access dataset (Section 5.4) and other audio-related datasets in its comparisons (Section 6). This supports citing Heidari et al. as one of the survey papers that mentioned audio datasets; the count of three among 15 is the citing authors’ comparison, not a finding attributed to this individual source. (LLM, conf=0.90)
- Supportedcites Khanjani et al., 2023 source
The citation identifies Khanjani et al. as a survey that mentions audio datasets. Section 4, “Audio deepfake datasets,” discusses datasets including FoR and WaveFake. The count of three among 15 surveys is the citing authors’ comparison, not a claim attributed to this paper. (LLM, conf=0.94)
- Not checked — no full text of the sourcecites Masood et al., 2023 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“ASVspoof 2019 challenge database (Wang et al., 2020): This dataset is based on the Voice Cloning Toolkit (VCTK) corpus,44 a multi-speaker English speech database captured from 107 speakers (46 males and 61 females).”
§ Deepfake: definitions, performance metrics and standards, datasets, and a meta-review > 4 Deepfake-related datasets > 4.3 Deepfake audio/speech datasets
What the sources say
- Supportedcites Wang et al., 2020 source
Section 2.1 (p. 3) states that ASVspoof 2019 is based on VCTK utterances from 107 speakers. Section 2.3 (p. 5) lists logical-access training, development, and evaluation counts that sum to 12,483 bona fide and 108,978 spoofed utterances. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Some survey papers include datasets that are likely deepfakes, e.g., Verdoliva (2020) covered many general fake image datasets where the manipulated images were not generated by deep learning or even AI-based methods”
§ Deepfake: definitions, performance metrics and standards, datasets, and a meta-review > 5 A meta-review of deepfake-related surveys > 5.3 Datasets
What the sources say
- Supportedcites Verdoliva, 2020 source
The Images section (pp. 12–13) surveys multiple general image-forgery datasets, including splicing and copy-move datasets, rather than limiting its coverage to deep-learning-generated images. The Introduction (p. 1) distinguishes deepfakes as images and videos manipulated using advanced deep-learning tools. The sentence's phrase “likely deepfakes” is at odds with its own example, but the substantive claim about Verdoliva's dataset coverage is supported. (LLM, conf=0.88)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Surprisingly, only two of the 15 surveys (Rana et al., 2022; Heidari et al., 2023) have covered performance metrics explicitly.”
§ Deepfake: definitions, performance metrics and standards, datasets, and a meta-review > 5 A meta-review of deepfake-related surveys > 5.2 Performance metrics
What the sources say
- Supportedcites Rana et al., 2022 source
Section SRQ-2.4 explicitly describes performance metrics for deepfake detection, including accuracy, ROC, AUC, recall, and precision. This confirms Rana et al. as one of the surveys that covers performance metrics; the comparison across 15 surveys is the citing authors’ assessment, not a finding attributed to Rana et al. (LLM, conf=0.93)
- Partly supportedcites Heidari et al., 2023 sourceExperimental — accuracy not yet measured
Section 6.2, “Criteria of DL/ML methods” (page unavailable), explicitly discusses performance metrics, so the cited review supports its own inclusion among surveys covering them. Section 3, “Relevant Reviews” (page unavailable), also describes Rana et al. as comparing detection capabilities. Neither passage establishes that exactly two of 15 surveys covered performance metrics explicitly. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Supported
None of the sources cited for this claim have been checked yet.
In the paper
“Interestingly, we can see a huge difference between the realness levels of different datasets.”
§ Deepfake: definitions, performance metrics and standards, datasets, and a meta-review > 4 Deepfake-related datasets > 4.7 Subjective quality of deepfakes in different databases
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
18 more claims
Create a free account to see all 148 claims and to verify the 8 references still open.
- Probably not a substantial claim. 105
- It may state two separate claims as one. 66
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The term deepfake has evolved from describing face-swapped videos to serving as an umbrella for AI-generated or AI-manipulated media, including images, video, audio voice cloning, and multimodal composites [1].”
From A Review of Tools and Technologies to Combat Deepfakes (2026) · cites this paper as [1]
Experimental — accuracy not yet measured
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Proceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691
Proceedings European Conference on Computer Vision 2002
Heyden, A., Sparr, G., Nielsen, M., & Johansen, P. (2002) · Lund University Publications (Lund University) · Cited by 5,153
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911 · cited 3×
“Compared to the 2016 challenge, more participating teams used DL-related techniques (e.g., WaveNet, LSTM, DNN, CycleGAN, DRM – deep relational models, and ARBM – adaptive restricted Boltzmann machines) in their VC systems.”
MethodChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 1×
“FaceForensics++ (FF++) (Rössler et al., 2019): This dataset contains 5,000 face videos with over 1.8 million manipulated frames, including 1,000 real videos (with 509,914 frames) downloaded from YouTube, and 4,000 fake videos created using four face manipulation methods (Deepfakes, Face2Face,…”
MethodInfluentialProceedings of IEEE International Conference on Computer Vision
(1995) · doi:10.1109/iccv.1995 · Cited by 2,280
Research works printed but not linked: 2
- Delgado, ASVspoof 2021 challenge
- Delgado, ASVspoof 2021 challenge
Cited by
GAN-Generated Faces Detection: A Survey and New Perspectives
Wang, X., Guo, H., Hu, S., Chang, M.-C., & Lyu, S. (2023) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/faia230558 · Cited by 98
“…conducted with a different perspective compared to existing survey papers, for example, the survey [51] mostly focus on the audio deepfakes that are overlooked in most of the previous surveys, [68] focus on the audio-visual DeepFakes generation and detection, and the benchmarks [2, 59, 86, 55].”
BackgroundGenerative AI and Deepfake Detection in Biometric Systems
Khan, F. A., & Khan, M. K. (2025) · Cognitive Computation · doi:10.1007/s12559-025-10469-3 · Cited by 18
“These AI-based systems mimic the richness of the tone and voice of the source speakers such that, over time, voice biometrics becomes unable to identify the true from the artificial voices [61, 66].”
Deepfake defense: Combining spatial and temporal cues with CNN–BiLSTM–transformer architecture
Yadav, S., & Mangalampalli, S. S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0334980 · Cited by 16
MDCF-Net: Multi-Scale Dual-Branch Network for Compressed Face Forgery Detection
Zhou, J., Zhao, X., Xu, Q., Zhang, P., & Zhou, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3390217 · Cited by 12
A comprehensive survey with critical analysis for deepfake speech detection
Pham, L. D., Lam, P., Tran, D., Tang, H., Nguyen, T., Schindler, A., Skopik, F., Polonsky, A., & Vu, H. C. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100757 · Cited by 29
Metadata
- Authors
- Altuncu, E., Franqueira, V. N. L., & Li, S.
- Affiliations
- University of Kent (GB)
- Corresponding author
- Li, Shujun
- Venue
- Frontiers in Big Data
- Publisher
- Frontiers Media SA
- Published
- 2024-09-04
- Volume
- 7
- Pages
- 1400024-1400024
- ISSN
- 2624-909X (ISSN-L)
- PMC
- PMC11408348
- OpenAlex
- W4402252380
- Cited by
- 134
- References
- 140
- Citations by year
- 2026: 64 · 2025: 46 · 2024: 17
- FWCI
- 3.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Big Data (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Kent Academic Repository (University of Kent) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- Kent Academic Repository (University of Kent) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Defence Science and Technology Laboratory
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- deepfakes, synthetic media, media manipulation, performance indicators, evaluation standards, deepfake datasets, deepfake detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deepfake: definitions, performance metrics and standards, datasets, and a meta-review. https://app.cochat.ai/papers/10.3389/fdata.2024.1400024
- The paper
- Altuncu, E., Franqueira, V. N. L., & Li, S. (2024). Deepfake: definitions, performance metrics and standards, datasets, and a meta-review. Frontiers in Big Data. https://doi.org/10.3389/fdata.2024.1400024
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
