Deepfake detection: humans vs. machines
Open accessAbstract
Deepfake videos, where a person's face is automatically swapped with a face of someone else, are becoming easier to generate with more realistic results. In response to the threat such manipulations can pose to our trust in video evidence, several large datasets of deepfake videos and many methods to detect them were proposed recently. However, it is still unclear how realistic deepfake videos are for an average person and whether the algorithms are significantly better than humans at detecting them. In this paper, we present a subjective study conducted in a crowdsourcing-like scenario, which systematically evaluates how hard it is for humans to see if the video is deepfake or not. For the evaluation, we used 120 different videos (60 deepfakes and 60 originals) manually pre-selected from the Facebook deepfake database, which was provided in the Kaggle's Deepfake Detection Challenge 2020. For each video, a simple question: "Is face of the person in the video real of fake?" was answered on average by 19 naïve subjects. The results of the subjective evaluation were compared with the performance of two different state of the art deepfake detection methods, based on Xception and EfficientNets (B4 variant) neural networks, which were pre-trained on two other large public databases: the Google's subset from FaceForensics++ and the recent Celeb-DF dataset. The evaluation demonstrates that while the human perception is very different from the perception of a machine, both successfully but in different ways are fooled by deepfakes. Specifically, algorithms struggle to detect those deepfake videos, which human subjects found to be very easy to spot.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Korshonuv and Marcel found participants outperformed state-of-the-art machine learning detection models, achieving average AUC scores of 87.47% compared to EfficientNet-trained-on-Google at AUC of 73.97%.
“Korshonuv and Marcel [36] found that participants outperformed state-of-the-art machine learning detection models, achieving average AUC scores of 87.47% compared to E ffi cientNet-trained-on-Google at AUC of 73.97%.”
From Human Performance in Deepfake Detection: A Systematic Review (2025) · cites this paper as [36]
In IV. Evaluation of Algorithms (p. 5), the paper says human subjects were more accurate than the evaluated models on the 120 subjective-test videos and gives the highest AUC as 87.47%. It does not call 87.47% an average, and our copy does not provide the claimed 73.97% AUC for EfficientNet trained on Google; Table I (p. 4) reports 99.99% on a different test set. (LLM, conf=0.87)
Experimental — accuracy not yet measuredHigher-quality deepfakes are associated with fewer visual artifacts, making them more difficult to detect.
“Higher-quality deepfakes are associated with fewer visual artifacts, making them more di ffi cult to detect [20, 35, 36, 42].”
From Human Performance in Deepfake Detection: A Systematic Review (2025) · cites this paper as [36]
The Introduction (p. 2) groups deepfakes by how easy their visual artifacts are to spot, and the Conclusion (p. 5) reports that people were confused by good-quality deepfakes in 75.5% of cases. This supports greater difficulty for human viewers, but not for detection generally: Evaluation of Algorithms (p. 5) says algorithm accuracy had no correlation with visual appearance. (LLM, conf=0.87)
Experimental — accuracy not yet measuredKorshonuv et al. noted that machine learning algorithms struggled most with deepfake videos that were easily discernible to human participants.
“Korshonuv et al. [36] noted that machine learning algorithms struggled the most with deepfake videos that were easily discernible to human participants.”
From Human Performance in Deepfake Detection: A Systematic Review (2025) · cites this paper as [36]
Lower-quality deepfakes are generally easier to detect.
“Second, while lower-quality deepfakes are generally easier to detect [20, 35, 36, 42], imperfections in authentic videos (e.g., lighting inconsistencies or unusual framing) can lead viewers to miscategorize genuine content as manipulated [2, 22, 27, 35, 37].”
From Human Performance in Deepfake Detection: A Systematic Review (2025) · cites this paper as [36]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 17 · Cited by 47
Research works printed but not linked: 15
- [1][1] P. Korshunov and S. Marcel, 'Vulnerability assessment and detection of Deepfake videos,' in International Conference on Biometrics (ICB 2019) , Crete, Greece, Jun. 2019. -
- [2][2] A. R¨ ossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, 'Faceforensics: A large-scale video dataset for forgery detection in human faces,' arXiv.org , 2018. [Online]. Available: http://arxiv.org/abs/1803.09179 - · arXiv
- [3][3] --, 'FaceForensics++: Learning to detect manipulated facial images,' in International Conference on Computer Vision (ICCV) , 2019. -
- [5][5] H. Nguyen, J. Yamagishi, and I. Echizen, 'Capsule-forensics: using capsule networks to detect forged images and videos,' in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2019, pp. 2307-2311. -
- [6][6] S.-Y. Wang, O. Wang, R. Zhang, A. Owens, and A. A. Efros, 'Cnngenerated images are surprisingly easy to spot...for now,' in CVPR , 2020. -
- [8][8] Y. Li, M. Chang, and S. Lyu, 'In ictu oculi: Exposing ai created fake videos by detecting eye blinking,' in IEEE International Workshop on Information Forensics and Security (WIFS) , 2018, pp. 1-7. -
- [9][9] X. Yang, Y. Li, and S. Lyu, 'Exposing deep fakes using inconsistent head pose,' in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 2019, pp. 8261-8265. -
- [10][10] S. Agarwal, T. El-Gaaly, H. Farid, and S. Lim, 'Detecting deep-fake videos from appearance and behavior,' arXiv preprint , 2020. [Online]. Available: https://arxiv.org/abs/2004.14491 - · arXiv
- [12][12] A. Agarwal, R. Singh, M. Vatsa, and A. Noore, 'Swapped! digital face presentation attack detection via weighted local magnitude pattern,' in IEEE International Joint Conference on Biometrics (IJCB) , Oct 2017, pp. 659-665. -
- [14][14] C. Keimel, J. Habigt, C. Horch, and K. Dieopold, 'Qualitycrowd -a framework for crowd-based quality evaluation,' in Picture Coding Symposium (PCS) , May 2012. -
- [15][15] F. Chollet, 'Xception: Deep learning with depthwise separable convolutions,' in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017, pp. 1800-1807. -
- [16][16] T. Hossfeld, C. Keimel, M. Hirth, B. Gardlo, J. Habigt, K. Diepold, and P. Tran-Gia, 'Best practices for qoe crowdtesting: Qoe assessment with crowdsourcing,' IEEE Transactions on Multimedia , vol. 16, no. 2, pp. 541-558, 2014. -
- [17][17] P. Korshunov, H. Nemoto, A. Skodras, and T. Ebrahimi, 'Crowdsourcing-based evaluation of privacy in HDR images,' in Optics, Photonics, and Digital Technologies for Multimedia Applications III , vol. 9138. SPIE, 2014, pp. 1 - 11. -
- [18][18] P. Hanhart, M. Rerabek, P. Korshunov, and T. Ebrahimi, 'Subjective evaluation of HEVC intra coding for still image compression,' in International workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM) , 2013. -
- [19][19] R. Tolosana, S. Romero-Tapiador, J. Fierrez, and R. Vera-Rodriguez, 'Deepfakes evolution: Analysis of facial regions and fake detection performance,' arXiv preprint , 2020. [Online]. Available: https: //arxiv.org/abs/2004.07532
- Authors
- Korshunov, P., & Marcel, S.
- Affiliations
- Idiap Research Institute (CH)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2009.03155
- OpenAlex
- W3083135743
- Cited by
- 47
- References
- 17
- Citations by year
- 2025: 7 · 2024: 9 · 2023: 16 · 2022: 9 · 2021: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- Hasler Stiftung
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- deepfake detection, face swapping, crowdsourced evaluation, Xception, EfficientNet, Celeb-DF, FaceForensics++, video forensics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
