DeepFake Detection Based on Discrepancies Between Faces and Their Context
Abstract
We propose a method for detecting face swapping and other identity manipulations in single images. Face swapping methods, such as DeepFake, manipulate the face region, aiming to adjust the face to the appearance of its context, while leaving the context unchanged. We show that this modus operandi produces discrepancies between the two regions (e.g., Fig. 1). These discrepancies offer exploitable telltale signs of manipulation. Our approach involves two networks: (i) a face identification network that considers the face region bounded by a tight semantic segmentation, and (ii) a context recognition network that considers the face context (e.g., hair, ears, neck). We describe a method which uses the recognition signals from our two networks to detect such discrepancies, providing a complementary detection signal that improves conventional real versus fake classifiers commonly used for detecting fake images. Our method achieves state of the art results on the FaceForensics++ and Celeb-DF-v2 benchmarks for face manipulation detection, and even generalizes to detect fakes produced by unseen methods.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · see citing works
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636 · see citing works
VGGFace2: A Dataset for Recognising Faces across Pose and Age
Cao, Q., Shen, L., Xie, W., Parkhi, O. M., & Zisserman, A. (2018) · 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018) · doi:10.1109/fg.2018.00020 · Cited by 3,032 · see citing works
Rich Models for Steganalysis of Digital Images
Fridrich, J., & Kodovsky, J. (2012) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2012.2190402 · Cited by 2,144 · see citing works
The Deepfake Detection Challenge (DFDC) Preview Dataset
Dolhansky, B., Howes, R., Pflaum, B., Baram, N., & Ferrer, C. C. (2019) · arXiv · doi:10.48550/arxiv.1910.08854 · Cited by 413 · see citing works
Cozzolino, D., Poggi, G., & Verdoliva, L. (2017) · Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/3082031.3083247 · Cited by 349 · see citing works
Face swapping: automatically replacing faces in photographs
Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P., & Nayar, S. K. (2008) · ACM Transactions on Graphics · doi:10.1145/1360612.1360638 · Cited by 333 · see citing works
DeepRhythm: Exposing DeepFakes with Attentional Visual Heartbeat Rhythms
Qi, H., Guo, Q., Juefei-Xu, F., Xie, X., Ma, L., Feng, W., Liu, Y., & Zhao, J. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413707 · Cited by 220 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Deepfake Detection: A Systematic Literature Review
Rana, M. S., Nobi, M. N., Murali, B., & Sung, A. H. (2022) · IEEE Access · doi:10.1109/access.2022.3154404 · Cited by 573 · see citing works
Learning Self-Consistency for Deepfake Detection
Zhao, T., Xu, X., Xu, M., Ding, H., Xiong, Y., & Xia, W. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01475 · Cited by 383 · see citing works
Deepfake detection using deep learning methods: A systematic and comprehensive review
Heidari, A., Jafari Navimipour, N., Dag, H., & Unal, M. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1520 · Cited by 349 · see citing works
“Deepfakes can thus be used to incite political or religious tensions between countries, deceive the public and influence election results, or create havoc in financial markets by spreading false information (Jeyaraj & Dwivedi, 2020; Nirkin et al., 2021).”
BackgroundSelf-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection
Chen, L., Zhang, Y., Song, Y., Liu, L., & Wang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01815 · Cited by 307 · see citing works
Implicit identity leakage: The stumbling block to improving deepfake detection generalization
Dong, S., Wang, J., Ji, R., Liang, J., Fan, H., & Ge, Z. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00389 · Cited by 220 · see citing works
Countering Malicious DeepFakes: Survey, Battleground, and Horizon
Juefei-Xu, F., Wang, R., Huang, Y., Guo, Q., Ma, L., & Liu, Y. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01606-8 · Cited by 191 · see citing works
AVoiD-DF: Audio-Visual Joint Learning for Detecting Deepfake
Yang, W., Zhou, X., Chen, Z., Guo, B., Ba, Z., Xia, Z., Cao, X., & Ren, K. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3262148 · Cited by 198 · see citing works
A Novel Blockchain-Based Deepfake Detection Method Using Federated and Deep Learning Models
Heidari, A., Navimipour, N. J., Dag, H., Talebi, S., & Unal, M. (2024) · Cognitive Computation · doi:10.1007/s12559-024-10255-7 · Cited by 164 · see citing works
Detect and locate: Exposing face manipulation by semantic-and noise-level telltales
Kong, C., Chen, B., Li, H., Wang, S., Rocha, A., & Kwong, S. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3169921 · Cited by 120 · see citing works
Deepfake Attacks: Generation, Detection, Datasets, Challenges, and Research Directions
Naitali, A., Ridouani, M., Salahdine, F., & Kaabouch, N. (2023) · Computers · doi:10.3390/computers12100216 · Cited by 117 · see citing works
TALL: Thumbnail Layout for Deepfake Video Detection
Xu, Y., Liang, J., Jia, G., Yang, Z., Zhang, Y., & He, R. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02071 · Cited by 109 · see citing works
Deepfakes and scientific knowledge dissemination
Doss, C., Mondschein, J., Shu, D., Wolfson, T., Kopecky, D., Fitton-Kane, V. A., Bush, L., & Tucker, C. (2023) · Scientific Reports · doi:10.1038/s41598-023-39944-3 · Cited by 90 · see citing works
An efficient deepfake video detection using robust deep learning
Qadir, A., Mahum, R., El-Meligy, M. A., Ragab, A. E., AlSalman, A., & Awais, M. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25757 · Cited by 78 · see citing works
Deepfake Video Detection Based on MesoNet with Preprocessing Module
Xia, Z., Qiao, T., Xu, M., Wu, X., Han, L., & Chen, Y. (2022) · Symmetry · doi:10.3390/sym14050939 · Cited by 56 · see citing works
Towards DeepFake video forensics based on facial textural disparities in multi-color channels
Xia, Z., Qiao, T., Xu, M., Zheng, N., & Xie, S. (2022) · Information Sciences · doi:10.1016/j.ins.2022.06.003 · Cited by 47 · see citing works
From Black Boxes to Glass Boxes: Explainable AI for Trustworthy Deepfake Forensics
Qian, H., Xia, L., Ge, R., Fan, Y., Wang, Q., & Jing, Z. (2025) · Cryptography · doi:10.3390/cryptography9040061 · Cited by 10 · see citing works
“Re-synthesis [21] evaluates an image’s compatibility with models trained on genuine data, while contextual identity checks [22] test for semantic contradictions.”
Multisemantic Path Neural Network for Deepfake Detection
Wu, N., Jin, X., Jiang, Q., Wang, P., Zhang, Y., Yao, S., & Zhou, W. (2022) · Security and Communication Networks · doi:10.1155/2022/4976848 · Cited by 9 · see citing works
UniForensics: Face Forgery Detection via General Facial Representation
Fang, Z., Zhao, H., Wei, T., Zhou, W., Wan, M., Wang, Z., Zhang, W., & Yu, N. (2024) · arXiv · doi:10.48550/arxiv.2407.19079 · Cited by 8 · see citing works
Nguyen-Le, H.-H., Tran, V.-T., Nguyen, D.-T., & Le-Khac, N.-A. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11608-4 · Cited by 1 · see citing works
- Authors
- Nirkin, Y., Wolf, L., Keller, Y., & Hassner, T.
- Affiliations
- Bar-Ilan University (IL)
- Tel Aviv University (IL)
- Meta (United States) (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2022-10-01
- Volume
- 44
- Issue
- 10
- Pages
- 6111-6121
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3173581984
- Cited by
- 312
- References
- 10
- Citations by year
- 2026: 47 · 2025: 88 · 2024: 74 · 2023: 65 · 2022: 31 · 2021: 7
- FWCI
- 15.93
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- Funders
- European Research Council (ERC CoG 725974)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- deepfake detection, face swapping, identity manipulation, face manipulation detection, face identification, context recognition, semantic segmentation, FaceForensics++
- MeSH terms
- Algorithms, Face
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
