Local Relation Learning for Face Forgery Detection
Open accessAbstract
With the rapid development of facial manipulation techniques, face forgery has received considerable attention in digital media forensics due to security concerns. Most existing methods formulate face forgery detection as a classification problem and utilize binary labels or manipulated region masks as supervision. However, without considering the correlation between local regions, these global supervisions are insufficient to learn a generalized feature and prone to overfitting. To address this issue, we propose a novel perspective of face forgery detection via local relation learning. Specifically, we propose a Multi-scale Patch Similarity Module (MPSM), which measures the similarity between features of local regions and forms a robust and generalized similarity pattern. Moreover, we propose an RGB-Frequency Attention Module (RFAM) to fuse information in both RGB and frequency domains for more comprehensive local feature representation, which further improves the reliability of the similarity pattern. Extensive experiments show that the proposed method consistently outperforms the state-of-the-arts on widely-used benchmarks. Furthermore, detailed visualization shows the robustness and interpretability of our method.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Rich Models for Steganalysis of Digital Images
Fridrich, J., & Kodovsky, J. (2012) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2012.2190402 · Cited by 2,144 · see citing works
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00327 · Cited by 1,846 · cited 7× · see citing works
“Face X-ray (Li et al. 2020a) localizes the forged boundary in a self-supervised mechanism, achieving remarkable performance at high-quality data.”
InfluentialDetecting and Simulating Artifacts in GAN Fake Images
Zhang, X., Karaman, S., & Chang, S.-F. (2019) · 2019 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs47025.2019.9035107 · Cited by 567 · see citing works
The Deepfake Detection Challenge (DFDC) Preview Dataset
Dolhansky, B., Howes, R., Pflaum, B., Baram, N., & Ferrer, C. C. (2019) · arXiv · doi:10.48550/arxiv.1910.08854 · Cited by 413 · see citing works
Global Texture Enhancement for Fake Face Detection in the Wild
Liu, Z., Qi, X., & Torr, P. H. S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00808 · Cited by 401 · see citing works
Cozzolino, D., Poggi, G., & Verdoliva, L. (2017) · Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/3082031.3083247 · Cited by 349 · see citing works
Flagged first, then by citations
Research works printed but not linked: 15
- Afchar, D.; Nozick, V.; Yamagishi, J.; and Echizen, I. 2018. MesoNet: a Compact Facial Video Forgery Detection Network. 2018 IEEE International Workshop on Information Forensics and Security (WIFS) 1-7.
- Bappy, J. H.; Roy-Chowdhury, A.; Bunk, J.; Nataraj, L.; and Manjunath, B. S. 2017. Exploiting Spatial Structure for Localizing Manipulated Image Regions. 2017 IEEE International Conference on Computer Vision (ICCV) 4980-4989.
- Durall, R.; Keuper, M.; Pfreundt, F.; and Keuper, J. 2019. Unmasking DeepFakes with simple Features. ArXiv abs/1911.00686. · arXiv
- Li, L.; Bao, J.; Zhang, T.; Yang, H.; Chen, D.; Wen, F.; and Guo, B. 2020a. Face X-Ray for More General Face Forgery Detection. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 5000-5009.
- Lin, M.; Ji, R.; Wang, Y.; Zhang, Y.; Zhang, B.; Tian, Y.; and Shao, L. 2020. HRank: Filter Pruning Using High-Rank Feature Map. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 1526-1535.
- Nguyen, H. H.; Fang, F.; Yamagishi, J.; and Echizen, I. 2019. Multi-task Learning For Detecting and Segmenting Manipulated Facial Images and Videos. ArXiv abs/1906.06876. · arXiv
- Nguyen, H. H.; Yamagishi, J.; and Echizen, I. 2019. Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos. ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2307-2311.
- Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; Devito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017. Automatic differentiation in PyTorch. In NeurIPS Autodiff Workshop .
- Qian, Y.; Yin, G.; Sheng, L.; Chen, Z.; and Shao, J. 2020. Thinking in Frequency: Face Forgery Detection by Mining Frequency-aware Clues. ArXiv abs/2007.09355. · arXiv
- Rahmouni, N.; Nozick, V.; Yamagishi, J.; and Echizen, I. 2017. Distinguishing computer graphics from natural images using convolution neural networks. 2017 IEEE Workshop on Information Forensics and Security (WIFS) 1-6.
- R¨ ossler, A.; Cozzolino, D.; Verdoliva, L.; Riess, C.; Thies, J.; and Nießner, M. 2019. FaceForensics++: Learning to Detect Manipulated Facial Images. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) 1-11.
- Salloum, R.; Ren, Y.; and Kuo, C.-C. J. 2018. Image Splicing Localization using a Multi-task Fully Convolutional Network (MFCN). ArXiv abs/1709.02016. · arXiv
- Stehouwer, J.; Dang, H.; Liu, F.; Liu, X.; and Jain, A. 2020. On the Detection of Digital Face Manipulation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 5780-5789.
- Thies, J.; Zollh¨ ofer, M.; and Nießner, M. 2019. Deferred Neural Rendering: Image Synthesis using Neural Textures. arXiv: Computer Vision and Pattern Recognition .
- Thies, J.; Zollh¨ ofer, M.; Stamminger, M.; Theobalt, C.; and Nießner, M. 2019. Face2Face: real-time face capture and reenactment of RGB videos. Commun. ACM 62: 96-104.
Cited by
Sorted by citations
Detecting Deepfakes with Self-Blended Images
Shiohara, K., & Yamasaki, T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01816 · Cited by 503 · see citing works
End-to-End Reconstruction-Classification Learning for Face Forgery Detection
Cao, J., Ma, C., Yao, T., Chen, S., Ding, S., & Yang, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00408 · Cited by 428 · see citing works
Self-citationM2TR: Multi-modal Multi-scale Transformers for Deepfake Detection
Wang, J., Wu, Z., Ouyang, W., Han, X., Chen, J., Jiang, Y.-G., & Li, S.-N. (2022) · Proceedings of the 2022 International Conference on Multimedia Retrieval · doi:10.1145/3512527.3531415 · Cited by 368 · see citing works
Implicit identity leakage: The stumbling block to improving deepfake detection generalization
Dong, S., Wang, J., Ji, R., Liang, J., Fan, H., & Ge, Z. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00389 · Cited by 220 · see citing works
Self-citationDual Contrastive Learning for General Face Forgery Detection
Sun, K., Yao, T., Chen, S., Ding, S., Li, J., & Ji, R. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20130 · Cited by 209 · see citing works
“The overall intra-instance contrastive loss L intra within a batch can be derived as : Different from (Chen et al. 2021) which directly use similarity pattern as features, our proposed intra-instance contrastive loss enhances the inconsistency of forgery face by pulling away the similarity of real…”
BackgroundSelf-citationSpatiotemporal Inconsistency Learning for DeepFake Video Detection
Gu, Z., Chen, Y., Yao, T., Ding, S., Li, J., Huang, F., & Ma, L. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475508 · Cited by 189 · see citing works
Self-citationF <sup>2</sup> Trans: High-Frequency Fine-Grained Transformer for Face Forgery Detection
Miao, C., Tan, Z., Chu, Q., Liu, H., Hu, H., & Yu, N. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3233774 · Cited by 172 · see citing works
Wang, Y., Yu, K., Chen, C., Hu, X., & Peng, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00703 · Cited by 161 · see citing works
CORE: Consistent Representation Learning for Face Forgery Detection
Ni, Y., Meng, D., Yu, C., Quan, C., Ren, D., & Zhao, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00011 · Cited by 136 · see citing works
Exploring Frequency Adversarial Attacks for Face Forgery Detection
Jia, S., Ma, C., Yao, T., Yin, B., Ding, S., & Yang, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00407 · Cited by 97 · see citing works
Self-citationDelving into the Local: Dynamic Inconsistency Learning for DeepFake Video Detection
Gu, Z., Chen, Y., Yao, T., Ding, S., Li, J., & Ma, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19955 · Cited by 93 · see citing works
Self-citationDeepFake Detection Based on High-Frequency Enhancement Network for Highly Compressed Content
Gao, J., Xia, Z., Marcialis, G. L., Dang, C., Dai, J., & Feng, X. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123732 · Cited by 67 · see citing works
Depth map guided triplet network for deepfake face detection
Liang, B., Wang, Z., Huang, B., Zou, Q., Wang, Q., & Liang, J. (2023) · Neural Networks · doi:10.1016/j.neunet.2022.11.031 · Cited by 54 · see citing works
Exploring Disentangled Content Information for Face Forgery Detection
Liang, J., Shi, H., & Deng, W. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19781-9_8 · Cited by 58 · see citing works
Locate and Verify: A Two-Stream Network for Improved Deepfake Detection
Shuai, C., Zhong, J., Wu, S., Lin, F., Wang, Z., Ba, Z., Liu, Z., Cavallaro, L., & Ren, K. (2023) · Proceedings of the 31st ACM International Conference on Multimedia · doi:10.1145/3581783.3612386 · Cited by 55 · see citing works
DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection
Shao, R., Wu, T., Nie, L., & Liu, Z. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-024-02274-6 · Cited by 48 · see citing works
Forgery-Aware Adaptive Learning With Vision Transformer for Generalized Face Forgery Detection
Luo, A., Cai, R., Kong, C., Ju, Y., Kang, X., Huang, J., & Kot, A. C. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3522091 · Cited by 41 · see citing works
DFIL: Deepfake Incremental Learning by Exploiting Domain-invariant Forgery Clues
Pan, K., Yin, Y., Wei, Y., Lin, F., Ba, Z., Liu, Z., Wang, Z., Cavallaro, L., & Ren, K. (2023) · Proceedings of the 31st ACM International Conference on Multimedia · doi:10.1145/3581783.3612377 · Cited by 37 · see citing works
Fsbi: Deepfake detection with frequency enhanced self-blended images
Abul Hasanaath, A., Luqman, H., Katib, R., & Anwar, S. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105418 · Cited by 34 · see citing works
A hierarchical feature selection strategy for deepfake video detection
Mohiuddin, S., Sheikh, K. H., Malakar, S., Velásquez, J. D., & Sarkar, R. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08201-z · Cited by 25 · see citing works
She, H., Hu, Y., Liu, B., Li, J., & Li, C.-T. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3451356 · Cited by 24 · see citing works
“Other examples include using texture features [35], frequency features [7], [36], [37], identify features [38] and patch consistency features [39], [40].”
BackgroundSelf-citationFine-Grained Multimodal DeepFake Classification via Heterogeneous Graphs
Yin, Q., Lu, W., Cao, X., Luo, X., Zhou, Y., & Huang, J. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02128-1 · Cited by 23 · see citing works
Mining Generalized Multi-timescale Inconsistency for Detecting Deepfake Videos
Yu, Y., Ni, R., Yang, S., Ni, Y., Zhao, Y., & Kot, A. C. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-024-02249-7 · Cited by 7 · see citing works
Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges
Liu, P., Tao, Q., & Zhou, J. T. (2024) · arXiv · doi:10.48550/arxiv.2406.06965 · Cited by 6 · see citing works
Liu, P., Tao, Q., & Zhou, J. (2024) · Proceedings of the 1st ACM Multimedia Workshop on Multi-modal Misinformation Governance in the Era of Foundation Models · doi:10.1145/3689090.3689386 · Cited by 3 · see citing works
- Authors
- Chen, S., Yao, T., Chen, Y., Ding, S., Li, J., & Ji, R.
- Affiliations
- Xiamen University (CN)
- Tencent (China) (CN)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2021-05-18
- Volume
- 35
- Issue
- 2
- Pages
- 1081-1088
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W3158353280
- Cited by
- 334
- References
- 10
- Citations by year
- 2026: 30 · 2025: 87 · 2024: 98 · 2023: 74 · 2022: 39 · 2021: 6
- FWCI
- 17.40
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62072387, 62025603, U1705262, 62072389, 61802324, 2020-08, No. 62025603, 61772443, 62002305, No.61702136, No. 62002305, No.61802324, 62072386, 61702136, No.U1705262)
- Basic and Applied Basic Research Foundation of Guangdong Province (No. 2019B1515120049, 2019B1515120049)
- Instituto de Ciencias del Mar y Limnología, Universidad Nacional Autónoma de México
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- face forgery detection, face manipulation, feature fusion, overfitting, interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
