Detection of Fake and Fraudulent Faces via Neural Memory Networks
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Advances in computer vision have brought us to the point where we have the ability to synthesise realistic fake content. Such approaches are seen as a source of disinformation and mistrust, and pose serious concerns to governments around the world. Convolutional Neural Networks (CNNs) demonstrate encouraging results when detecting fake images that arise from the specific type of manipulation they are trained on. However, this success has not transitioned to unseen manipulation types, resulting in a significant gap in the line-of-defense. We propose a Hierarchical Attention Memory Network (HAMN), motivated by the social cognition processes of the human brain, for the detection of fake faces. Through visual cues and by utilising knowledge stored in neural memories, we allow the network to reason about the perceived face and anticipate it's future semantic embeddings. This renders a generalisable face tampering detection framework. Experimental results demonstrate the proposed approach achieves superior performance for fake and fraudulent face detection.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
An introduction to biometric recognition
Jain, A. K., Ross, A., & Prabhakar, S. (2004) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2003.818349 · Cited by 4,974 · cited 1×
“In addition, for different kinds of applications such as biometric-based authentication [5], [6] filtering out forged images and videos is a prominent issue [3].”
BackgroundStarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., & Choo, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00916 · Cited by 4,163
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 10×
“TABLE 10: Evaluation of the contribution of the multitask learning on the seen attacks in terms of accuracy on FaceForensics++ dataset [62], (higher is better).”
BackgroundMethodInfluentialSocial GAN: Socially Acceptable Trajectories with Generative Adversarial Networks
Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., & Alahi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00240 · Cited by 2,466
Social perception from visual cues: Role of the STS region
Allison, T., Puce, A., & McCarthy, G. (2000) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01501-1 · Cited by 2,461 · cited 1×
“In this paper, we propose a deep learning framework to detect fake and fraudulent faces in images and video which is motivated by the social perception [10] and social cognition [11] processes in the human brain.”
MethodMesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Sukhbaatar, S., Szlam, A., Weston, J., & Fergus, R. (2015) · arXiv · doi:10.48550/arxiv.1503.08895 · Cited by 1,345
A simple neural network module for relational reasoning
Santoro, A., Raposo, D., Barrett, D. G. T., Malinowski, M., Pascanu, R., Battaglia, P., & Lillicrap, T. (2017) · arXiv · doi:10.48550/arxiv.1706.01427 · Cited by 1,281 · cited 1×
“Furthermore, in [52] a similar unstacking process is utilised in order to represent relationships among image patches.”
MethodIn Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking
Li, Y., Chang, M.-C., & Lyu, S. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630787 · Cited by 1,089 · cited 2×
“Hence, the general applicability of [34] is questionable.”
BackgroundMethodGraves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058 · cited 9×
“Furthermore, in contrast to [14], [15], [44]–[47] the proposed memory addressing mechanism preserves the identity of facts stored in the memory, and effectively attends to those facts through a hierarchical attention mechanism to retrieve query specific information.”
BackgroundMethodResultInfluentialSynthesizing Obama: learning lip sync from audio
Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073640 · Cited by 1,057
Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos
Nguyen, H. H., Yamagishi, J., & Echizen, I. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682602 · Cited by 951
Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security
Chesney, R., & Citron, D. K. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3213954 · Cited by 880 · cited 1×
“As demonstrated by the DeepFake app [2], the effortless and seamless nature of synthesising realistic fake content could have a crucial impact on people’s lives and severely undermine the trust in digital media [3], [4].”
MethodA Deep Learning Approach to Universal Image Manipulation Detection Using a New Convolutional Layer
Bayar, B., & Stamm, M. C. (2016) · Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/2909827.2930786 · Cited by 838 · cited 2×
“For instance, in [7] the authors introduce a novel predictive kernel to detect pixels that deviate from the logical local structural relationships in the neighbourhood.”
BackgroundTwo-Stream Neural Networks for Tampered Face Detection
Zhou, P., Han, X., Morariu, V. I., & Davis, L. S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.229 · Cited by 700 · cited 1×
“…real face and 1,500 fake face images tampered with known attacks; (ii) TestSet-II with 1,500 real and 1,500 fake face samples with unknown attacks; and (iii) TestSet-III which is comprised of 1,776 real and 1,576 fake faces generated using the FaceSwap and SwapMe applications proposed by [67].”
MethodMulti-task Learning For Detecting and Segmenting Manipulated Facial Images and Videos
Nguyen, H. H., Fang, F., Yamagishi, J., & Echizen, I. (2019) · arXiv · doi:10.48550/arxiv.1906.06876 · Cited by 551
Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos
Nguyen, H. H., Fang, F., Yamagishi, J., & Echizen, I. (2019) · 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS) · doi:10.1109/btas46853.2019.9185974 · Cited by 549 · cited 5×
“A multi-task learning paradigm is proposed in [39] where the authors combine the fake face classification task together with the segmentation of manipulated regions.”
MethodResultInfluentialRecurrent convolutional strategies for face manipulation detection in videos
Sabir, E., Cheng, J., Jaiswal, A., AbdAlmageed, W., Masi, I., & Natarajan, P. (2019) · arXiv · doi:10.48550/arxiv.1905.00582 · Cited by 443 · cited 2×
“The authors in [36] exploit both spatial and temporal information through utilising a recurrent convolutional model.”
BackgroundMethodDeepRhythm: Exposing DeepFakes with Attentional Visual Heartbeat Rhythms
Qi, H., Guo, Q., Juefei-Xu, F., Xie, X., Ma, L., Feng, W., Liu, Y., & Zhao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.07634 · Cited by 280
DeepRhythm: Exposing DeepFakes with Attentional Visual Heartbeat Rhythms
Qi, H., Guo, Q., Juefei-Xu, F., Xie, X., Ma, L., Feng, W., Liu, Y., & Zhao, J. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413707 · Cited by 220 · cited 1×
“Furthermore, we would like to acknowledge the work done by Qi et al. [41] where they illustrate that fake and fraudulent faces can be detected by observing the remote heart-rate representation.”
MethodNeural Map: Structured Memory for Deep Reinforcement Learning
Parisotto, E., & Salakhutdinov, R. (2017) · arXiv · doi:10.48550/arxiv.1702.08360 · Cited by 178
Fake Face Detection Methods: Can They Be Generalized?
Khodabakhsh, A., Ramachandra, R., Raja, K., Wasnik, P., & Busch, C. (2018) · 2018 International Conference of the Biometrics Special Interest Group (BIOSIG) · doi:10.23919/biosig.2018.8553251 · Cited by 151
FakeLocator: Robust Localization of GAN-Based Face Manipulations
Huang, Y., Juefei-Xu, F., Guo, Q., Liu, Y., & Pu, G. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3141262 · Cited by 102 · cited 1×
“In addition, a method for detection and a localisation of GAN based face manipulations is proposed in [40].”
MethodMunkhdalai, T., & Yu, H. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1038 · Cited by 73
FakePolisher: Making DeepFakes More Detection-Evasive by Shallow Reconstruction
Huang, Y., Juefei-Xu, F., Wang, R., Guo, Q., Ma, L., Xie, X., Li, J., Miao, W., Liu, Y., & Pu, G. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413732 · Cited by 71
Cited by
A comprehensive overview of Deepfake: generation, detection, datasets, and opportunities
Seow, J. W., Lim, M. K., Phan, R. C. W., & Liu, J. K. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.09.135 · Cited by 149
A Robust GAN-Generated Face Detection Method Based on Dual-Color Spaces and an Improved Xception
Chen, B., Liu, X., Zheng, Y., Zhao, G., & Shi, Y.-Q. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3116679 · Cited by 125
“The most recent detection studies [14-34], except for two works [18] and [26], consider only the RGB color space.”
BackgroundVisual-Semantic Transformer for Face Forgery Detection
Xu, Y., Jia, G., Huang, H., Duan, J., & He, R. (2021) · 2021 IEEE International Joint Conference on Biometrics (IJCB) · doi:10.1109/ijcb52358.2021.9484407 · Cited by 24
Singh, S., & Dhumane, A. (2025) · MethodsX · doi:10.1016/j.mex.2025.103632 · Cited by 24
Metadata
- Authors
- Fernando, T., Fookes, C., Denman, S., & Sridharan, S.
- Venue
- IEEE Transactions on Information Forensics and Security
- Published
- 2021
- Volume
- 16
- Pages
- 1973-1988
- ISSN
- 1556-6013 (ISSN-L)
- 1556-6021
- OpenAlex
- W3114749049
- Cited by
- 28
- References
- 30
- FWCI
- 1.56
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.