MRE-Net: Multi-Rate Excitation Network for Deepfake Video Detection
The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
The current social media is flooded with hyper realistic face-synthetic videos due to the explosion of DeepFake technology that has brought a serious impact on human society security, which calls for further exploring on deepfake video detection methods. Existing methods attempt to isolated capture spatial artifacts or extract the homogeneous temporal inconsistency to detect deepfake video, but little attention has been paid to the exploitation of dynamic spatial-temporal inconsistency. To mitigate this issue, in this paper, we propose a novel Multi-Rate Excitation Network (MRE-Net) to effectively excite dynamic spatial-temporal inconsistency from the perspective of multiple rates for deepfake video detection. The proposed MRE-Net is composed of two components: Bipartite Group Sampling (BGS) and multiple rate branches. The BGS draws the entire video into multiple bipartite groups with different rates to cover various face motion dynamic evolution. We further design multiple rate branches to capture both short-term and long-term spatial-temporal inconsistency from corresponding bipartite groups of BGS. Concretely, for the early stages of the multi-rate branches, Momentary Inconsistency Excitation (MIE) module is developed to encode the spatial artifacts and intra-group short-term temporal inconsistency. Meanwhile, for the last stages of the multi-rate branches, Longstanding Inconsistency Excitation (LIE) module is constructed to perceive inter-group long-term temporal dynamics. Extensive experiments and visualizations conducted on four popular datasets demonstrate the effectiveness of the proposed method against state-of-the-art deepfake detection methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00813 · Cited by 6,332
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2019) · arXiv · doi:10.48550/arxiv.1912.04958 · Cited by 6,332
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., & Choo, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00916 · Cited by 4,163
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 3×
“1) Faceforensics++: Faceforensics++ [48] is known as FF++ and is a large-scale benchmark for deepfake detection, which consists of 1000 original videos collected from YouTube and 4000 corresponding fake videos manipulated by four facial forgery methods, i.”
BackgroundMesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00327 · Cited by 1,846 · cited 2×
“To evaluate the effectiveness of MRE-Net, we conduct extensive experiments on three popular and challenging deepfake detection datasets, including Faceforensics++ (FF++) [48], Celeb-DF [49], DFDC [50], and Wild Deep-fake [51].”
MethodFace2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild
Deng, J., Guo, J., Ververas, E., Kotsia, I., & Zafeiriou, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00525 · Cited by 1,681
Deepfakes and beyond: A Survey of face manipulation and fake detection
Tolosana, R., Vera-Rodriguez, R., Fierrez, J., Morales, A., & Ortega-Garcia, J. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.014 · Cited by 1,237
Face X-Ray for More General Face Forgery Detection
Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., & Guo, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00505 · Cited by 1,211
Thinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues
Qian, Y., Yin, G., Sheng, L., Chen, Z., & Shao, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58610-2_6 · Cited by 1,058
Multi-attentional Deepfake Detection
Zhao, H., Wei, T., Zhou, W., Zhang, W., Chen, D., & Yu, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00222 · Cited by 1,019
Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos
Nguyen, H. H., Yamagishi, J., & Echizen, I. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682602 · Cited by 951
Generalizing Face Forgery Detection with High-frequency Features
Luo, Y., Zhang, Y., Yan, J., & Liu, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01605 · Cited by 609
Fast Face-Swap Using Convolutional Neural Networks
Korshunova, I., Shi, W., Dambre, J., & Theis, L. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.397 · Cited by 501
DeepFakes: a New Threat to Face Recognition? Assessment and Detection
Korshunov, P., & Marcel, S. (2018) · arXiv · doi:10.48550/arxiv.1812.08685 · Cited by 497 · cited 1×
“F AKE videos partly or completely generated by digital manipulation, in particular with DeepFake methods, have become a great public concern recently [1], [2].”
MethodTwo-Branch Recurrent Network for Isolating Deepfakes in Videos
Masi, I., Killekar, A., Mascarenhas, R. M., Gurudatt, S. P., & AbdAlmageed, W. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58571-6_39 · Cited by 471
WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection
Zi, B., Chang, M., Chen, J., Ma, X., & Jiang, Y.-G. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413769 · Cited by 448 · cited 3×
“To assess the effectiveness of the proposed MRE-Net, we conduct comparative experiments against numerous state-of-the-art methods on the datasets of FF++ and WildDeepfake, including MesoNet [13], Xception [14], Face X-ray [15], Two-branch RN [24], Add-Net [51], F 3 -Net [9], and Multi-Att [19].”
BackgroundMethodThe deepfake detection challenge (dfdc) dataset
Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., & Ferrer, C. C. (2020) · arXiv · doi:10.48550/arxiv.2006.07397 · Cited by 427
Deepfake Video Detection through Optical Flow Based CNN
Amerini, I., Galteri, L., Caldelli, R., & Del Bimbo, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00152 · Cited by 424 · cited 3×
“Many researchers invest their efforts to explore the temporal modeling strategies on deepfake video detection tasks, such as optical flow ([22], [23]), “2D CNN + LSTM”-like ([24], [25], [26]), 3D CNN ([27], [28], [29]), etc.”
BackgroundMethodGlobal Texture Enhancement for Fake Face Detection in the Wild
Liu, Z., Qi, X., & Torr, P. H. S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00808 · Cited by 401
Local Relation Learning for Face Forgery Detection
Chen, S., Yao, T., Chen, Y., Ding, S., Li, J., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16193 · Cited by 334
Artificial intelligence in digital media: the era of deepfakes
Karnouskos, S. (2020) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2020.3001312 · Cited by 239
Joint Audio-Visual Deepfake Detection
Zhou, Y., & Lim, S.-N. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01453 · Cited by 217 · cited 2×
“The multimodal [46] or multi-view [41] learning aims to understand forged video from multiple modalities or views under the principles of consensus and complementation.”
BackgroundMTD-Net: Learning to Detect Deepfakes Images by Multi-Scale Texture Difference
Yang, J., Li, A., Xiao, S., Lu, W., & Gao, X. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2021.3102487 · Cited by 184 · cited 3×
“The comparative results are reflected in Table V and the performance of our MRE-Net is compared against four state-of-the-art deepfake detection methods, including MTD-Net [18], EfficientNet [58], F 3 -Net [9], and XceptionNet [14].”
BackgroundMethodOptical Flow based CNN for detection of unlearnt deepfake manipulations
Caldelli, R., Galteri, L., Amerini, I., & Del Bimbo, A. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.03.005 · Cited by 115
FakeLocator: Robust Localization of GAN-Based Face Manipulations
Huang, Y., Juefei-Xu, F., Guo, Q., Liu, Y., & Pu, G. (2022) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2022.3141262 · Cited by 102
Detecting Deepfake Videos with Temporal Dropout 3DCNN
Zhang, D., Li, C., Lin, F., Zeng, D., & Ge, S. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/178 · Cited by 77
Binh, L. M., & Woo, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19886 · Cited by 63
Nguyen, X. H., Tran, T. S., Le, V. T., Nguyen, K. D., & Truong, D.-T. (2021) · Forensic Science International: Digital Investigation · doi:10.1016/j.fsidi.2021.301108 · Cited by 53 · cited 2×
“Many researchers invest their efforts to explore the temporal modeling strategies on deepfake video detection tasks, such as optical flow ([22], [23]), “2D CNN + LSTM”-like ([24], [25], [26]), 3D CNN ([27], [28], [29]), etc.”
BackgroundMethodVideo Face Manipulation Detection Through Ensemble of CNNs
Bonettini, N., Cannas, E. D., Mandelli, S., Bondi, L., Bestagini, P., & Tubaro, S. (2021) · 2020 25th International Conference on Pattern Recognition (ICPR) · doi:10.1109/icpr48806.2021.9412711 · Cited by 27
Research works printed but not linked: 28
- 10.1109/TPAMI.2020.3034267 · DOI · Paper page
- Karras, 2017, Progressive growing of GANs for improved quality, stability, and variation, arXiv:1710.10196 · arXiv
- 10.1016/j.patcog.2021.107950 · DOI · Paper page
- 10.1109/CVPR.2017.195 · DOI · Paper page
- 10.1109/TIFS.2022.3146781 · DOI · Paper page
- 10.1109/TCSVT.2021.3133859 · DOI · Paper page
- Tariq, 2020, A convolutional LSTM based residual network for deepfake video detection, arXiv:2009.07480 · arXiv
- 10.1109/AVSS.2018.8639163 · DOI · Paper page
- de Lima, 2020, Deepfake detection using spatiotemporal convolutional networks, arXiv:2006.14749 · arXiv
- 10.1109/WIFS.2013.6707813 · DOI
- 10.1109/TIFS.2013.2265677 · DOI · Paper page
- 10.1109/TIFS.2012.2202227 · DOI · Paper page
- 10.1109/TIFS.2015.2394231 · DOI · Paper page
- 10.1007/s11042-010-0620-1 · DOI · Paper page
- 10.1109/WIFS.2015.7368565 · DOI · Paper page
- 10.1016/j.ins.2020.11.026 · DOI
- 10.1109/TMM.2021.3073267 · DOI
- 2020, Deepfakes Github
- 10.1145/3306346.3323035 · DOI · Paper page
- 10.1109/tcsvt.2022.3217950 · DOI · Paper page
- 10.1007/978-3-031-19781-9_2 · DOI
- 10.1109/TCSVT.2022.3177238 · DOI
- 10.1109/TCSVT.2021.3074259 · DOI · Paper page
- 10.1007/978-3-642-35289-8_25 · DOI · Paper page
- Tan, EfficientNet: Rethinking model scaling for convolutional neural networks, 6105
- 10.1109/ICCV.2015.510 · DOI · Paper page
- Van der Maaten, 2008, Visualizing data using t-SNE, J. Mach. Learn. Res., 9, 1
- 10.1109/ICCV.2017.74 · DOI · Paper page
Cited by
FAMM: Facial Muscle Motions for Detecting Compressed Deepfake Videos Over Social Networks
Liao, X., Wang, Y., Wang, T., Hu, J., & Wu, X. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3278310 · Cited by 169
“A new multi-rate excitation network [26] was utilized to detect Deepfake videos, which can efficiently excite dynamic spatiotemporal inconsistencies from a multi-rate perspective.”
MethodDeepfake Attacks: Generation, Detection, Datasets, Challenges, and Research Directions
Naitali, A., Ridouani, M., Salahdine, F., & Kaabouch, N. (2023) · Computers · doi:10.3390/computers12100216 · Cited by 117
DeepFake on Face and Expression Swap: A Review
Waseem, S., Abu Bakar, S. A. R. S., Ahmed, B. A., Omar, Z., Eisa, T. A. E., & Dalam, M. E. E. (2023) · IEEE Access · doi:10.1109/access.2023.3324403 · Cited by 65
PVASS-MDD: Predictive Visual-Audio Alignment Self-Supervision for Multimodal Deepfake Detection
Yu, Y., Liu, X., Ni, R., Yang, S., Zhao, Y., & Kot, A. C. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3309899 · Cited by 63
A Review of Deepfake Techniques: Architecture, Detection and Datasets
Edwards, P., Nebel, J.-C., Greenhill, D., & Liang, X. (2024) · IEEE Access · doi:10.1109/access.2024.3477257 · Cited by 50
Metadata
- Authors
- Pang, G., Zhang, B., Teng, Z., Qi, Z., & Fan, J.
- Venue
- IEEE Transactions on Circuits and Systems for Video Technology
- Published
- 2023-08
- Volume
- 33
- Issue
- 8
- Pages
- 3663-3676
- ISSN
- 1051-8215 (ISSN-L)
- 1558-2205
- OpenAlex
- W4317987748
- Cited by
- 72
- References
- 33
- FWCI
- 6.55
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Funders
- Fundamental Research Funds for the Central Universities of China (2022JBMC009)
- Natural Science Foundation of China (61972027)
- Beijing Municipal Natural Science Foundation (4212041)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.