Sharp Multiple Instance Learning for DeepFake Video Detection
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
With the rapid development of facial manipulation techniques, face forgery has received considerable attention in multimedia and computer vision community due to security concerns. Existing methods are mostly designed for single-frame detection trained with precise image-level labels or for video-level prediction by only modeling the inter-frame inconsistency, leaving potential high risks for DeepFake attackers. In this paper, we introduce a new problem of partial face attack in DeepFake video, where only video-level labels are provided but not all the faces in the fake videos are manipulated. We address this problem by multiple instance learning framework, treating faces and input video as instances and bag respectively. A sharp MIL (S-MIL) is proposed which builds direct mapping from instance embeddings to bag prediction, rather than from instance embeddings to instance prediction and then to bag prediction in traditional MIL. Theoretical analysis proves that the gradient vanishing in traditional MIL is relieved in S-MIL. To generate instances that can accurately incorporate the partially manipulated faces, spatial-temporal encoded instance is designed to fully model the intra-frame and inter-frame inconsistency, which further helps to promote the detection performance. We also construct a new dataset FFPMS for partially attacked DeepFake video detection, which can benefit the evaluation of different methods at both frame and video levels. Experiments on FFPMS and the widely used DFDC dataset verify that S-MIL is superior to other counterparts for partially attacked DeepFake video detection. In addition, S-MIL can also be adapted to traditional DeepFake image detection tasks and achieve state-of-the-art performance on single-frame datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 5×
“whether a video is fake or not than other methods on DFDC benchmark. S-MIL can also be adapted to traditional singleframe detection task and achieve state-of-the-art performance on the FF++ datasets [43], showing a promising result for face information protection. The major contributions of our paper include: •We intr…”
BackgroundMethodInfluentialSolving the multiple instance problem with axis-parallel rectangles
Dietterich, T. G., Lathrop, R. H., & Lozano-Pérez, T. (1997) · Artificial Intelligence · doi:10.1016/s0004-3702(96)00034-3 · Cited by 2,866 · cited 1×
“eled individually. MIL aims to learn a model that can predict the bag label [2, 24, 52]. In the early work of MIL, pre-computed features or hand-craft features are utilized to represent the instances [12, 32]. The boom of deep learning-based approaches further enhances the ability of multi-instance learning by a large…”
BackgroundRich Models for Steganalysis of Digital Images
Fridrich, J., & Kodovsky, J. (2012) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2012.2190402 · Cited by 2,144 · cited 1×
“works related to multi-instance learning. 2.1 DeepFake Detection According to the detection target, DeepFake detection can be divided into image-level detection and video-level detection. Early works [18, 19, 35] achieve the forgery face detection by handcraft features,asthefacemanipulationtechniquesarelimitedatthatti…”
BackgroundInfluentialZagoruyko, S., & Komodakis, N. (2016) · arXiv · doi:10.48550/arxiv.1612.03928 · Cited by 2,027
MesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Sirinukunwattana, K., Raza, S. E. A., Tsang, Y.-W., Snead, D. R. J., Cree, I. A., & Rajpoot, N. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2525803 · Cited by 1,224 · cited 1×
“The boom of deep learning-based approaches further enhances the ability of multi-instance learning by a large margin [34, 52], especially for medical image scenarios [39, 47].”
BackgroundFace X-Ray for More General Face Forgery Detection
Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., & Guo, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00505 · Cited by 1,211
A Deep Learning Approach to Universal Image Manipulation Detection Using a New Convolutional Layer
Bayar, B., & Stamm, M. C. (2016) · Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/2909827.2930786 · Cited by 838
Exposing DeepFake Videos By Detecting Face Warping Artifacts
Li, Y., & Lyu, S. (2018) · arXiv · doi:10.48550/arxiv.1811.00656 · Cited by 785 · cited 2×
“es the interpretability of the proposed method. Methods DFDC Celeb FFPMS DF F2F FS NT XN-avg [43] 0.8458 0.9944 0.8036 0.7714 0.8036 0.7000 XN-max [43] 0.7687 0.8989 0.8536 0.7821 0.8571 0.6571 D-FWA [28] 0.8511 0.9858 0.7964 0.7571 0.8036 0.7214 LSTM [22] 0.7902 0.9573 0.8500 0.7564 0.7750 0.7393 I3D [7] 0.8082 0.992…”
MethodInfluentialRecurrent convolutional strategies for face manipulation detection in videos
Sabir, E., Cheng, J., Jaiswal, A., AbdAlmageed, W., Masi, I., & Natarajan, P. (2019) · arXiv · doi:10.48550/arxiv.1905.00582 · Cited by 443 · cited 3×
“lly manipulated in one DeepFake video due to arbitrary content attack behaviors. Existing DeepFake video detection methods can be divided into frame-based [13, 33, 42, 43, 48] and video-based methods [45]. Frame-based methods meet with three major difficulties when applied to fake video detection with partial faces at…”
BackgroundMethodInfluentialDetection of GAN-Generated Fake Images over Social Networks
Marra, F., Gragnaniello, D., Cozzolino, D., & Verdoliva, L. (2018) · 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr.2018.00084 · Cited by 399 · cited 1×
“dly, faces of one target person may be partially manipulated in one DeepFake video due to arbitrary content attack behaviors. Existing DeepFake video detection methods can be divided into frame-based [13, 33, 42, 43, 48] and video-based methods [45]. Frame-based methods meet with three major difficulties when applied…”
MethodCozzolino, D., Poggi, G., & Verdoliva, L. (2017) · Proceedings of the 5th ACM Workshop on Information Hiding and Multimedia Security · doi:10.1145/3082031.3083247 · Cited by 349
Exposing image splicing with inconsistent local noise variances
Pan, X., Zhang, X., & Lyu, S. (2012) · 2012 IEEE International Conference on Computational Photography (ICCP) · doi:10.1109/iccphot.2012.6215223 · Cited by 120 · cited 1×
“works related to multi-instance learning. 2.1 DeepFake Detection According to the detection target, DeepFake detection can be divided into image-level detection and video-level detection. Early works [18, 19, 35] achieve the forgery face detection by handcraft features,asthefacemanipulationtechniquesarelimitedatthatti…”
BackgroundInfluential
Research works printed but not linked: 29
- 10.1016/j.artint.2013.06.003 · DOI · Paper page
- Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In arXiv preprint:1409.0473. Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In arXiv preprint:1409.0473. · arXiv
- Xinyi Ding Zohreh Raziei Eric C Larson Eli V Olinick Paul S Krueger and Michael Hahsler. 2019. Swapped Face Detection using Deep Learning and Subjective Assessment. arXiv: Learning (2019). Xinyi Ding Zohreh Raziei Eric C Larson Eli V Olinick Paul S Krueger and Michael Hahsler. 2019. Swapped Face Detection using Deep Learning and Subjective Assessment. arXiv: Learning (2019). · DOI · Paper page
- Brian Dolhansky Russ Howes Ben Pflaum Nicole Baram and Cristian Canton Ferrer. 2019. The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv preprint:1910.08854 (2019). Brian Dolhansky Russ Howes Ben Pflaum Nicole Baram and Cristian Canton Ferrer. 2019. The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv preprint:1910.08854 (2019). · arXiv
- Jiashi Feng Bingbing Ni Qi Tian and Shuicheng Yan. 2011. Geometric lp-norm feature pooling for image classification. In CVPR. Jiashi Feng Bingbing Ni Qi Tian and Shuicheng Yan. 2011. Geometric lp-norm feature pooling for image classification. In CVPR.
- Ji Feng and Zhihua Zhou. 2017. Deep MIML Network. In AAAI. Ji Feng and Zhihua Zhou. 2017. Deep MIML Network. In AAAI. · DOI · Paper page
- Miroslav Goljan and Jessica Fridrich. 2015. CFA-aware features for steganalysis of color images. electronic imaging Vol. 9409. Miroslav Goljan and Jessica Fridrich. 2015. CFA-aware features for steganalysis of color images. electronic imaging Vol. 9409. · DOI · Paper page
- Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS. 2672--2680. Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS. 2672--2680.
- David Guera and Edward J Delp. 2018. Deepfake Video Detection Using Recurrent Neural Networks. In AVSS. 1--6. David Guera and Edward J Delp. 2018. Deepfake Video Detection Using Recurrent Neural Networks. In AVSS. 1--6. · DOI · Paper page
- Maximilian Ilse Jakub M Tomczak and Max Welling. 2018. Attention-based deep multiple instance learning. arXiv preprint:1802.04712 (2018). Maximilian Ilse Jakub M Tomczak and Max Welling. 2018. Attention-based deep multiple instance learning. arXiv preprint:1802.04712 (2018). · arXiv
- James D. Keeler David E. Rumelhart and Wee Kheng Leow. 1990. Integrated segmentation and recognition of hand-printed numerals. In NIPS. James D. Keeler David E. Rumelhart and Wee Kheng Leow. 1990. Integrated segmentation and recognition of hand-printed numerals. In NIPS.
- Kingma Diederik P, 2014, Adam: A method for stochastic optimization. arXiv preprint:1412.6980 · arXiv
- Yuezun Li Xin Yang Pu Sun Honggang Qi and Siwei Lyu. 2019 b. Celeb-df: A new dataset for deepfake forensics. arXiv preprint:1909.12962 (2019). Yuezun Li Xin Yang Pu Sun Honggang Qi and Siwei Lyu. 2019 b. Celeb-df: A new dataset for deepfake forensics. arXiv preprint:1909.12962 (2019). · arXiv
- Zhouhan Lin Minwei Feng Cicero Nogueira Dos Santos Mo Yu Bing Xiang Bowen Zhou and Yoshua Bengio. 2017a. A Structured Self-attentive Sentence Embedding. arXiv: Computation and Language. Zhouhan Lin Minwei Feng Cicero Nogueira Dos Santos Mo Yu Bing Xiang Bowen Zhou and Yoshua Bengio. 2017a. A Structured Self-attentive Sentence Embedding. arXiv: Computation and Language.
- Oded Maron and Tomas Lozano-Perez. 1998. A framework for multiple-instance learning. In NIPS. Oded Maron and Tomas Lozano-Perez. 1998. A framework for multiple-instance learning. In NIPS.
- Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic. 2014. Weakly supervised object recognition with convolutional neural networks. In NIPS. Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic. 2014. Weakly supervised object recognition with convolutional neural networks. In NIPS.
- Nikolaos Pappas and Andrei Popescu-Belis. 2014. Explaining the stars: Weighted multiple-instance learning for aspectbased sentiment analysis. In EMNLP. Nikolaos Pappas and Andrei Popescu-Belis. 2014. Explaining the stars: Weighted multiple-instance learning for aspectbased sentiment analysis. In EMNLP. · DOI · Paper page
- Nikolaos Pappas and Andrei Popescu-Belis. 2017. Explicit Document Modeling through Weighted Multiple-Instance Learning. In Journal of Artificial Intelligence Research. Nikolaos Pappas and Andrei Popescu-Belis. 2017. Explicit Document Modeling through Weighted Multiple-Instance Learning. In Journal of Artificial Intelligence Research. · DOI
- Pedro O Pinheiro and Ronan Collobert. 2015. From image-level to pixel-level labeling with convolutional networks. In CVPR. Pedro O Pinheiro and Ronan Collobert. 2015. From image-level to pixel-level labeling with convolutional networks. In CVPR. · DOI · Paper page
- 10.1109/RBME.2017.2651164 · DOI · Paper page
- 10.1109/WIFS.2017.8267647 · DOI · Paper page
- Andreas Rossler Davide Cozzolino Luisa Verdoliva Christian Riess Justus Thies and Matthias Niesner. 2018. FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces. CVPR. Andreas Rossler Davide Cozzolino Luisa Verdoliva Christian Riess Justus Thies and Matthias Niesner. 2018. FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces. CVPR.
- 10.1145/3267357.3267367 · DOI · Paper page
- 10.1145/3306346.3323035 · DOI · Paper page
- 10.1145/3292039 · DOI · Paper page
- Wang Run, 2019, Fakespotter: A simple baseline for spotting ai-synthesized fake faces. In arXiv preprint arXiv:1909.06122. · arXiv
- 10.1016/j.patcog.2017.08.026 · DOI · Paper page
- Kelvin Xu Jimmy Ba Ryan Kiros Kyunghyun Cho Aaron Courville Ruslan Salakhudinov Rich Zemel and Yoshua Bengio. 2015. Show attend and tell: Neural image caption generation with visual attention. In ICML. Kelvin Xu Jimmy Ba Ryan Kiros Kyunghyun Cho Aaron Courville Ruslan Salakhudinov Rich Zemel and Yoshua Bengio. 2015. Show attend and tell: Neural image caption generation with visual attention. In ICML.
- Wentao Zhu Qi Lou Yeeleng Scott Vang and Xiaohui Xie. 2017. Deep multi-instance networks with sparse label assignment for whole mammogram classification. In MICCAI. Wentao Zhu Qi Lou Yeeleng Scott Vang and Xiaohui Xie. 2017. Deep multi-instance networks with sparse label assignment for whole mammogram classification. In MICCAI. · DOI
Cited by
Deepfake Detection: A Systematic Literature Review
Rana, M. S., Nobi, M. N., Murali, B., & Sung, A. H. (2022) · IEEE Access · doi:10.1109/access.2022.3154404 · Cited by 573
Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain
Liu, H., Li, X., Zhou, W., Chen, Y., He, Y., Xue, H., Zhang, W., & Yu, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00083 · Cited by 600
Self-citationLearning Self-Consistency for Deepfake Detection
Zhao, T., Xu, X., Xu, M., Ding, H., Xiong, Y., & Xia, W. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01475 · Cited by 383
Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection
Chen, L., Zhang, Y., Song, Y., Liu, L., & Wang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01815 · Cited by 307
“We further evaluate our method against several state-of-the-art models, including the Two-stream [59], MesoNet [1], Headpose [53], FWA [27], VA-MLP [33], Capsule [35], SMIL [25], Two-branch [32], SPSL [29], MADD [57].”
MethodUCF: Uncovering Common Features for Generalizable Deepfake Detection
Yan, Z., Zhang, Y., Fan, Y., & Wu, B. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02048 · Cited by 241
“…DSP-FWA [25] FF++ 0.646 0.646 VA-MLP [32] FF++ 0.550 - Multi-task [33] FF++ 0.543 - Headpose [52] UADFV 0.546 - Capsule [34] FF++ 0.575 0.575 SMIL [24] FF++ 0.563 0.563 Two-branch [ operate in the entire feature space, which inevitably includes these unrelated aspects (similarly indicated in…”
Influential
Metadata
- Authors
- Li, X., Lang, Y., Chen, Y., Mao, X., He, Y., Wang, S., Xue, H., & Lu, Q.
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1864-1872
- OpenAlex
- W3048932886
- Cited by
- 157
- References
- 20
- FWCI
- 6.81
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.