Sharp Multiple Instance Learning for DeepFake Video Detection
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
With the rapid development of facial manipulation techniques, face forgery has received considerable attention in multimedia and computer vision community due to security concerns. Existing methods are mostly designed for single-frame detection trained with precise image-level labels or for video-level prediction by only modeling the inter-frame inconsistency, leaving potential high risks for DeepFake attackers. In this paper, we introduce a new problem of partial face attack in DeepFake video, where only video-level labels are provided but not all the faces in the fake videos are manipulated. We address this problem by multiple instance learning framework, treating faces and input video as instances and bag respectively. A sharp MIL (S-MIL) is proposed which builds direct mapping from instance embeddings to bag prediction, rather than from instance embeddings to instance prediction and then to bag prediction in traditional MIL. Theoretical analysis proves that the gradient vanishing in traditional MIL is relieved in S-MIL. To generate instances that can accurately incorporate the partially manipulated faces, spatial-temporal encoded instance is designed to fully model the intra-frame and inter-frame inconsistency, which further helps to promote the detection performance. We also construct a new dataset FFPMS for partially attacked DeepFake video detection, which can benefit the evaluation of different methods at both frame and video levels. Experiments on FFPMS and the widely used DFDC dataset verify that S-MIL is superior to other counterparts for partially attacked DeepFake video detection. In addition, S-MIL can also be adapted to traditional DeepFake image detection tasks and achieve state-of-the-art performance on single-frame datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
Solving the multiple instance problem with axis-parallel rectangles
Dietterich, T. G., Lathrop, R. H., & Lozano-Pérez, T. (1997) · Artificial Intelligence · doi:10.1016/s0004-3702(96)00034-3 · Cited by 2,866 · cited 1×
“eled individually. MIL aims to learn a model that can predict the bag label [2, 24, 52]. In the early work of MIL, pre-computed features or hand-craft features are utilized to represent the instances [12, 32]. The boom of deep learning-based approaches further enhances the ability of multi-instance learning by a large…”
BackgroundRich Models for Steganalysis of Digital Images
Fridrich, J., & Kodovsky, J. (2012) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2012.2190402 · Cited by 2,144 · cited 1×
“works related to multi-instance learning. 2.1 DeepFake Detection According to the detection target, DeepFake detection can be divided into image-level detection and video-level detection. Early works [18, 19, 35] achieve the forgery face detection by handcraft features,asthefacemanipulationtechniquesarelimitedatthatti…”
BackgroundInfluentialMesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Sirinukunwattana, K., Raza, S. E. A., Tsang, Y.-W., Snead, D. R. J., Cree, I. A., & Rajpoot, N. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2525803 · Cited by 1,224 · cited 1×
“The boom of deep learning-based approaches further enhances the ability of multi-instance learning by a large margin [34, 52], especially for medical image scenarios [39, 47].”
Background
Research works printed but not linked: 32
- 10.1016/j.artint.2013.06.003 · DOI · Paper page
- Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In arXiv preprint:1409.0473. Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In arXiv preprint:1409.0473. · arXiv
- Xinyi Ding Zohreh Raziei Eric C Larson Eli V Olinick Paul S Krueger and Michael Hahsler. 2019. Swapped Face Detection using Deep Learning and Subjective Assessment. arXiv: Learning (2019). Xinyi Ding Zohreh Raziei Eric C Larson Eli V Olinick Paul S Krueger and Michael Hahsler. 2019. Swapped Face Detection using Deep Learning and Subjective Assessment. arXiv: Learning (2019). · DOI · Paper page
- Brian Dolhansky Russ Howes Ben Pflaum Nicole Baram and Cristian Canton Ferrer. 2019. The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv preprint:1910.08854 (2019). Brian Dolhansky Russ Howes Ben Pflaum Nicole Baram and Cristian Canton Ferrer. 2019. The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv preprint:1910.08854 (2019). · arXiv
- Jiashi Feng Bingbing Ni Qi Tian and Shuicheng Yan. 2011. Geometric lp-norm feature pooling for image classification. In CVPR. Jiashi Feng Bingbing Ni Qi Tian and Shuicheng Yan. 2011. Geometric lp-norm feature pooling for image classification. In CVPR.
- Ji Feng and Zhihua Zhou. 2017. Deep MIML Network. In AAAI. Ji Feng and Zhihua Zhou. 2017. Deep MIML Network. In AAAI. · DOI · Paper page
- Miroslav Goljan and Jessica Fridrich. 2015. CFA-aware features for steganalysis of color images. electronic imaging Vol. 9409. Miroslav Goljan and Jessica Fridrich. 2015. CFA-aware features for steganalysis of color images. electronic imaging Vol. 9409. · DOI
- Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS. 2672--2680. Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS. 2672--2680.
- David Guera and Edward J Delp. 2018. Deepfake Video Detection Using Recurrent Neural Networks. In AVSS. 1--6. David Guera and Edward J Delp. 2018. Deepfake Video Detection Using Recurrent Neural Networks. In AVSS. 1--6. · DOI · Paper page
- Maximilian Ilse Jakub M Tomczak and Max Welling. 2018. Attention-based deep multiple instance learning. arXiv preprint:1802.04712 (2018). Maximilian Ilse Jakub M Tomczak and Max Welling. 2018. Attention-based deep multiple instance learning. arXiv preprint:1802.04712 (2018). · arXiv
- James D. Keeler David E. Rumelhart and Wee Kheng Leow. 1990. Integrated segmentation and recognition of hand-printed numerals. In NIPS. James D. Keeler David E. Rumelhart and Wee Kheng Leow. 1990. Integrated segmentation and recognition of hand-printed numerals. In NIPS.
- Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint:1408.5882 (2014). Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint:1408.5882 (2014). · DOI · Paper page
- Kingma Diederik P, 2014, Adam: A method for stochastic optimization. arXiv preprint:1412.6980 · arXiv
- Yuezun Li Xin Yang Pu Sun Honggang Qi and Siwei Lyu. 2019 b. Celeb-df: A new dataset for deepfake forensics. arXiv preprint:1909.12962 (2019). Yuezun Li Xin Yang Pu Sun Honggang Qi and Siwei Lyu. 2019 b. Celeb-df: A new dataset for deepfake forensics. arXiv preprint:1909.12962 (2019). · arXiv
- Tsung-Yi Lin Priya Goyal Ross Girshick Kaiming He and Piotr Dollár. 2017b. Focal loss for dense object detection. In ICCV. 2980--2988. Tsung-Yi Lin Priya Goyal Ross Girshick Kaiming He and Piotr Dollár. 2017b. Focal loss for dense object detection. In ICCV. 2980--2988.
- Zhouhan Lin Minwei Feng Cicero Nogueira Dos Santos Mo Yu Bing Xiang Bowen Zhou and Yoshua Bengio. 2017a. A Structured Self-attentive Sentence Embedding. arXiv: Computation and Language. Zhouhan Lin Minwei Feng Cicero Nogueira Dos Santos Mo Yu Bing Xiang Bowen Zhou and Yoshua Bengio. 2017a. A Structured Self-attentive Sentence Embedding. arXiv: Computation and Language.
- Oded Maron and Tomas Lozano-Perez. 1998. A framework for multiple-instance learning. In NIPS. Oded Maron and Tomas Lozano-Perez. 1998. A framework for multiple-instance learning. In NIPS.
- Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic. 2014. Weakly supervised object recognition with convolutional neural networks. In NIPS. Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic. 2014. Weakly supervised object recognition with convolutional neural networks. In NIPS.
- Nikolaos Pappas and Andrei Popescu-Belis. 2014. Explaining the stars: Weighted multiple-instance learning for aspectbased sentiment analysis. In EMNLP. Nikolaos Pappas and Andrei Popescu-Belis. 2014. Explaining the stars: Weighted multiple-instance learning for aspectbased sentiment analysis. In EMNLP. · DOI
- Nikolaos Pappas and Andrei Popescu-Belis. 2017. Explicit Document Modeling through Weighted Multiple-Instance Learning. In Journal of Artificial Intelligence Research. Nikolaos Pappas and Andrei Popescu-Belis. 2017. Explicit Document Modeling through Weighted Multiple-Instance Learning. In Journal of Artificial Intelligence Research. · DOI
- Pedro O Pinheiro and Ronan Collobert. 2015. From image-level to pixel-level labeling with convolutional networks. In CVPR. Pedro O Pinheiro and Ronan Collobert. 2015. From image-level to pixel-level labeling with convolutional networks. In CVPR. · DOI · Paper page
- 10.1109/RBME.2017.2651164 · DOI · Paper page
- 10.1109/WIFS.2017.8267647 · DOI · Paper page
- Andreas Rossler Davide Cozzolino Luisa Verdoliva Christian Riess Justus Thies and Matthias Niesner. 2018. FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces. CVPR. Andreas Rossler Davide Cozzolino Luisa Verdoliva Christian Riess Justus Thies and Matthias Niesner. 2018. FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces. CVPR.
- 10.1145/3267357.3267367 · DOI · Paper page
- 10.1145/3306346.3323035 · DOI · Paper page
- 10.1145/3292039 · DOI · Paper page
- Wang Run, 2019, Fakespotter: A simple baseline for spotting ai-synthesized fake faces. In arXiv preprint arXiv:1909.06122. · arXiv
- 10.1016/j.patcog.2017.08.026 · DOI · Paper page
- Kelvin Xu Jimmy Ba Ryan Kiros Kyunghyun Cho Aaron Courville Ruslan Salakhudinov Rich Zemel and Yoshua Bengio. 2015. Show attend and tell: Neural image caption generation with visual attention. In ICML. Kelvin Xu Jimmy Ba Ryan Kiros Kyunghyun Cho Aaron Courville Ruslan Salakhudinov Rich Zemel and Yoshua Bengio. 2015. Show attend and tell: Neural image caption generation with visual attention. In ICML.
- Sergey Zagoruyko and Nikos Komodakis. 2016. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In arXiv preprint. Sergey Zagoruyko and Nikos Komodakis. 2016. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In arXiv preprint.
- Wentao Zhu Qi Lou Yeeleng Scott Vang and Xiaohui Xie. 2017. Deep multi-instance networks with sparse label assignment for whole mammogram classification. In MICCAI. Wentao Zhu Qi Lou Yeeleng Scott Vang and Xiaohui Xie. 2017. Deep multi-instance networks with sparse label assignment for whole mammogram classification. In MICCAI. · DOI
Cited by
Deepfake Detection: A Systematic Literature Review
Rana, M. S., Nobi, M. N., Murali, B., & Sung, A. H. (2022) · IEEE Access · doi:10.1109/access.2022.3154404 · Cited by 573
Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain
Liu, H., Li, X., Zhou, W., Chen, Y., He, Y., Xue, H., Zhang, W., & Yu, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00083 · Cited by 600
Self-citationLearning Self-Consistency for Deepfake Detection
Zhao, T., Xu, X., Xu, M., Ding, H., Xiong, Y., & Xia, W. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01475 · Cited by 383
Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection
Chen, L., Zhang, Y., Song, Y., Liu, L., & Wang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01815 · Cited by 307
“We further evaluate our method against several state-of-the-art models, including the Two-stream [59], MesoNet [1], Headpose [53], FWA [27], VA-MLP [33], Capsule [35], SMIL [25], Two-branch [32], SPSL [29], MADD [57].”
MethodUCF: Uncovering Common Features for Generalizable Deepfake Detection
Yan, Z., Zhang, Y., Fan, Y., & Wu, B. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02048 · Cited by 241
“…DSP-FWA [25] FF++ 0.646 0.646 VA-MLP [32] FF++ 0.550 - Multi-task [33] FF++ 0.543 - Headpose [52] UADFV 0.546 - Capsule [34] FF++ 0.575 0.575 SMIL [24] FF++ 0.563 0.563 Two-branch [ operate in the entire feature space, which inevitably includes these unrelated aspects (similarly indicated in…”
Influential
Metadata
- Authors
- Li, X., Lang, Y., Chen, Y., Mao, X., He, Y., Wang, S., Xue, H., & Lu, Q.
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1864-1872
- OpenAlex
- W3048932886
- Cited by
- 157
- References
- 17
- FWCI
- 6.81
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Sharp Multiple Instance Learning for DeepFake Video Detection. https://app.cochat.ai/papers/10.1145/3394171.3414034
- The paper
- Li, X., Lang, Y., Chen, Y., Mao, X., He, Y., Wang, S., Xue, H., & Lu, Q. (2020). Sharp Multiple Instance Learning for DeepFake Video Detection. Proceedings of the 28th ACM International Conference on Multimedia. https://doi.org/10.1145/3394171.3414034
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.