CoChat

Sharp Multiple Instance Learning for DeepFake Video Detection

Open access
Li, Xiaodan, Lang, Yining, Chen, Yuefeng, Mao, Xiaofeng, He, Yuan, Wang, Shuhui, Xue, Hui, Lu, Quan · 2020 · Proceedings of the 28th ACM International Conference on Multimedia
Cited by 157

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Abstract

With the rapid development of facial manipulation techniques, face forgery has received considerable attention in multimedia and computer vision community due to security concerns. Existing methods are mostly designed for single-frame detection trained with precise image-level labels or for video-level prediction by only modeling the inter-frame inconsistency, leaving potential high risks for DeepFake attackers. In this paper, we introduce a new problem of partial face attack in DeepFake video, where only video-level labels are provided but not all the faces in the fake videos are manipulated. We address this problem by multiple instance learning framework, treating faces and input video as instances and bag respectively. A sharp MIL (S-MIL) is proposed which builds direct mapping from instance embeddings to bag prediction, rather than from instance embeddings to instance prediction and then to bag prediction in traditional MIL. Theoretical analysis proves that the gradient vanishing in traditional MIL is relieved in S-MIL. To generate instances that can accurately incorporate the partially manipulated faces, spatial-temporal encoded instance is designed to fully model the intra-frame and inter-frame inconsistency, which further helps to promote the detection performance. We also construct a new dataset FFPMS for partially attacked DeepFake video detection, which can benefit the evaluation of different methods at both frame and video levels. Experiments on FFPMS and the widely used DFDC dataset verify that S-MIL is superior to other counterparts for partially attacked DeepFake video detection. In addition, S-MIL can also be adapted to traditional DeepFake image detection tasks and achieve state-of-the-art performance on single-frame datasets.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3394171.3414034

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 17 references

Research works printed but not linked: 32

  1. 10.1016/j.artint.2013.06.003 · DOI · Paper page
  2. Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In arXiv preprint:1409.0473. Dzmitry Bahdanau Kyunghyun Cho and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In arXiv preprint:1409.0473. · arXiv
  3. Xinyi Ding Zohreh Raziei Eric C Larson Eli V Olinick Paul S Krueger and Michael Hahsler. 2019. Swapped Face Detection using Deep Learning and Subjective Assessment. arXiv: Learning (2019). Xinyi Ding Zohreh Raziei Eric C Larson Eli V Olinick Paul S Krueger and Michael Hahsler. 2019. Swapped Face Detection using Deep Learning and Subjective Assessment. arXiv: Learning (2019). · DOI · Paper page
  4. Brian Dolhansky Russ Howes Ben Pflaum Nicole Baram and Cristian Canton Ferrer. 2019. The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv preprint:1910.08854 (2019). Brian Dolhansky Russ Howes Ben Pflaum Nicole Baram and Cristian Canton Ferrer. 2019. The Deepfake Detection Challenge (DFDC) Preview Dataset. arXiv preprint:1910.08854 (2019). · arXiv
  5. Jiashi Feng Bingbing Ni Qi Tian and Shuicheng Yan. 2011. Geometric lp-norm feature pooling for image classification. In CVPR. Jiashi Feng Bingbing Ni Qi Tian and Shuicheng Yan. 2011. Geometric lp-norm feature pooling for image classification. In CVPR.
  6. Ji Feng and Zhihua Zhou. 2017. Deep MIML Network. In AAAI. Ji Feng and Zhihua Zhou. 2017. Deep MIML Network. In AAAI. · DOI · Paper page
  7. Miroslav Goljan and Jessica Fridrich. 2015. CFA-aware features for steganalysis of color images. electronic imaging Vol. 9409. Miroslav Goljan and Jessica Fridrich. 2015. CFA-aware features for steganalysis of color images. electronic imaging Vol. 9409. · DOI
  8. Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS. 2672--2680. Ian Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron Courville and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS. 2672--2680.
  9. David Guera and Edward J Delp. 2018. Deepfake Video Detection Using Recurrent Neural Networks. In AVSS. 1--6. David Guera and Edward J Delp. 2018. Deepfake Video Detection Using Recurrent Neural Networks. In AVSS. 1--6. · DOI · Paper page
  10. Maximilian Ilse Jakub M Tomczak and Max Welling. 2018. Attention-based deep multiple instance learning. arXiv preprint:1802.04712 (2018). Maximilian Ilse Jakub M Tomczak and Max Welling. 2018. Attention-based deep multiple instance learning. arXiv preprint:1802.04712 (2018). · arXiv
  11. James D. Keeler David E. Rumelhart and Wee Kheng Leow. 1990. Integrated segmentation and recognition of hand-printed numerals. In NIPS. James D. Keeler David E. Rumelhart and Wee Kheng Leow. 1990. Integrated segmentation and recognition of hand-printed numerals. In NIPS.
  12. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint:1408.5882 (2014). Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint:1408.5882 (2014). · DOI · Paper page
  13. Kingma Diederik P, 2014, Adam: A method for stochastic optimization. arXiv preprint:1412.6980 · arXiv
  14. Yuezun Li Xin Yang Pu Sun Honggang Qi and Siwei Lyu. 2019 b. Celeb-df: A new dataset for deepfake forensics. arXiv preprint:1909.12962 (2019). Yuezun Li Xin Yang Pu Sun Honggang Qi and Siwei Lyu. 2019 b. Celeb-df: A new dataset for deepfake forensics. arXiv preprint:1909.12962 (2019). · arXiv
  15. Tsung-Yi Lin Priya Goyal Ross Girshick Kaiming He and Piotr Dollár. 2017b. Focal loss for dense object detection. In ICCV. 2980--2988. Tsung-Yi Lin Priya Goyal Ross Girshick Kaiming He and Piotr Dollár. 2017b. Focal loss for dense object detection. In ICCV. 2980--2988.
  16. Zhouhan Lin Minwei Feng Cicero Nogueira Dos Santos Mo Yu Bing Xiang Bowen Zhou and Yoshua Bengio. 2017a. A Structured Self-attentive Sentence Embedding. arXiv: Computation and Language. Zhouhan Lin Minwei Feng Cicero Nogueira Dos Santos Mo Yu Bing Xiang Bowen Zhou and Yoshua Bengio. 2017a. A Structured Self-attentive Sentence Embedding. arXiv: Computation and Language.
  17. Oded Maron and Tomas Lozano-Perez. 1998. A framework for multiple-instance learning. In NIPS. Oded Maron and Tomas Lozano-Perez. 1998. A framework for multiple-instance learning. In NIPS.
  18. Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic. 2014. Weakly supervised object recognition with convolutional neural networks. In NIPS. Maxime Oquab Leon Bottou Ivan Laptev and Josef Sivic. 2014. Weakly supervised object recognition with convolutional neural networks. In NIPS.
  19. Nikolaos Pappas and Andrei Popescu-Belis. 2014. Explaining the stars: Weighted multiple-instance learning for aspectbased sentiment analysis. In EMNLP. Nikolaos Pappas and Andrei Popescu-Belis. 2014. Explaining the stars: Weighted multiple-instance learning for aspectbased sentiment analysis. In EMNLP. · DOI
  20. Nikolaos Pappas and Andrei Popescu-Belis. 2017. Explicit Document Modeling through Weighted Multiple-Instance Learning. In Journal of Artificial Intelligence Research. Nikolaos Pappas and Andrei Popescu-Belis. 2017. Explicit Document Modeling through Weighted Multiple-Instance Learning. In Journal of Artificial Intelligence Research. · DOI
  21. Pedro O Pinheiro and Ronan Collobert. 2015. From image-level to pixel-level labeling with convolutional networks. In CVPR. Pedro O Pinheiro and Ronan Collobert. 2015. From image-level to pixel-level labeling with convolutional networks. In CVPR. · DOI · Paper page
  22. 10.1109/RBME.2017.2651164 · DOI · Paper page
  23. 10.1109/WIFS.2017.8267647 · DOI · Paper page
  24. Andreas Rossler Davide Cozzolino Luisa Verdoliva Christian Riess Justus Thies and Matthias Niesner. 2018. FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces. CVPR. Andreas Rossler Davide Cozzolino Luisa Verdoliva Christian Riess Justus Thies and Matthias Niesner. 2018. FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces. CVPR.
  25. 10.1145/3267357.3267367 · DOI · Paper page
  26. 10.1145/3306346.3323035 · DOI · Paper page
  27. 10.1145/3292039 · DOI · Paper page
  28. Wang Run, 2019, Fakespotter: A simple baseline for spotting ai-synthesized fake faces. In arXiv preprint arXiv:1909.06122. · arXiv
  29. 10.1016/j.patcog.2017.08.026 · DOI · Paper page
  30. Kelvin Xu Jimmy Ba Ryan Kiros Kyunghyun Cho Aaron Courville Ruslan Salakhudinov Rich Zemel and Yoshua Bengio. 2015. Show attend and tell: Neural image caption generation with visual attention. In ICML. Kelvin Xu Jimmy Ba Ryan Kiros Kyunghyun Cho Aaron Courville Ruslan Salakhudinov Rich Zemel and Yoshua Bengio. 2015. Show attend and tell: Neural image caption generation with visual attention. In ICML.
  31. Sergey Zagoruyko and Nikos Komodakis. 2016. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In arXiv preprint. Sergey Zagoruyko and Nikos Komodakis. 2016. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In arXiv preprint.
  32. Wentao Zhu Qi Lou Yeeleng Scott Vang and Xiaohui Xie. 2017. Deep multi-instance networks with sparse label assignment for whole mammogram classification. In MICCAI. Wentao Zhu Qi Lou Yeeleng Scott Vang and Xiaohui Xie. 2017. Deep multi-instance networks with sparse label assignment for whole mammogram classification. In MICCAI. · DOI

Cited by

See all 157

Metadata

Authors
Li, X., Lang, Y., Chen, Y., Mao, X., He, Y., Wang, S., Xue, H., & Lu, Q.
Venue
Proceedings of the 28th ACM International Conference on Multimedia
Publisher
ACM
Published
2020-10-12
Pages
1864-1872
OpenAlex
W3048932886
Cited by
157
References
17
FWCI
6.81
Type
conference paper
Language
en
Open access
bronze

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Sharp Multiple Instance Learning for DeepFake Video Detection. https://app.cochat.ai/papers/10.1145/3394171.3414034
The paper
Li, X., Lang, Y., Chen, Y., Mao, X., He, Y., Wang, S., Xue, H., & Lu, Q. (2020). Sharp Multiple Instance Learning for DeepFake Video Detection. Proceedings of the 28th ACM International Conference on Multimedia. https://doi.org/10.1145/3394171.3414034

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.