The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Contrastive Learning for DeepFake Classification and Localization via Multi-Label Ranking
Abstract
We propose a unified approach to simultaneously addressing the conventional setting of binary deepfake classification and a more challenging scenario of uncovering what facial components have been forged as well as the exact order of the manipulations. To solve the former task, we consider mul-tiple instance learning (MIL) that takes each image as a bag and its patches as instances. A positive bag corresponds to a forged image that includes at least one manipulated patch (i.e., a pixel in the feature map). The formulation allows us to estimate the probability of an input image being a fake one and establish the corresponding contrastive MIL loss. On the other hand, tackling the component-wise deepfake problem can be reduced to solving multi-label prediction, but the requirement to recover the manipulation order further complicates the learning task into a multi-label ranking prob-lem. We resolve this difficulty by designing a tailor-made loss term to enforce that the rank order of the predicted multi-label probabilities respects the ground-truth order of the sequential modifications of a deepfake image. Through extensive experiments and comparisons with other relevant techniques, we provide extensive results and ablation studies to demonstrate that the proposed method is an overall more comprehensive solution to deepfake detection.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Despite these efforts, recent methods still face challenges, often learning dataset-specific cues that fail to detect unseen forgery samples.”
From Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model (2024) · cites this paper as [14]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · cited 2×
“The initial category concentrates on classification tasks [3, 24, 40], employing exclusively authentic and counterfeit annotations for training.”
BackgroundMethodCeleb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
Li, Y., Yang, X., Sun, P., Qi, H., & Lyu, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00327 · Cited by 1,846 · cited 1×
“…standard protocol was adhered to by training each model exclusively on the FaceForensics++ (FF++) [30] dataset, followed by evaluating their performance on the test sets of Celeb-DF (CDF) [22], WildDeep-fake (WDF) [44], DeepFakeDetection (DFD) [30], and Deep-Fake Detection Challenge (DFDC) [12]..”
MethodSpatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain
Liu, H., Li, X., Zhou, W., Chen, Y., He, Y., Xue, H., Zhang, W., & Yu, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00083 · Cited by 600
Detecting Deepfakes with Self-Blended Images
Shiohara, K., & Yamasaki, T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01816 · Cited by 503
End-to-End Reconstruction-Classification Learning for Face Forgery Detection
Cao, J., Ma, C., Yao, T., Chen, S., Ding, S., & Yang, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00408 · Cited by 428
The Deepfake Detection Challenge (DFDC) Preview Dataset
Dolhansky, B., Howes, R., Pflaum, B., Baram, N., & Ferrer, C. C. (2019) · arXiv · doi:10.48550/arxiv.1910.08854 · Cited by 413
Learning Self-Consistency for Deepfake Detection
Zhao, T., Xu, X., Xu, M., Ding, H., Xiong, Y., & Xia, W. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01475 · Cited by 383
Self-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection
Chen, L., Zhang, Y., Song, Y., Liu, L., & Wang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01815 · Cited by 307
Joint Audio-Visual Deepfake Detection
Zhou, Y., & Lim, S.-N. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01453 · Cited by 217
Face Forgery Detection by 3D Decomposition
Zhu, X., Wang, H., Fei, H., Lei, Z., & Li, S. Z. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00295 · Cited by 106
Research works printed but not linked: 32
- 10.1145/1468075.1468121 · DOI
- Chen, 2021, Local relation learning for face forgery detection, 1081
- 10.1109/TPAMI.2006.248 · DOI
- 10.1109/CVPR.2017.195 · DOI · Paper page
- 10.1109/iccv48922.2021.01483 · DOI · Paper page
- 10.1109/CVPR42600.2020.00525 · DOI · Paper page
- 10.1007/978-3-031-24628-9_23 · DOI
- 10.1016/S0004-3702(96)00034-3 · DOI · Paper page
- Dong, 2022, Protecting celebrities from deepfake with iden-tity consistency transformer, 9458
- 10.1007/978-3-031-19775-8_35 · DOI
- Ilse, 2018, Attention-based deep multiple instance learning, 2132
- 10.1109/TPAMI.2020.2970919 · DOI · Paper page
- 10.1109/TPAMI.2020.2970919 · DOI · Paper page
- 10.1109/CVPR46437.2021.00091 · DOI
- 10.1109/CVPR42600.2020.00505 · DOI · Paper page
- Li, 2020, Sharp mul-tiple instance learning for deepfake video detection, 1864, 10.1145/3394171.3414034 · DOI
- Li, 2020, Celebdf: A large-scale challenging dataset for deep fake forensics, 3204
- Han Lim, 2016, A box-constrained approach for hard permutation problems, 2454
- Luo, 2021, Gener-alizing face forgery detection with high-frequency features, 16317
- Petersen, Differentiable sorting networks for scalable sorting and ranking supervision, 8546
- Petersen, Monotonic differentiable sorting networks
- Rossler, 2019, Faceforen-sics++: Learning to detect manipulated facial images, 1
- 10.1109/ICCV.2017.74 · DOI · Paper page
- 10.1007/978-3-031-19778-9_41 · DOI
- Shao, 2023, Robust sequential deepfake detection, CoRR, abs/2309.14991 · arXiv
- Tan, 2019, Efficientnet: Rethinking model scaling for convolutional neural networks, 6105
- 10.1007/978-3-031-19803-8_41 · DOI
- 10.1109/iccv.2019.01017 · DOI
- 10.1109/tifs.2024.3361151 · DOI
- 10.1109/CVPR52688.2022.01824 · DOI
- Zhao, 2021, Multi-attentional deep-fake detection, 2185
- 10.1145/3394171.3413769 · DOI · Paper page
Cited by
Sorted by citations
Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges
Liu, P., Tao, Q., & Zhou, J. T. (2024) · arXiv · doi:10.48550/arxiv.2406.06965 · Cited by 6
Nguyen-Le, H.-H., Tran, V.-T., Nguyen, D.-T., & Le-Khac, N.-A. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11608-4 · Cited by 28
Han, Y.-H., Huang, T.-M., Hua, K.-L., & Chen, J.-C. (2024) · arXiv · doi:10.48550/arxiv.2404.05583 · Cited by 45
Metadata
- Authors
- Hong, C.-Y., Hsu, Y.-C., & Liu, T.-L.
- Affiliations
- Institute of Information Science, Academia Sinica (TW)
- Venue
- 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2024-06-16
- Pages
- 17627-17637
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4402754235
- Cited by
- 23
- References
- 13
- Citations by year
- 2026: 8 · 2025: 13 · 2024: 2
- FWCI
- 4.76
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- deepfake detection, contrastive learning, multiple instance learning, face manipulation, ranking loss
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
