PVASS-MDD: Predictive Visual-Audio Alignment Self-Supervision for Multimodal Deepfake Detection
The DOI matches its Crossref record; 71 of the 83 references it cites were found in a registry or an index.
Abstract
Deepfake techniques can forge the visual or audio signals in the video, which leads to inconsistencies between visual and audio (VA) signals. Therefore, multimodal detection methods expose deepfake videos by extracting VA inconsistencies. Recently, deepfake technology has started VA collaborative forgery to obtain more realistic deepfake videos, which poses new challenges for extracting VA inconsistencies. Recent multimodal detection methods propose to first extract natural VA correspondences in real videos in a self-supervised manner, and then use the learned real correspondences as targets to guide the extraction of VA inconsistencies in the subsequent deepfake detection stage. However, the inherent VA relations are difficult to extract due to the modality gap, which leads to the limited auxiliary performance of the aforementioned self-supervised methods. In this paper, we propose Predictive Visual-audio Alignment Self-supervision for Multimodal Deepfake Detection (PVASS-MDD), which consists of PVASS auxiliary and MDD stages. In the PVASS auxiliary stage in real videos, we first devise a three-stream network to associate two augmented visual views with corresponding audio clues, leading to explore common VA correspondences based on cross-view learning. Secondly, we introduce a novel cross-modal predictive align module for eliminating VA gaps to provide inherent VA correspondences. In the MDD stage, we propose to the auxiliary loss to utilize the frozen PVASS network to align VA features of real videos, to better assist multimodal deepfake detector for capturing subtle VA inconsistencies. We conduct extensive experiments on existing widely used and latest multimodal deepfake datasets. Our method obtains a significant performance improvement compared to state-of-the-art methods.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Constructed upon the foundation of YouTubeVOS 2018 [60], which aggregates over 4,000 videos from YouTube, the TVIL dataset leverages one of the most prolific platforms for video content as its base.”
From Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization (2024) · cites this paper as [60]
Citations
Cites
Generative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314 · cited 2×
“Afterwards, researchers propose some simple CNN methods [21], [51], [52] to learn spatial clues.”
MethodLiu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00320 · Cited by 2,294
Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., & Hu, H. (2021) · arXiv · doi:10.48550/arxiv.2106.13230 · Cited by 2,294
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952132 · Cited by 2,023
MesoNet: a Compact Facial Video Forgery Detection Network
Afchar, D., Nozick, V., Yamagishi, J., & Echizen, I. (2018) · 2018 IEEE International Workshop on Information Forensics and Security (WIFS) · doi:10.1109/wifs.2018.8630761 · Cited by 1,872
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Niessner, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.262 · Cited by 1,747
Desplanques, B., Thienpondt, J., & Demuynck, K. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2650 · Cited by 1,568
Deferred neural rendering: image synthesis using neural textures
Thies, J., Zollhöfer, M., & Nießner, M. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323035 · Cited by 1,356
Exposing Deep Fakes Using Inconsistent Head Poses
Yang, X., Li, Y., & Lyu, S. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8683164 · Cited by 1,196
Multi-attentional Deepfake Detection
Zhao, H., Wei, T., Zhou, W., Zhang, W., Chen, D., & Yu, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00222 · Cited by 1,019 · cited 4×
“To explore more fine-grained clues, some detection methods focuses on local regions for finding tiny defects that various forgery methods left [18], [22], [23], [24], [53].”
BackgroundMethodInfluentialCapsule-forensics: Using Capsule Networks to Detect Forged Images and Videos
Nguyen, H. H., Yamagishi, J., & Echizen, I. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682602 · Cited by 951
Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations
Matern, F., Riess, C., & Stamminger, M. (2019) · 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw.2019.00020 · Cited by 824
Exposing DeepFake Videos By Detecting Face Warping Artifacts
Li, Y., & Lyu, S. (2018) · arXiv · doi:10.48550/arxiv.1811.00656 · Cited by 785 · cited 1×
“For detecting visual deepfake , early methods are proposed to focus on visual artifacts, such as face warping artifacts [48], [49] and absonant head poses [50].”
MethodSpatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain
Liu, H., Li, X., Zhou, W., Chen, Y., He, Y., Xue, H., Zhang, W., & Yu, N. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00083 · Cited by 600
DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection
Jiang, L., Li, R., Wu, W., Qian, C., & Loy, C. C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00296 · Cited by 599
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537 · cited 1×
“A. Datasets 1) Auxiliary Datasets: LRW [66]: We use the LRW dataset without the labels for our extra real samples.”
MethodLips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection
Haliassos, A., Vougioukas, K., Petridis, S., & Pantic, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00500 · Cited by 537 · cited 4×
“Moreover, because our experimental dataset and settings are follow [44], thus the results of methods [23], [26], [30], [31], [33], [34], [35], [36], [37], [44], [46], [69] are from [44].”
BackgroundMethodInfluentialTwo-Branch Recurrent Network for Isolating Deepfakes in Videos
Masi, I., Killekar, A., Mascarenhas, R. M., Gurudatt, S. P., & AbdAlmageed, W. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58571-6_39 · Cited by 471
The Deepfake Detection Challenge (DFDC) Preview Dataset
Dolhansky, B., Howes, R., Pflaum, B., Baram, N., & Ferrer, C. C. (2019) · arXiv · doi:10.48550/arxiv.1910.08854 · Cited by 413 · cited 1×
“DFDC [67]: Deepfake Detection Challenge (DFDC) dataset is a publicly available face swap video dataset.”
MethodAASIST: Audio Anti-Spoofing Using Integrated Spectro-Temporal Graph Attention Networks
Jung, J.-W., Heo, H.-S., Tak, H., Shim, H.-J., Chung, J. S., Lee, B.-J., Yu, H.-J., & Evans, N. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747766 · Cited by 399
SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
Mehri, S., Kumar, K., Gulrajani, I., Kumar, R., Jain, S., Sotelo, J., Courville, A. C., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.07837 · Cited by 392
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
Zhou, H., Sun, Y., Wu, W., Loy, C. C., Wang, X., & Liu, Z. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00416 · Cited by 370
What Makes Fake Images Detectable? Understanding Properties that Generalize
Chai, L., Bau, D., Lim, S.-N., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58574-7_7 · Cited by 357
Exploring Temporal Coherence for More General Video Face Forgery Detection
Zheng, Y., Bao, J., Chen, D., Zeng, M., & Wen, F. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01477 · Cited by 320
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413570 · Cited by 319 · cited 5×
“Furthermore, we compare the multimodal detection methods [35], [36], [37], [43], [44].”
BackgroundMethodInfluentialSelf-supervised Learning of Adversarial Example: Towards Good Generalizations for Deepfake Detection
Chen, L., Zhang, Y., Song, Y., Liu, L., & Wang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01815 · Cited by 307
Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization
Chugh, K., Gupta, P., Dhall, A., & Subramanian, R. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413700 · Cited by 225 · cited 5×
“Furthermore, we compare the multimodal detection methods [35], [36], [37], [43], [44].”
BackgroundMethodInfluentialJoint Audio-Visual Deepfake Detection
Zhou, Y., & Lim, S.-N. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01453 · Cited by 217 · cited 5×
“Furthermore, we compare the multimodal detection methods [35], [36], [37], [43], [44].”
MethodInfluentialAudio-Driven Emotional Video Portraits
Ji, X., Zhou, H., Wang, K., Wu, W., Loy, C. C., Cao, X., & Xu, F. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01386 · Cited by 212
AVoiD-DF: Audio-Visual Joint Learning for Detecting Deepfake
Yang, W., Zhou, X., Chen, Z., Guo, B., Ba, Z., Xia, Z., Cao, X., & Ren, K. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3262148 · Cited by 198
AnoPCN: Video Anomaly Detection via Deep Predictive Coding Network
Ye, M., Peng, X., Gan, W., Wu, W., & Qiao, Y. (2019) · Proceedings of the 27th ACM International Conference on Multimedia · doi:10.1145/3343031.3350899 · Cited by 197
Spatiotemporal Inconsistency Learning for DeepFake Video Detection
Gu, Z., Chen, Y., Yao, T., Ding, S., Li, J., Huang, F., & Ma, L. (2021) · Proceedings of the 29th ACM International Conference on Multimedia · doi:10.1145/3474085.3475508 · Cited by 189 · cited 2×
“Therefore, some recent detectors [15], [16], [17], [19], [20], [25], [26], [27], [28] capture spatiotemporal information clues that are vital for visual deepfake detection.”
BackgroundMethodIlyas, H., Javed, A., & Malik, K. M. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110124 · Cited by 186
Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection
Haliassos, A., Mira, R., Petridis, S., & Pantic, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01453 · Cited by 180
ResNet and Model Fusion for Automatic Spoofing Detection
Chen, Z., Xie, Z., Zhang, W., & Xu, X. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1085 · Cited by 171
FAMM: Facial Muscle Motions for Detecting Compressed Deepfake Videos Over Social Networks
Liao, X., Wang, Y., Wang, T., Hu, J., & Wu, X. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3278310 · Cited by 169 · cited 2×
“Therefore, some recent detectors [15], [16], [17], [19], [20], [25], [26], [27], [28] capture spatiotemporal information clues that are vital for visual deepfake detection.”
BackgroundMethodJung, J.-W., Heo, H.-S., Kim, J.-H., Shim, H.-J., & Yu, H.-J. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1982 · Cited by 166
DeepFake video detection using convolutional vision transformer
Wodajo, D., & Atnafu, S. (2021) · arXiv · doi:10.48550/arxiv.2102.11126 · Cited by 147
Masked Relation Learning for DeepFake Detection
Yang, Z., Liang, J., Xu, Y., Zhang, X.-Y., & He, R. (2023) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3249566 · Cited by 143
FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset
Khalid, H., Tariq, S., Kim, M., & Woo, S. S. (2021) · arXiv · doi:10.48550/arxiv.2108.05080 · Cited by 116
Yang, J., Xiao, S., Li, A., Lu, W., Gao, X., & Li, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3133859 · Cited by 115
Jung, J.-W., Kim, S.-B., Shim, H.-J., Kim, J.-H., & Yu, H.-J. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1011 · Cited by 109
Voice-Face Homogeneity Tells Deepfake
Cheng, H., Guo, Y., Wang, T., Li, Q., Chang, X., & Nie, L. (2024) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3625231 · Cited by 102
SPAct: Self-supervised Privacy Preservation for Action Recognition
Dave, I. R., Chen, C., & Shah, M. A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01953 · Cited by 102
Self-Supervised Video Forensics by Audio-Visual Anomaly Detection
Feng, C., Chen, Z., & Owens, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01011 · Cited by 101 · cited 9×
“The SS Auxiliary methods [46], [47], [58] and multimodal detection method [43] are not compared in [44], thus the results of these methods are reproduced by ourselves.”
BackgroundMethodInfluentialCai, Z., Stefanov, K., Dhall, A., & Hayat, M. (2022) · 2022 International Conference on Digital Image Computing: Techniques and Applications (DICTA) · doi:10.1109/dicta56598.2022.10034605 · Cited by 99
Audeo: Audio Generation for a Silent Performance VideoPreprint
Su, K., Liu, X., & Shlizerman, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14348 · Cited by 85
Self-citationChen, H., Lin, Y., Li, B., & Tan, S. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3209336 · Cited by 80
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
Liu, H., Chen, Z., Yuan, Y. G., Mei, X., Liu, X., Mandic, D. P., Wang, W., & Plumbley, M. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.12503 · Cited by 79
Self-citationMRE-Net: Multi-Rate Excitation Network for Deepfake Video Detection
Pang, G., Zhang, B., Teng, Z., Qi, Z., & Fan, J. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3239607 · Cited by 72
MSVT: Multiple Spatiotemporal Views Transformer for DeepFake Video Detection
Yu, Y., Ni, R., Zhao, Y., Yang, S., Xia, F., Jiang, N., & Zhao, G. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3281448 · Cited by 67
Self-citationDeepfake Video Detection via Predictive Representation Learning
Ge, S., Lin, F., Li, C., Zhang, D., Wang, W., & Zeng, D. (2022) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3536426 · Cited by 64
ArchiSound: Audio Generation with DiffusionPreprint
Schneider, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.13267 · Cited by 57
Exploiting Complementary Dynamic Incoherence for DeepFake Video Detection
Wang, H., Liu, Z., & Wang, S. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3238517 · Cited by 55
Real face foundation representation learning for generalized deepfake detection
Shi, L., Zhang, J., Ji, Z., Bai, J., & Shan, S. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111299 · Cited by 55
DEAR-GAN: Degradation-Aware Face Restoration With GAN Prior
Hu, Y., Wang, Y., & Zhang, J. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3244786 · Cited by 52
Unconditional Audio Generation with Generative Adversarial Networks and Cycle Regularization
Liu, J.-Y., Chen, Y.-H., Yeh, Y.-C., & Yang, Y. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1137 · Cited by 42
Towards an End-to-End Visual-to-Raw-Audio Generation With GAN
Liu, S., Li, S., & Cheng, H. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3079897 · Cited by 42
Applications of Deep Learning to Audio Generation
Zhao, Y., Xia, X., & Togneri, R. B. (2019) · IEEE Circuits and Systems Magazine · doi:10.1109/mcas.2019.2945210 · Cited by 28
Self-citationExploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generation
Lin, Y.-B., & Wang, Y.-C. F. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16302 · Cited by 28
Abnormal event detection using adversarial predictive coding for motion and appearance
Yu, J., Kim, J.-G., Gwak, J., Lee, B., & Jeon, M. (2021) · Information Sciences · doi:10.1016/j.ins.2021.11.001 · Cited by 24
Self-supervised Transformer for Deepfake Detection
Zhao, H., Zhou, W., Chen, D., Zhang, W., & Yu, N. (2022) · arXiv · doi:10.48550/arxiv.2203.01265 · Cited by 24 · cited 1×
“Some methods [58], [59], [60] learn temporally dense video representations in a self-supervised way, then to assist visual deepfake detection.”
MethodMonteiro, J., Alam, J., & Falk, T. H. (2019) · Machine learning for signal processing ... · doi:10.1109/mlsp.2019.8918703 · Cited by 22
Reciprocal GAN Through Characteristic Functions (RCF-GAN)
Li, S., Yu, Z., Xiang, M., & Mandic, D. P. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3157444 · Cited by 20
Unsupervised Audiovisual Synthesis via Exemplar AutoencodersPreprint
Deng, K., Bansal, A., & Ramanan, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.04463 · Cited by 17
Self-Supervised Human Activity Recognition by Augmenting Generative Adversarial Networks
Zadeh, M. Z., Babu, A. R., Jaiswal, A., Kyrarini, M., & Makedon, F. (2021) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3453892.3453893 · Cited by 13
Real Face Foundation Representation Learning for Generalized Deepfake DetectionPreprint
Shi, L., Zhang, J., & Shan, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.08439 · Cited by 10
Improving Weakly Supervised Sound Event Detection with Self-Supervised Auxiliary Tasks
Deshmukh, S., Raj, B., & Singh, R. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-2079 · Cited by 9
SP-SEDT: Self-supervised Pre-training for Sound Event Detection TransformerPreprint
Ye, Z., Wang, X., Liu, H., Qian, Y., Tao, R., Yan, L., & Ouchi, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.15222 · Cited by 2
Research works printed but not linked: 2
- 2020, Faceswap
- 2020, Deepfakes
Cited by
A Comprehensive Review of DeepFake Detection Using Advanced Machine Learning and Fusion Methods
Gupta, G., Raja, K., Gupta, M., Jan, T., Whiteside, S. T., & Prasad, M. (2023) · Electronics · doi:10.3390/electronics13010095 · Cited by 99
A systematic literature review on deepfake detection techniques
Sharma, V. K., Garg, R. K., & Caudron, Q. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-19906-1 · Cited by 88
Audio–visual deepfake detection using articulatory representation learning
Wang, Y., & Huang, H. (2024) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2024.104133 · Cited by 28
Locality-Aware Cross-Modal Correspondence Learning for Dense Audio-Visual Events Detection
Xing, L., Qu, H., Yan, R., Shu, X., & Tang, J. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3629609 · Cited by 70
Multimodal Imbalance-Aware Gradient Modulation for Weakly-Supervised Audio-Visual Video Parsing
Fu, J., Gao, J., Bao, B., & Xu, C. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3337134 · Cited by 33
Metadata
- Authors
- Yu, Y., Liu, X., Ni, R., Yang, S., Zhao, Y., & Kot, A. C.
- Affiliations
- Beijing Jiaotong University (CN)
- Nanyang Technological University (SG)
- Venue
- IEEE Transactions on Circuits and Systems for Video Technology
- Published
- 2024-08
- Volume
- 34
- Issue
- 8
- Pages
- 6926-6936
- ISSN
- 1051-8215 (ISSN-L)
- 1558-2205
- OpenAlex
- W4386267173
- Cited by
- 63
- References
- 83
- Citations by year
- 2026: 26 · 2025: 24 · 2024: 11
- FWCI
- 5.70
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Circuits and Systems for Video Technology (publishedVersion)
- DR-NTU (Nanyang Technological University) (submittedVersion)
- Funders
- National Key Research and Development Program of China (2021ZD0112100)
- National NSF of China (62336001, U1936212, 62120106009)
- Beijing NSF (4222014)
- Rapid-Rich Object Search (ROSE) Laboratory, Nanyang Technological University, Singapore
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- deepfake detection, self-supervised learning, modality gap
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.