The DOI matches its Crossref record; 117 of the 117 references it cites were found in a registry or an index.
Self-Supervised Video Forensics by Audio-Visual Anomaly Detection
Open accessAbstract
Manipulated videos often contain subtle inconsistencies between their visual and audio signals. We propose a video forensics method, based on anomaly detection, that can identify these inconsistencies, and that can be trained solely using real, unlabeled data. We train an autoregressive model to generate sequences of audio-visual features, using feature sets that capture the temporal synchronization between video frames and sound. At test time, we then flag videos that the model assigns low probability. Despite being trained entirely on real videos, our model obtains strong performance on the task of detecting manipulated speech videos. Project site: https://cfeng16.github.io/audio-visual-forensics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“The performance of any deepfake detection method is highly dependent on the quantitative and qualitative aspects of the datasets used for development [18, 21, 25, 40, 44, 47, 49, 51, 52, 61, 63, 69, 72, 75].”
From AV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset (2024) · cites this paper as [18]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Existing surveys do not examine how detection methods withstand adversarial attacks designed to evade detection [17, 21] or postprocessing operations that degrade detection performance [22-24].”
From Deepfake detection across image, video, and audio: a comprehensive survey with empirical evaluation of generalization and robustness (2026) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 6×
“To detect anomalies, we fit an autoregressive generative model [84, 99] to sequences of synchronization features extracted from real videos, and identify low probability examples.”
MethodInfluentialAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
(2014) · Cited by 33,419
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Labaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,119
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309
The Kinetics Human Action Video DatasetPreprint
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 5,004
The generalized correlation method for estimation of time delay
Knapp, C., & Carter, G. (1976) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1976.1162830 · Cited by 4,433
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · cited 8×
“U ns up er vi se d AVBYOL [38, 40] AV LRW [22] 50.”
BackgroundMethodInfluentialA Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,451
Return of the Devil in the Details: Delving Deep into Convolutional Nets
Chatfield, K., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · Proceedings of the British Machine Vision Conference 2014 · doi:10.5244/c.28.6 · Cited by 3,335
FaceForensics++: Learning to Detect Manipulated Facial Images
Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., & Niessner, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00009 · Cited by 3,314
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729 · cited 1×
“We first consider a simple model that uses discrete time delay as our features, following the success of autoregressive models for fitting discrete data [32, 86, 97].”
MethodUnsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery
Schlegl, T., Seeböck, P., Waldstein, S. M., Schmidt-Erfurth, U., & Langs, G. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-59050-9_12 · Cited by 2,627
Taming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 1×
“To provide a straightforward comparison with the time delay distribution model, we reduce the dimensionality of the features by projecting them onto the top 2τ + 1 principal components, following other work in autoregressive models of features [85].”
MethodCan Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
Hara, K., Kataoka, H., & Satoh, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00685 · Cited by 2,346
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
Conditional Image Generation with PixelCNN DecodersPreprint
van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L., Graves, A., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05328 · Cited by 2,032
Pixel Recurrent Neural NetworksPreprint
van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.06759 · Cited by 1,793
The kinetics human action video dataset
Kay, W. (2024) · TIB Data Manager · doi:10.57702/jtuxnt5m · Cited by 1,646
Bulat, A., & Tzimiropoulos, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.116 · Cited by 1,464 · cited 1×
“We extract faces by using face detection [107] and alignment [15].”
MethodEnhancing The Reliability of Out-of-distribution Image Detection in Neural NetworksPreprint
Liang, S., Li, Y., & Srikant, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02690 · Cited by 1,463
Face X-Ray for More General Face Forgery Detection
Li, L., Bao, J., Zhang, T., Yang, H., Chen, D., Wen, F., & Guo, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00505 · Cited by 1,211
CNN-Generated Images Are Surprisingly Easy to Spot… for Now
Wang, S.-Y., Wang, O., Zhang, R., Owens, A., & Efros, A. A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00872 · Cited by 1,210
Self-citationThinking in Frequency: Face Forgery Detection by Mining Frequency-Aware Clues
Qian, Y., Yin, G., Sheng, L., Chen, Z., & Shao, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58610-2_6 · Cited by 1,058
Self-citationDeep Anomaly Detection with Outlier Exposure
Hendrycks, D., Mazeika, M., & Dietterich, T. (2018) · arXiv · doi:10.48550/arxiv.1812.04606 · Cited by 932 · cited 1×
“Other work addresses goals similar to anomaly detection by creating methods that model uncertainty [64] or that leverage outlier exposure [30, 44, 88].”
BackgroundA Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897
Deep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801 · cited 5×
“Second, we use a generative model VQGAN [32], trained on LRS2 [3], for anomaly detection.”
MethodInfluentialExposing DeepFake Videos By Detecting Face Warping Artifacts
Li, Y., & Lyu, S. (2018) · arXiv · doi:10.48550/arxiv.1811.00656 · Cited by 785
Generating Wikipedia by Summarizing Long SequencesPreprint
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, Ł., & Shazeer, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10198 · Cited by 784
Adversarially Learned One-Class Classifier for Novelty Detection
Sabokrou, M., Khalooei, M., Fathy, M., & Adeli, E. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00356 · Cited by 765
Generating diverse high-fidelity images with vq-vae-2
Razavi, A., Oord, A. V. D., & Vinyals, O. (2019) · arXiv · doi:10.48550/arxiv.1906.00446 · Cited by 709
FSGAN: Subject Agnostic Face Swapping and Reenactment
Nirkin, Y., Keller, Y., & Hassner, T. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00728 · Cited by 706
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
Owens, A., & Efros, A. A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_39 · Cited by 704
Self-citationWu, Y., AbdAlmageed, W., & Natarajan, P. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00977 · Cited by 702
Out of Time: Automated Lip Sync in the Wild
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54427-4_19 · Cited by 689 · cited 4×
“Unfortunately, videos in the wild are often “naturally” misaligned due to errors in encoding or recording, such as by having a single, consistent shift by a few frames [2, 23].”
BackgroundMethodInfluentialS^3FD: Single Shot Scale-Invariant Face Detector
Zhang, S., Zhu, X., Lei, Z., Shi, H., Wang, X., & Li, S. Z. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.30 · Cited by 675
Return of the Devil in the Details: Delving Deep into Convolutional NetsPreprint
Chatfield, K., Simonyan, K., Vedaldi, A., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.3531 · Cited by 663
DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection
Jiang, L., Li, R., Wu, W., Qian, C., & Loy, C. C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00296 · Cited by 599
First Order Motion Model for Image Animation
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., & Sebe, N. (2020) · arXiv · doi:10.48550/arxiv.2003.00196 · Cited by 585
Salimans, T., Karpathy, A., Chen, X., & Kingma, D. P. (2017) · arXiv · doi:10.48550/arxiv.1701.05517 · Cited by 584
PixelDefend: Leveraging Generative Models to Understand and Defend against Adversarial Examples
Song, Y., Kim, T., Nowozin, S., Ermon, S., & Kushman, N. (2017) · arXiv · doi:10.48550/arxiv.1710.10766 · Cited by 562 · cited 1×
“There have been a variety of methods proposed for learning this distribution, such as GAN discriminators [37,56,66,81,89,91,104,113], flow-based models [106], and autoregressive models [93].”
MethodLips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection
Haliassos, A., Vougioukas, K., Petridis, S., & Pantic, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00500 · Cited by 537
Chung, J. S., & Zisserman, A. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54184-6_6 · Cited by 537 · cited 2×
“U ns up er vi se d AVBYOL [38, 40] AV LRW [22] 50.”
BackgroundTransfer learning from speaker verification to multispeaker text-to-speech synthesis
Jia, Y., Zhang, Y., Weiss, R. J., Wang, Q., Shen, J., Ren, F., Chen, Z., Nguyen, P., Pang, R., Moreno, I. L., et al. (2018) · arXiv · doi:10.48550/arxiv.1806.04558 · Cited by 519
Self-citationFast Face-Swap Using Convolutional Neural Networks
Korshunova, I., Shi, W., Dambre, J., & Theis, L. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.397 · Cited by 501
Noiseprint: A CNN-Based Camera Model Fingerprint
Cozzolino, D., & Verdoliva, L. (2020) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2019.2916364 · Cited by 499
Fighting Fake News: Image Splice Detection via Learned Self-Consistency
Huh, M., Liu, A., Owens, A., & Efros, A. A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_7 · Cited by 478
Self-citationImage Forgery Localization via Block-Grained Analysis of JPEG Artifacts
Bianchi, T., & Piva, A. (2012) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2012.2187516 · Cited by 458
Adversarially Learned Anomaly Detection
Zenati, H., Romain, M., Foo, C.-S., Lecouat, B., & Chandrasekhar, V. (2018) · Proceedings · doi:10.1109/icdm.2018.00088 · Cited by 450
Arandjelović, R., & Zisserman, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_27 · Cited by 448
Global Texture Enhancement for Fake Face Detection in the Wild
Liu, Z., Qi, X., & Torr, P. H. S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00808 · Cited by 401
Leveraging Frequency Analysis for Deep Fake Image Recognition
Frank, J., Eisenhofer, T., Schönherr, L., Fischer, A., Kolossa, D., & Holz, T. (2020) · arXiv · doi:10.48550/arxiv.2003.08685 · Cited by 390 · cited 1×
“Other work has proposed to use deep learning to inspect specific artifacts, such as blending [62], frequency domain [31, 35, 83], or texture [67].”
BackgroundDeep Semi-Supervised Anomaly Detection
Ruff, L., Vandermeulen, R. A., Görnitz, N., Binder, A., Müller, E., Müller, K.-R., & Kloft, M. (2019) · arXiv · doi:10.48550/arxiv.1906.02694 · Cited by 388 · cited 1×
“Other work addresses goals similar to anomaly detection by creating methods that model uncertainty [64] or that leverage outlier exposure [30, 44, 88].”
BackgroundLearning Self-Consistency for Deepfake Detection
Zhao, T., Xu, X., Xu, M., Ding, H., Xiong, Y., & Xia, W. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01475 · Cited by 383
Cooperative Learning of Audio and Video Models from Self-Supervised SynchronizationPreprint
Korbar, B., Tran, D., & Torresani, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00230 · Cited by 376 · cited 1×
“Other work studies temporal synchronization between audio and visual signals to learn audio-visual features [24, 60, 79], which can be used for active speaker detection [8, 59, 96], source separation [37, 71, 112], lip reading [5, 70, 73] and so on.”
BackgroundScaling Autoregressive Models for Content-Rich Text-to-Image Generation
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.10789 · Cited by 371
What Makes Fake Images Detectable? Understanding Properties that Generalize
Chai, L., Bau, D., Lim, S.-N., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58574-7_7 · Cited by 357
Generative Adversarial Active Learning for Unsupervised Outlier Detection
Liu, Y., Li, Z., Zhou, C., Jiang, Y., Sun, J., Wang, M., & He, X. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2905606 · Cited by 338
Exploring Temporal Coherence for More General Video Face Forgery Detection
Zheng, Y., Bao, J., Chen, D., Zeng, M., & Wen, F. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01477 · Cited by 320
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413570 · Cited by 319
Lipreading Using Temporal Convolutional Networks
Martínez, B., Ma, P., Petridis, S., & Pantić, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053841 · Cited by 293
LRS3-TED: a large-scale dataset for visual speech recognitionPreprint
Afouras, T., Chung, J. S., & Zisserman, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00496 · Cited by 263 · cited 4×
“We train our models on Lip Reading Sentences 2 (LRS2, 97k videos) [3] and Lip Reading Sentences 3 (LRS3, 120k videos) [4].”
MethodInfluentialGenerative Probabilistic Novelty Detection with Adversarial Autoencoders
Pidhorskyi, S., Almohsen, R., Adjeroh, D. A., & Doretto, G. (2018) · arXiv · doi:10.48550/arxiv.1807.02588 · Cited by 255 · cited 2×
“There have been a variety of methods proposed for learning this distribution, such as GAN discriminators [37,56,66,81,89,91,104,113], flow-based models [106], and autoregressive models [93].”
BackgroundMethodReducing Network AgnostophobiaPreprint
Dhamija, A. R., Günther, M., & Boult, T. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.04110 · Cited by 251
Detecting Deep-Fake Videos from Phoneme-Viseme Mismatches
Agarwal, S., Farid, H., Fried, O., & Agrawala, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00338 · Cited by 234
Audio-Visual Instance Discrimination with Cross-Modal Agreement
Morgado, P. M., Vasconcelos, N. M., & Misra, I. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01229 · Cited by 227
OpenGAN: Open-Set Recognition via Open Data Generation
Kong, S., & Ramanan, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00085 · Cited by 225
Not made for each other- Audio-Visual Dissonance-based Deepfake Detection and Localization
Chugh, K., Gupta, P., Dhall, A., & Subramanian, R. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413700 · Cited by 225
Tampering Detection and Localization Through Clustering of Camera-Based CNN Features
Bondi, L., Lameri, S., Güera, D., Bestagini, P., Delp, E. J., & Tubaro, S. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.232 · Cited by 222 · cited 1×
“Some work [13, 26, 28, 47, 50, 53, 79] has used specialpurpose anomaly detection methods for image/video forensics.”
MethodSelf-supervised Learning of Audio-Visual Objects from Video
Afouras, T., Owens, A., Chung, J. S., & Zisserman, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_13 · Cited by 221
Self-citationJoint Audio-Visual Deepfake Detection
Zhou, Y., & Lim, S.-N. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01453 · Cited by 217
OC-FakeDect: Classifying Deepfakes Using One-class Variational Autoencoder
Khalid, H., & Woo, S. S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00336 · Cited by 197 · cited 1×
“Some work [13, 26, 28, 47, 50, 53, 79] has used specialpurpose anomaly detection methods for image/video forensics.”
MethodVisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
Gao, R., & Grauman, K. L. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01524 · Cited by 193
Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection
Haliassos, A., Mira, R., Petridis, S., & Pantic, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01453 · Cited by 180
Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., & Li, H. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475587 · Cited by 176
Zhou, T., Wang, W., Liang, Z., & Shen, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00572 · Cited by 156
Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., & Li, H. (2021) · arXiv · doi:10.48550/arxiv.2107.06592 · Cited by 156
Towards Automatic Face-to-Face Translation
R, P. K., Mukhopadhyay, R., Philip, J., Jha, A., Namboodiri, V. P., & Jawahar, C. V. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351066 · Cited by 151
Hybrid Models for Open Set Recognition
Zhang, H., Li, A., Guo, J. J., & Guo, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_7 · Cited by 151
Audio-driven Talking Face Video Generation with Learning-based Personalized Head PosePreprint
Yi, R., Ye, Z., Zhang, J., Bao, H., & Liu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10137 · Cited by 147
KoDF: A Large-scale Korean DeepFake Detection Dataset
Kwon, P., You, J., Nam, G., Park, S., & Chae, G. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01057 · Cited by 145
Single-image splicing localization through autoencoder-based anomaly detection
Cozzolino, D., & Verdoliva, L. (2016) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs.2016.7823921 · Cited by 122
FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset
Khalid, H., Tariq, S., Kim, M., & Woo, S. S. (2021) · arXiv · doi:10.48550/arxiv.2108.05080 · Cited by 116 · cited 10×
“Thus, following the standard procedure used in [40, 41, 109, 112], we conduct the experiment to evaluate the cross-manipulation generalization ability of our model on the FakeAVCeleb dataset [52] which videos are manipulated in various ways.”
BackgroundMethodInfluentialTowards Practical Lipreading with Distilled and Efficient Models
Ma, P., Martínez, B., Petridis, S., & Pantić, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9415063 · Cited by 115
Autoencoder with recurrent neural networks for video forgery detection
D'Avino, D., Cozzolino, D., Poggi, G., & Verdoliva, L. (2017) · Electronic Imaging · doi:10.2352/issn.2470-1173.2017.7.mwsf-330 · Cited by 109
Perfect Match: Improved Cross-modal Embeddings for Audio-visual Synchronisation
Chung, S.-W., Chung, J. S., & Kang, H.-G. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682524 · Cited by 108
DeepFaceLab: A simple, flexible and extensible face swapping framework.
Perov, I., Gao, D., Chervoniy, N., Liu, K., Marangonda, S., Umé, C., Dpfks, Facenheim, C. S., Rp, L., Jiang, J., et al. (2020) · Cited by 106
Audio-Visual Person-of-Interest DeepFake Detection
Cozzolino, D., Pianese, A., Nießner, M., & Verdoliva, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00101 · Cited by 104 · cited 3×
“First, we consider an audio-visual contrastive learning model, which we call AVBYOL, that learns to determine whether the visual and audio streams of a video do (or do not) match, an approach that has been used as a part of other audio-visual forensics models [25, 40].”
MethodResultEyes Tell All: Irregular Pupil Shapes Reveal GAN-Generated Faces
Hui, G., Hu, S., Wang, X., Chang, M., & Lyu, S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746597 · Cited by 102
Labelling unlabelled videos from scratch with multi-modal self-supervision
Asano, Y. M., Patrick, M., Rupprecht, C., & Vedaldi, A. (2020) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.2006.13662 · Cited by 94 · cited 1×
“between frames and audio tracks [10, 75, 105] to learn multimodal features and applied them to downstream tasks such as sound localization [9, 45, 74].”
BackgroundLearning Second Order Local Anomaly for General Face Forgery Detection
Fei, J., Dai, Y., Yu, P., Shen, T., Xia, Z., & Weng, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01963 · Cited by 88
Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation
Zhou, H., Xu, X., Lin, D., Wang, X., & Liu, Z. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58610-2_4 · Cited by 80
Localizing Visual Sounds the Easy Way
Mo, S., & Morgado, P. M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_13 · Cited by 74
Audio forensics from acoustic reverberation
Malik, H. M. A., & Farid, H. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5495479 · Cited by 68
Deep Anomaly Detection for Generalized Face Anti-Spoofing
Pérez‐Cabo, D., Jiménez‐Cabello, D., Costa‐Pazo, A., & López-Sastre, R. J. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00201 · Cited by 68
Satellite image forgery detection and localization using GAN and One-Class classifier
Yarlagadda, S. K., Güera, D., Bestagini, P., Zhu, F. M., Tubaro, S., & Delp, E. J. (2018) · Virtual Community of Pathological Anatomy (University of Castilla La Mancha) · doi:10.2352/issn.2470-1173.2018.07.mwsf-214 · Cited by 67
MAAS: Multi-modal Assignation for Active Speaker Detection
Alcázar, J. L., Heilbron, F. C., Thabet, A., & Ghanem, B. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00033 · Cited by 57
Self-supervised object detection from audio-visual correspondence
Afouras, T., Asano, Y. M., Fagan, F., Vedaldi, A., & Metze, F. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01032 · Cited by 56
How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wild
Köpüklü, O., Taseska, M., & Rigoll, G. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00123 · Cited by 54
Mix and Localize: Localizing Sound Sources in Mixtures
Hu, X., Chen, Z., & Owens, A. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01023 · Cited by 53
Self-citationDynamic Inconsistency-aware DeepFake Video Detection
Hu, Z., Xie, H., Wang, Y., Li, J., Wang, Z., & Zhang, Y. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/102 · Cited by 51
Move2Hear: Active Audio-Visual Source Separation
Majumder, S., Al-Halah, Z., & Grauman, K. L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00034 · Cited by 40
Durall, R., Keuper, M., & Keuper, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00791 · Cited by 26
ASR is All You Need: Cross-Modal Distillation for Lip Reading
Afouras, T., Chung, J. S., & Zisserman, A. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054253 · Cited by 25
Contrastive Learning of Global and Local Video Representations
Ma, S., Zeng, Z., McDuff, D., & Song, Y. (2021) · neural information processing systems · Cited by 23
Videoforensicshq: Detecting High-Quality Manipulated Face Videos
Fox, G., Liu, W., Kim, H., Seidel, H.-P., Elgharib, M., & Theobalt, C. (2021) · 2021 IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428101 · Cited by 22
Audio-Visual Synchronisation in the wildPreprint
Chen, H., Xie, W., Afouras, T., Nagrani, A., Vedaldi, A., & Zisserman, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.04432 · Cited by 21 · cited 3×
“[18], we use ResNet-18 2D+3D [42, 43] as the visual encoder, using 5 frames (25 fps) as input.”
MethodSound Localization by Self-supervised Time Delay Estimation
Chen, Z., Fouhey, D. F., & Owens, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_28 · Cited by 12
Self-citationMAAS: Multi-modal Assignation for Active Speaker DetectionPreprint
Alcázar, J. L., Heilbron, F. C., Thabet, A., & Ghanem, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.03682 · Cited by 8
Audio-Visual Person-of-Interest DeepFake DetectionPreprint
Cozzolino, D., Pianese, A., Nießner, M., & Verdoliva, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.03083 · Cited by 3 · cited 3×
“First, we consider an audio-visual contrastive learning model, which we call AVBYOL, that learns to determine whether the visual and audio streams of a video do (or do not) match, an approach that has been used as a part of other audio-visual forensics models [25, 40].”
MethodResultDeep Audio-Visual Speech Recognition
Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2018) · arXiv · doi:10.48550/arxiv.1809.02108 · cited 5×
“Second, we use a generative model VQGAN [32], trained on LRS2 [3], for anomaly detection.”
MethodInfluentialAutoencoder with recurrent neural networks for video forgery detection
D'Avino, D., Cozzolino, D., Poggi, G., & Verdoliva, L. (2017) · arXiv · doi:10.48550/arxiv.1708.08754 · cited 1×
“Some work [13, 26, 28, 47, 50, 53, 79] has used specialpurpose anomaly detection methods for image/video forensics.”
Method
Research works printed but not linked: 4
- radford, 2019, Language models are unsupervised multitask learners, OpenAIRE blog, 1, 9
- zhang, 2017, S3fd: Single shot scale-invariant face detector, Proceedings of the IEEE International Conference on Computer Vision, 192
- 2022, Faceswap
- zong, 2018, Deep auto-encoding gaussian mixture model for unsupervised anomaly detection, International Conference on Learning Representations
Cited by
Sorted by citations
Binding Touch to Everything: Learning Unified Multimodal Tactile Representations
Yang, F., Feng, C., Chen, Z., Park, H., Wang, D., Dou, Y., Zeng, Z., Chen, X., Gangopadhyay, S., Owens, A., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02488 · Cited by 98
“Thus, we minimize the overall loss: Naturally, minimizing the contrastive objective [31, 113, 127, 145] will “pull” a visuo-tactile pair close together and “push” it away from other pairs, achieving the alignment between touch and visual embedding.”
Self-citationPVASS-MDD: Predictive Visual-Audio Alignment Self-Supervision for Multimodal Deepfake Detection
Yu, Y., Liu, X., Ni, R., Yang, S., Zhao, Y., & Kot, A. C. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3309899 · Cited by 63
“The SS Auxiliary methods [46], [47], [58] and multimodal detection method [43] are not compared in [44], thus the results of these methods are reproduced by ourselves.”
BackgroundMethodInfluentialAV-Deepfake1M: A Large-Scale LLM-Driven Audio-Visual Deepfake Dataset
Cai, Z., Ghosh, S., Adatia, A. P., Hayat, M., Dhall, A., Gedeon, T., & Stefanov, K. (2024) · Proceedings of the 32nd ACM International Conference on Multimedia · doi:10.1145/3664647.3680795 · Cited by 59
SeeABLE: Soft Discrepancies and Bounded Contrastive Learning for Exposing Deepfakes
Larue, N., Vu, N., Štruc, V., Peer, P., & Christophides, V. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01921 · Cited by 59
“Another category of models [22,27,78,80] utilize temporal features for deepfake detection.”
BackgroundLiz-López, H., Keïta, M., Ahmed, A. T., Hadid, A., Huertas‐Tato, J., & Camacho, D. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102103 · Cited by 46
Metadata
- Authors
- Feng, C., Chen, Z., & Owens, A.
- Affiliations
- University of Michigan (US)
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 10491-10503
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386076652
- Cited by
- 101
- References
- 117
- Citations by year
- 2026: 35 · 2025: 39 · 2024: 20 · 2023: 7
- FWCI
- 10.80
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- U.S. Department of Defense
- Defense Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Mathematics › Statistics and Probability › Benford’s Law and Fraud Detection
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- video forensics, self-supervised learning, audio-video synchronization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
