Towards Attention-based Contrastive Learning for Audio Spoof Detection
Open accessAbstract
Vision transformers (ViT) have made substantial progress for classification tasks in computer vision. Recently, Gong et. al. '21, introduced attention-based modeling for several audio tasks. However, relatively unexplored is the use of a ViT for audio spoof detection task. We bridge this gap and introduce ViTs for this task. A vanilla baseline built on fine-tuning the SSAST (Gong et. al. '22) audio ViT model achieves sub-optimal equal error rates (EERs). To improve performance, we propose a novel attention-based contrastive learning framework (SSAST-CL) that uses cross-attention to aid the representation learning. Experiments show that our framework successfully disentangles the bonafide and spoof classes and helps learn better classifiers for the task. With appropriate data augmentations policy, a model trained on our framework achieves competitive performance on the ASVSpoof 2021 challenge. We provide comparisons and ablation studies to justify our claim.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Goel et al. used a self-supervised audio spectrogram Transformer to develop SSAST-CL for detecting audio deepfakes; their approach integrates Siamese networks with self-attention and cross-attention mechanisms to enhance ViT feature representation.
“Goel et al. [143] used a self-supervised audio spectrogram Transformer (SSAST) [144,145], a ViT-based architecture, to develop a contrastive learning framework, SSAST-CL, for detecting audio deepfakes. Their approach integrates Siamese networks with self-attention and cross-attention mechanisms to enhance the ViT’s feature representation”
From Audio Deepfake Detection: What Has Been Achieved and What Lies Ahead (2025) · cites this paper as [143]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
Siamese Neural Networks for One-shot Image Recognition
Koch, G. E., Zemel, R. S., & Salakhutdinov, R. (2015) · Cited by 3,712 · see citing works
Supervised Contrastive Learning
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., & Krishnan, D. (2020) · arXiv · doi:10.48550/arxiv.2004.11362 · Cited by 3,569 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369 · see citing works
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
Chen, C.-F. R., Fan, Q., & Panda, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00041 · Cited by 2,242 · see citing works
AST: Audio Spectrogram Transformer
Gong, Y., Chung, Y.-A., & Glass, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-698 · Cited by 1,159 · see citing works
A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. V. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413532 · Cited by 897 · see citing works
Neural Speech Synthesis with Transformer Network
Li, N., Liu, S., Liu, Y., Zhao, S., & Liu, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016706 · Cited by 754 · see citing works
Constant Q Cepstral Coefficients: A Spoofing Countermeasure for Automatic Speaker Verification
Todisco, M., Delgado, H., & Evans, N. (2017) · Computer Speech & Language · doi:10.1016/j.csl.2017.01.001 · Cited by 424 · see citing works
Efficient emotion recognition from speech using deep learning on spectrograms
Satt, A., Rozenberg, S., & Hoory, R. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-200 · Cited by 422 · see citing works
End-to-End anti-spoofing with RawNet2
Tak, H., Patino, J., Todisco, M., Nautsch, A., Evans, N., & Larcher, A. (2021) · ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp39728.2021.9414234 · Cited by 402 · see citing works
A comparison of features for synthetic speech detection
Sahidullah, M., Kinnunen, T., & Hanilçi, C. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-472 · Cited by 382 · see citing works
ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection
Yamagishi, J., Wang, X., Todisco, M., Sahidullah, M., Patino, J., Nautsch, A., Liu, X., Lee, K. A., Kinnunen, T., Evans, N., et al. (2021) · 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge · doi:10.21437/asvspoof.2021-8 · Cited by 380 · see citing works
ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild
Liu, X., Wang, X., Sahidullah, M., Patino, J., Delgado, H., Kinnunen, T., Todisco, M., Yamagishi, J., Evans, N., Nautsch, A., et al. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3285283 · Cited by 312 · cited 1× · see citing works
“Introduction of attention mechanism into CNN models has further enhanced the single model performance for the ASVSpoof21 challenge [26], particularly those employing temporal and/or spatial attention.”
MethodVOLO: Vision Outlooker for Visual Recognition
Yuan, L., Hou, Q., Jiang, Z., Feng, J., & Yan, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3206108 · Cited by 309 · see citing works
Ssast: Self-supervised audio spectrogram transformer
Gong, Y., Lai, C.-I., Chung, Y.-A., & Glass, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21315 · Cited by 294 · see citing works
STC Antispoofing Systems for the ASVspoof2019 Challenge
Lavrentyeva, G., Novoselov, S., Andzhukaev, T., Volkova, M., Gorlanov, A., & Kozlov, A. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1768 · Cited by 256 · see citing works
The Pytorch-kaldi Speech Recognition Toolkit
Ravanelli, M., Parcollet, T., & Bengio, Y. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683713 · Cited by 252 · see citing works
Global Context Vision TransformersPreprint
Hatamizadeh, A., Yin, H., Heinrich, G., et al. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2206.09959 · Cited by 246 · see citing works
Contrastive Learning of General-Purpose Audio Representations
Saeed, A., Grangier, D., & Zeghidour, N. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413528 · Cited by 237 · see citing works
Tak, H., Todisco, M., Wang, X., Jung, J.-W., Yamagishi, J., & Evans, N. (2022) · The Speaker and Language Recognition Workshop (Odyssey 2022) · doi:10.21437/odyssey.2022-16 · Cited by 237 · see citing works
CDTrans: Cross-domain Transformer for Unsupervised Domain AdaptationPreprint
Xu, T., Chen, W., Wang, P., Wang, F., Li, H., & Jin, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06165 · Cited by 195 · see citing works
Towards End-to-End Synthetic Speech Detection
Hua, G., Teoh, A. B. J., & Zhang, H. (2021) · IEEE Signal Processing Letters · doi:10.1109/lsp.2021.3089437 · Cited by 185 · see citing works
Tak, H., Kamble, M. R., Patino, J., Todisco, M., & Evans, N. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746213 · Cited by 133 · see citing works
LEAF: A Learnable Frontend for Audio ClassificationPreprint
Zeghidour, N., Teboul, O., Quitry, F. D. C., & Tagliasacchi, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.08596 · Cited by 116 · see citing works
Data Augmenting Contrastive Learning of Speech Representations in the Time Domain
Kharitonov, E., Rivière, M., Synnaeve, G., Wolf, L., Mazaré, P.-E., Douze, M., & Emmanuel Dupoux, A. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383605 · Cited by 115 · see citing works
Global Context Vision TransformersPreprint
Hatamizadeh, A., Yin, H., Heinrich, G., Kautz, J., & Molchanov, P. (2022) · arXiv (Cornell University) · doi:10.13140/rg.2.2.10442.85444 · Cited by 99 · see citing works
Entroformer: A Transformer-based Entropy Model for Learned Image CompressionPreprint
Qian, Y., Lin, M., Sun, X., Tan, Z., & Jin, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.05492 · Cited by 96 · see citing works
STC anti-spoofing systems for the ASVspoof 2015 challenge
Novoselov, S., Kozlov, A., Lavrentyeva, G., Simonchik, K., & Shchemelinin, V. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472724 · Cited by 75 · see citing works
A Novel Learnable Dictionary Encoding Layer for End-to-End Language Identification
Cai, W., Cai, Z., Zhang, X., Wang, X., & Li, M. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462025 · Cited by 74 · see citing works
A study on data augmentation in voice anti-spoofing
Cohen, A., Rimon, I., Aflalo, E., & Permuter, H. H. (2022) · Speech Communication · doi:10.1016/j.specom.2022.04.005 · Cited by 60 · see citing works
Siamese Network with wav2vec Feature for Spoofing Speech Detection
Xie, Y., Zhang, Z., & Yang, Y. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-847 · Cited by 41 · see citing works
Generalized Spoofing Detection Inspired from Audio Generation Artifacts
Gao, Y., Vuong, T., Elyasi, M., Bharaj, G., & Singh, R. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1705 · Cited by 31 · see citing works
Self-citationPindrop Labs' Submission to the ASVspoof 2021 Challenge
Chen, T., Khoury, E., Phatak, K., & Sivaraman, G. (2021) · 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge · doi:10.21437/asvspoof.2021-14 · Cited by 30 · see citing works
Anti-Spoofing Using Transfer Learning with Variational Information Bottleneck
Eom, Y., Lee, Y., Um, J. S., & Kim, H. R. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10200 · Cited by 26 · see citing works
Tak, H., Kamble, M. R., Patino, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.04433 · Cited by 7 · see citing works
ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild
Liu, X., Wang, X., Sahidullah, M., Patino, J., Delgado, H., Kinnunen, T., Todisco, M., Yamagishi, J., Evans, N., Nautsch, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02437
Flagged first, then by citations
Research works printed but not linked: 3
- A. Nautsch, X. Wang, N. W. D. Evans, T. H. Kinnunen, V. Vestman, M. Todisco, H. Delgado, M. Sahidullah, J. Yamagishi, and K.-A. Lee, "Asvspoof 2019: Spoofing countermeasures for the detection of synthesized, converted and replayed speech," IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. 3, pp. 252-265, 2021.
- R. Hu and A. Singh, "Unit: Multimodal multitask learning with a unified transformer," in Proc. ICCV, 2021, pp. 1419-1429.
- A. Tomilov, A. F. Svishchev, M. Volkova, A. Chirkovskiy, A. S. Kondratev, and G. Lavrentyeva, "Stc antispoofing systems for the asvspoof2021 challenge," 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge, pp. 61-67, 2021.
Cited by
Sorted by citations
Audio Deepfake Detection: What Has Been Achieved and What Lies Ahead
Zhang, B., Cui, H., Nguyen, V., & Whitty, M. (2025) · Sensors · doi:10.3390/s25071989 · Cited by 72 · see citing works
Zhang, K., Hua, Z., Zhang, Y., Guo, Y., & Xiang, T. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2024.3520001 · Cited by 17 · see citing works
Detecting AI-Generated Speech Manipulation through CNN-BiLSTM Hybrid Networks
Dixit, R., Agrawal, A., Dixit, A., & Tripathi, S. (2025) · International Conference on Signal Processing, Computing and Control (ISPCC) · doi:10.1109/ispcc66872.2025.11039608 · Cited by 3 · see citing works
RPRA-ADD: Forgery Trace Enhancement-Driven Audio Deepfake Detection
Fu, R., Wang, X., Wen, Z., Tao, J., Xie, Y., Wang, Z., Qiang, C., Liu, X., Fan, C., Li, C., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3646051 · Cited by 2 · see citing works
Fullwood, B., & Monrose, F. (2025) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3733799.3762979 · Cited by 1 · see citing works
A Noval Feature via Color Quantisation for Fake Audio Detection
Wang, Z., Wang, X., Xie, Y., Fu, R., Wen, Z., Tao, J., Liu, Y., Li, G., Qi, X., Lu, Y., et al. (2024) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp63861.2024.10800257 · Cited by 1 · see citing works
Fullwood, B., & Monrose, F. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11465034
- Authors
- Goel, C., Koppisetti, S., Colman, B., Shahriyari, A., & Bharaj, G.
- ORCID
- Bharaj, Gaurav
- Venue
- INTERSPEECH 2023
- Publisher
- ISCA
- Published
- 2023-08-20
- Pages
- 2758-2762
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W4385823108
- Cited by
- 7
- References
- 39
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2
- FWCI
- 1.24
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- public-domain
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Biometric Identification and Security
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- vision transformer, cross-attention, representation learning, anti-spoofing, speaker verification, data augmentation, ASVspoof 2021
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
