Characterizing the Temporal Dynamics of Universal Speech Representations for Generalizable Deepfake Detection
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Existing deepfake speech detection systems lack generalizability to unseen attacks (i.e., samples generated by generative algorithms not seen during training). Recent studies have explored the use of universal speech representations to tackle this issue and have obtained inspiring results. These works, however, have focused on innovating downstream classifiers while leaving the representation itself untouched. In this study, we argue that characterizing the long-term temporal dynamics of these representations is crucial for generalizability and propose a new method to assess representation dynamics. Indeed, we show that different generative models generate similar representation dynamics patterns with our proposed method. Experiments on the ASVspoof 2019 and 2021 datasets validate the benefits of the proposed method to detect deepfakes from methods unseen during training, significantly improving on several benchmark methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Zhu et al. [130] proposed the Modulation Transformation Block (MTB) to address the absence of long-term temporal dynamics in the universal speech representations extracted by self-supervised speech models. Experimental findings revealed that the integration of MTB significantly improved WavLM’s detection performance on the ASVspoof 2021 dataset, outperforming W2V2 in overall effectiveness.”
From Audio Deepfake Detection: What Has Been Achieved and What Lies Ahead (2025) · cites this paper as [130]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926
ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection
Todisco, M., Wang, X., Vestman, V., Sahidullah, M., Delgado, H., Nautsch, A., Yamagishi, J., Evans, N., Kinnunen, T., & Lee, K. A. (2019) · arXiv · doi:10.48550/arxiv.1904.05441 · Cited by 688 · cited 6×
“To tackle these potential threats, numerous efforts have been made including the curation of DF speech datasets and challenges (e.g., ASVspoof 2019 and 2021 [3, 4]), as well as the development of DF speech detection models [3, 4].”
BackgroundMethodInfluentialASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection
Todisco, M., Wang, X., Vestman, V., Sahidullah, M., Delgado, H., Nautsch, A., Yamagishi, J., Evans, N., Kinnunen, T. H., & Lee, K. A. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2249 · Cited by 687
Temporal modulations in speech and music
Ding, N., Patel, A. D., Chen, L., Butler, H., Luo, C., & Poeppel, D. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2017.02.011 · Cited by 502
End-to-End anti-spoofing with RawNet2
Tak, H., Patino, J., Todisco, M., Nautsch, A., Evans, N., & Larcher, A. (2021) · ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp39728.2021.9414234 · Cited by 402 · cited 2×
“RawNet2 is an end-to-end network tailored specifically for anti-spoofing, comprising sinc filters, convolutional blocks, and a GRU layer [6].”
BackgroundMethodASVspoof 2021: accelerating progress in spoofed and deepfake speech detection
Yamagishi, J., Wang, X., Todisco, M., Sahidullah, M., Patino, J., Nautsch, A., Liu, X., Lee, K. A., Kinnunen, T., Evans, N., et al. (2021) · 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge · doi:10.21437/asvspoof.2021-8 · Cited by 380
ASVspoof 2021: Towards Spoofed and Deepfake Speech Detection in the Wild
Liu, X., Wang, X., Sahidullah, M., Patino, J., Delgado, H., Kinnunen, T., Todisco, M., Yamagishi, J., Evans, N., Nautsch, A., et al. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3285283 · Cited by 312 · cited 1×
“For example, high equal error rates (EER) were reported in the ASVspoof 2021 DF track [7], of which the evaluation set contains multiple unseen DF generation algorithms.”
BackgroundTak, H., Todisco, M., Wang, X., Jung, J.-W., Yamagishi, J., & Evans, N. (2022) · The Speaker and Language Recognition Workshop (Odyssey 2022) · doi:10.21437/odyssey.2022-16 · Cited by 237
Research works printed but not linked: 2
- Stock, Detecting deepfake audio can be hit and miss for humans, COSMOS.
- Ito, 2017, The lj speech dataset
Cited by
A Survey on Speech Deepfake Detection
Li, M., Ahmadiadli, Y., & Zhang, X.-P. (2025) · ACM Computing Surveys · doi:10.1145/3714458 · Cited by 101
Audio Deepfake Detection: What Has Been Achieved and What Lies Ahead
Zhang, B., Cui, H., Nguyen, V., & Whitty, M. (2025) · Sensors · doi:10.3390/s25071989 · Cited by 72
Deepfake audio detection with spectral features and ResNeXt-based architecture
Tahaoglu, G., Baracchi, D., Shullani, D., Iuliani, M., & Piva, A. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113726 · Cited by 24
Wu, Y., Hua, H., Li, Z., & Zhang, S. (2025) · Electronics · doi:10.3390/electronics14122456 · Cited by 12
Efficient Audio Deepfake Detection using WavLM with Early Exiting
Pimentel, A., Zhu, Y. M., Guimarães, H. R., & Falk, T. H. (2024) · IEEE International Workshop on Information Forensics and Security · doi:10.1109/wifs61860.2024.10810699 · Cited by 4
Self-citation
Metadata
- Authors
- Zhu, Y., Powar, S., & Falk, T. H.
- Affiliations
- Institut National de la Recherche Scientifique (CA)
- Venue
- 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW)
- Publisher
- IEEE
- Published
- 2024-04-14
- Pages
- 139-143
- OpenAlex
- W4401609584
- Cited by
- 11
- References
- 19
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 1
- FWCI
- 3.18
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- speech deepfake detection, temporal dynamics, generative adversarial networks, unseen attack generalization, anti-spoofing, ASVspoof
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Characterizing the Temporal Dynamics of Universal Speech Representations for Generalizable Deepfake Detection. https://app.cochat.ai/papers/10.1109/icasspw62465.2024.10626010
- The paper
- Zhu, Y., Powar, S., & Falk, T. H. (2024). Characterizing the Temporal Dynamics of Universal Speech Representations for Generalizable Deepfake Detection. 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW). https://doi.org/10.1109/icasspw62465.2024.10626010
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.