The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
WhiADD: Semantic-Acoustic Fusion for Robust Audio Deepfake Detection
Open accessAbstract
This paper addresses the critical challenge of detecting codec-based audio deepfakes in multilingual and dynamically evolving adversarial scenarios. While existing detection systems exhibit performance degradation against codec-generated forgeries and unseen linguistic environments, we propose a novel audio deepfake detection framework ''WhiADD'' enhanced by semantic-acoustic fusion and cross-modal generalization. Our methodology introduces three key innovations: (1) The Union CodecFake (UCF) dataset, synthesized by extending the CodecFake generation pipeline to the multilingual Common Voice corpus, significantly expands acoustic diversity with 1.9M samples across varied phonetic, channel, and codec manipulation patterns. (2) A semantic-prompted Whisper architecture that integrates full-transcript linguistic constraints into decoder fine-tuning, enabling detection of semantic inconsistencies. (3) A gated cross-attention mechanism that dynamically fuses multi-source audio features with the proposed model's frozen encoder outputs, enhancing artifact detection through adaptive attention to pre-trained representations. Extensive experiments demonstrate state-of-the-art performance, achieving 0.55% EER on UCF testing data and less than 3% EER in zero-shot cross-lingual detection (German, French, Italian). The framework reduces false negatives by up to 24% compared to conventional models through improved semantic-acoustic alignment. These advancements establish a robust paradigm fo…
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Common Voice: A Massively-Multilingual Speech Corpus
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., & Weber, G. (2019) · arXiv · doi:10.48550/arxiv.1912.06670 · Cited by 905 · cited 1×
“To address this critical challenge, we propose the Union CodecFake (UCF) dataset - an augmented training resource that extends the CodecFake synthesis methodology to the linguistically diverse Common Voice corpus [26].”
ASVspoof 2019: Future Horizons in Spoofed and Fake Audio Detection
Todisco, M., Wang, X., Vestman, V., Sahidullah, M., Delgado, H., Nautsch, A., Yamagishi, J., Evans, N., Kinnunen, T., & Lee, K. A. (2019) · arXiv · doi:10.48550/arxiv.1904.05441 · Cited by 688
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazare, P. E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9052942 · Cited by 458
ASVspoof 2021: accelerating progress in spoofed and deepfake speech detection
Yamagishi, J., Wang, X., Todisco, M., Sahidullah, M., Patino, J., Nautsch, A., Liu, X., Lee, K. A., Kinnunen, T., Evans, N., et al. (2021) · 2021 Edition of the Automatic Speaker Verification and Spoofing Countermeasures Challenge · doi:10.21437/asvspoof.2021-8 · Cited by 380
ADD 2022: the first Audio Deep Synthesis Detection Challenge
Yi, J., Fu, R., Tao, J., Nie, S., Ma, H., Wang, C., Wang, T., Tian, Z., Bai, Y., Fan, C., et al. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9746939 · Cited by 165
A comprehensive overview of Deepfake: generation, detection, datasets, and opportunities
Seow, J. W., Lim, M. K., Phan, R. C. W., & Liu, J. K. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.09.135 · Cited by 149
ASVspoof 5: crowdsourced speech data, deepfakes, and adversarial attacks at scale
Wang, X., Delgado, H., Tak, H., Jung, J.-W., Shim, H.-J., Todisco, M., Kukanov, I., Liu, X., Sahidullah, M., Kinnunen, T. H., et al. (2024) · The Automatic Speaker Verification Spoofing Countermeasures Workshop (ASVspoof 2024) · doi:10.21437/asvspoof.2024-1 · Cited by 101
Leveraging Positional-Related Local-Global Dependency for Synthetic Speech Detection
Liu, X., Liu, M., Wang, L., Lee, K. A., Zhang, H., & Dang, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096278 · Cited by 76
SafeEar: Content Privacy-Preserving Audio Deepfake Detection
Li, X., Li, K., Zheng, Y., Yan, C., Ji, X., & Xu, W. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670285 · Cited by 39 · cited 3×
“The main challenge in generating spoofing audio is replicating the characteristics of real speakers, such as timbre and prosody, while ignoring the actual speech content [19].”
Deepfake Speech Recognition and Detection
Chang, H.-C. (2023) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001423500155 · Cited by 7
Research works printed but not linked: 28
- Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023. · arXiv
- Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint arXiv:2303.03926, 2023. · arXiv
- Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. Viola: Unified codec language models for speech recognition, synthesis, and translation. arXiv preprint arXiv:2305.16107, 2023. · arXiv
- Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704, 2023. · arXiv
- Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, et al. Lauragpt: Listen, attend, understand, and regenerate audio with gpt. arXiv preprint arXiv:2310.04673, 2023. · arXiv
- Xiaofei Wang, Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujie Liu, Jinyu Li, and Takuya Yoshioka. Speechx: Neural codec language model as a versatile speech transformer. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
- Rohan Kumar Das, Xiaohai Tian, Tomi Kinnunen, and Haizhou Li. The attacker's perspective on automatic speaker verification: An overview. In Proceedings of the 21st Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 4213-4217, Shanghai, China, 2020. ISCA.
- 10.21437/Interspeech.2024-2093 · DOI
- Haibin Wu, Yuan Tseng, and Hung-yi Lee. Codecfake: Enhancing anti-spoofing models against deepfake audios from codec-based speech synthesis systems. arXiv preprint arXiv:2406.07237, 2024. · arXiv
- 10.1109/TASLPRO.2025.3525966 · DOI
- 10.21437/Interspeech.2023-1537 · DOI
- 10.1007/978-3-031-61382-1_8 · DOI
- Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492-28518. PMLR, 2023.
- 10.18653/v1/2024.emnlp-main.286 · DOI
- ZHANG Yuxiang, LI Zhuo, LU Jingze, SHANG Zengqiang, CHEN Shuli, WANG Wenchao, and ZHANG Pengyuan. Spoof speech detection based on speaker features. ACTA ACUSTICA, 50(1):201-210, 2025.
- Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2019.
- 10.21437/Interspeech.2023-2193 · DOI · Paper page
- 10.21437/Interspeech.2023-1505 · doi:10.21437/interspeech.2023-1505
- 10.21437/Interspeech.2019-2680 · DOI · Paper page
- Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449-12460, 2020.
- Youxiang Zhu, Abdelrahman Obyat, Xiaohui Liang, John A Batsis, and Robert M Roth. Wavbert: Exploiting semantic and non-semantic speech using wav2vec and bert for dementia detection. In Interspeech, volume 2021, page 3790, 2021.
- Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
- Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
- Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
- 10.21437/Interspeech.2019-1768 · DOI · Paper page
- Jee-weon Jung, Hee-Soo Heo, Hemlata Tak, Hye-jin Shim, Joon Son Chung, Bong-Jin Lee, Ha-Jin Yu, and Nicholas Evans. Aasist: Audio anti-spoofing using integrated spectro-temporal graph attention networks. In ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 6367-6371. IEEE, 2022.
- Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430, 2024. · arXiv
- Ye Jia, Michelle Tadmor Ramanovich, Quan Wang, and Heiga Zen. Cvss corpus and massively multilingual speech-to-speech translation. In Proceedings of the 13th Language Resources and Evaluation Conference (LREC), pages 6691-6703, Marseille, France, 2022. European Language Resources Association (ELRA).
Cited by
Nguyen-Le, H.-H., Tran, V.-T., Nguyen, D.-T., & Le-Khac, N.-A. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11608-4 · Cited by 28
Metadata
- Authors
- Cui, J., Yu, B., Wang, Q., Meng, F., & Lu, J.
- Affiliations
- Tsinghua University (CN)
- Guangxi University (CN)
- Venue
- Proceedings of the 33rd ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2025-10-27
- Pages
- 11610-11618
- OpenAlex
- W4415539263
- Cited by
- 1
- References
- 10
- Citations by year
- 2026: 1
- FWCI
- 0.95
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by-nc)
- Funders
- National Nature Science Foundation of China (Grant 62441616, Grant 62336004, Grant 62406172, Grant 62125603.)
- China Postdoctoral Science Foundation (No. 2023M741964)
- Postdoctoral Fellowship Program of CPSF (No. GZC20240841)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- audio deepfake detection, cross-modal generalization, gated cross-attention, audio forensics, multilingual speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.