High Fidelity Neural Audio Compression
Open accessAbstract
We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The most challenging codecs for the non-SSL based detector were codec-4 (Encodec), codec-7 (mp3+Encodec), and codec-10 (speex), with pooled min DCF of 0.675, 0.683, and 0.701 respectively.
“The most challenging codecs for our non-SSL based detectors were codec-4 (Encodec [26]), 7 (mp3+Encodec) and 10 (speex) with pooled min DCF of 0.675, 0.683 and 0.701.”
From Robust audio deepfake detection: exploring front-/back-end combinations and data augmentation strategies for the ASVspoof5 Challenge (2024) · cites this paper as [26]
Cites
Still being checked.
Cited by
Sorted by citations
Edge Computing on IoT for Machine Signal Processing and Fault Diagnosis: A Review
Lu, S., Lu, J., An, K., Wang, X., & He, Q. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3239944 · Cited by 306 · see citing works
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.02111 · Cited by 250 · see citing works
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chen, S., Wang, C., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, T., Wang, H., Li, J., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3530270 · Cited by 128 · see citing works
ASVspoof 5: crowdsourced speech data, deepfakes, and adversarial attacks at scale
Wang, X., Delgado, H., Tak, H., Jung, J.-W., Shim, H.-J., Todisco, M., Kukanov, I., Liu, X., Sahidullah, M., Kinnunen, T. H., et al. (2024) · The Automatic Speaker Verification Spoofing Countermeasures Workshop (ASVspoof 2024) · doi:10.21437/asvspoof.2024-1 · Cited by 101 · see citing works
Evans, Z., Parker, J. D., Carr, C., Zukowski, Z., Taylor, J., & Pons, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888461 · Cited by 73 · see citing works
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt
Yang, D., Liu, S., Huang, R., Weng, C., & Meng, H. M. L. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3402088 · Cited by 76 · see citing works
A Robust Audio Deepfake Detection System via Multi-View Feature
Yang, Y., Qin, H., Zhou, H., Wang, C., Guo, T., Han, K., & Wang, Y. (2024) · ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp48485.2024.10446560 · Cited by 61 · see citing works
“For audio neural codec models, we use EnCodec [12] and AudioDec [13] mainly consisted of autoencoder architecture and aimed to encode audio compactly.”
MethodAdapting Frechet Audio Distance for Generative Music Evaluation
Gui, A., Gamper, H., Braun, S., & Emmanouilidou, D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446663 · Cited by 44 · see citing works
The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audio
Xie, Y., Lu, Y., Fu, R., Wen, Z., Wang, Z., Tao, J., Qi, X., Wang, X., Liu, Y., Cheng, H., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3525966 · Cited by 51 · see citing works
APCodec: A Neural Audio Codec With Parallel Amplitude and Phase Spectrum Encoding and Decoding
Ai, Y., Jiang, X.-H., Lu, Y.-X., Du, H.-P., & Ling, Z.-H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3417347 · Cited by 42 · see citing works
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
Wang, X., Thakker, M., Chen, Z., Kanda, N., Eskimez, Ş. E., Chen, S., Tang, M., Liu, S., Li, J., & Yoshioka, T. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3419418 · Cited by 41 · see citing works
FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec
Du, Z., Zhang, S., Hu, K., & Zheng, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447523 · Cited by 40 · see citing works
SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound
Liu, H., Xu, X., Yuan, Y., Wu, M., Wang, W., & Plumbley, M. D. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3506286 · Cited by 40 · see citing works
SafeEar: Content Privacy-Preserving Audio Deepfake Detection
Li, X., Li, K., Zheng, Y., Yan, C., Ji, X., & Xu, W. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670285 · Cited by 39 · see citing works
Choi, H.-Y., Lee, S.-H., & Lee, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29740 · Cited by 39 · see citing works
Expresso: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Nguyen, T. A., Hsu, W.-N., D’Avirro, A., Shi, B., Gat, I., Fazel-Zarani, M., Remez, T., Copet, J., Synnaeve, G., Hassid, M., et al. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-1905 · Cited by 48 · see citing works
Self-citationChang, X., Yan, B., Choi, K., Jung, J.-W., Lu, Y., Maiti, S., Sharma, R., Shi, J., Tian, J., Watanabe, S., et al. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447929 · Cited by 37 · see citing works
Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation
Xue, J., Deng, Y., Gao, Y., & Li, Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3485485 · Cited by 40 · see citing works
Speaker Anonymization Using Neural Audio Codec Language Models
Panariello, M., Nespoli, F., Todisco, M., & Evans, N. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447871 · Cited by 31 · see citing works
Maiti, S., Peng, Y., Choi, S., Jung, J.-W., Chang, X., & Watanabe, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447112 · Cited by 31 · see citing works
Cheng, G., Wang, X., Cooper, E., Wells, D., Wang, L., Dang, J., Richmond, K., & Yamagishi, J. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3451951 · Cited by 31 · see citing works
Music Generation Using Deep Learning and Generative AI: A Systematic Review
Mitra, R., & Zualkernan, I. A. (2025) · IEEE Access · doi:10.1109/access.2025.3531798 · Cited by 36 · see citing works
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
Du, C., Guo, Y., Shen, F., Liu, Z., Zheng, L., Chen, X., Wang, S., Zhang, H., & Yu, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29747 · Cited by 27 · see citing works
Wang, X., Delgado, H., Tak, H., Jung, J.-W., Shim, H.-J., Todisco, M., Kukanov, I., Liu, X., Sahidullah, M., Kinnunen, T., et al. (2026) · Computer Speech & Language · doi:10.1016/j.csl.2025.101825 · Cited by 30 · see citing works
Amphion: an Open-Source Audio, Music, and Speech Generation Toolkit
Zhang, X., Xue, L., Gu, Y., Wang, Y., Li, J., He, H., Wang, C., Liu, S., Chen, X., Zhang, J. S., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832255 · Cited by 26 · see citing works
A Foundation Model for Music Informatics
Won, M., Hung, Y.-N. A., & Le, D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448314 · Cited by 25 · see citing works
VioLA: Conditional Language Models for Speech Recognition, Synthesis, and Translation
Wang, T., Zhou, L., Zhang, Z., Wu, Y., Liu, S., Gaur, Y., Chen, Z., Li, J., & Wei, F. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3434425 · Cited by 71 · see citing works
Opportunities and Challenges of Foundation Models in Industrial Manufacturing
Ayyat, M., Osman, M., & Nadeem, T. (2025) · IEEE Access · doi:10.1109/access.2025.3589777 · Cited by 25 · see citing works
TextrolSpeech: A Text Style Control Speech Corpus with Codec Language Text-to-Speech Models
Ji, S., Zuo, J., Fang, M., Jiang, Z., Chen, F., Duan, X., Huai, B., & Zhao, Z. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10445879 · Cited by 24 · see citing works
Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS
Yang, Y., Shen, F., Du, C., Ma, Z., Yu, K., Povey, D., & Chen, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447751 · Cited by 24 · see citing works
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2311.07919 · Cited by 61 · see citing works
“…unify data format and tasks by directly feeding speech representation (Nachmani et al., 2023) or encoding continuous speech signals as discrete codes (Défossez et al., 2022; Zeghidour et al., 2022; Zhang et al., 2023c), and treating different human speech tasks as conditional generative tasks.”
BackgroundAudio Super-Resolution With Robust Speech Representation Learning of Masked Autoencoder
Kim, S.-B., Lee, S.-H., Choi, H.-Y., & Lee, S. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3349053 · Cited by 23 · see citing works
Latent-Domain Predictive Neural Speech Coding
Jiang, X., Peng, X., Xue, H., Zhang, Y., & Lu, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3277693 · Cited by 23 · see citing works
Textless Unit-to-Unit Training for Many-to-Many Multilingual Speech-to-Speech Translation
Kim, M., Choi, J., Kim, D., & Ro, Y. M. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3444470 · Cited by 30 · see citing works
TellMeTalk: Multimodal-driven talking face video generation
Li, P., Zhao, H., Liu, Q., Tang, P., & Zhang, L. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2023.109049 · Cited by 22 · see citing works
STEMGEN: A Music Generation Model That Listens
Parker, J. D., Spijkervet, J., Kosta, K., Yesiler, F., Кузнецов, Б. Н., Wang, J.-C., Avent, M., Chen, J., & Le, D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446088 · Cited by 22 · see citing works
Jiang, X., Li, Y. A., Florea, A. N., Han, C., & Mesgarani, N. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889391 · Cited by 27 · see citing works
AV-RIR: Audio-Visual Room Impulse Response Estimation
Ratnarajah, A. J., Ghosh, S., Kumar, S., Chiniya, P., & Manocha, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02565 · Cited by 20 · see citing works
Hwang, J.-S., Lee, S.-H., & Lee, S.-W. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106762 · Cited by 20 · see citing works
EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning
Kim, J., Jung, J., Lee, J., & Woo, S. H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446672 · Cited by 21 · see citing works
Fewer-Token Neural Speech Codec with Time-Invariant Codes
Ren, Y. S., Wang, T., Yi, J., Xu, L., Tao, J., Zhang, C. Y., & Zhou, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448454 · Cited by 19 · see citing works
Anomalous Sound Detection by Feature-Level Anomaly Simulation
Zavrtanik, V., Marolt, M., Kristan, M., & Skočaj, D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447941 · Cited by 19 · see citing works
V2Meow: Meowing to the Visual Beat via Video-to-Music Generation
Su, K., Li, J. Y., Huang, Q., Kuzmin, D., Lee, J., Donahue, C., Sha, F., Jansen, A., Wang, Y., Verzetti, M., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28299 · Cited by 19 · see citing works
JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models
Li, P. P., Chen, B., Yao, Y., Wang, Y., Wang, A., & Wang, A. Y. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00146 · Cited by 23 · see citing works
Foleygen: Visually-Guided Audio Generation
Mei, X., Nagaraja, V., Lan, G. L., Ni, Z., Chang, E., Shi, Y., & Chandra, V. (2024) · Machine learning for signal processing ... · doi:10.1109/mlsp58920.2024.10734721 · Cited by 19 · see citing works
Translatotron 3: Speech to Speech Translation with Monolingual Data
Nachmani, E., Levkovitch, A., Ding, Y., Asawaroengchai, C., Zen, H., & Ramanovich, M. T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448426 · Cited by 18 · see citing works
Voice user interfaces for effortless navigation in medical virtual reality environments
Hombeck, J., Voigt, H., & Lawonn, K. (2024) · Computers & Graphics · doi:10.1016/j.cag.2024.104069 · Cited by 19 · see citing works
HILCodec: High-Fidelity and Lightweight Neural Audio Codec
Ahn, S., Woo, B. J., Han, M. H., Moon, C., & Kim, N. S. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3469530 · Cited by 18 · see citing works
VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech
Du, C., Guo, Y., Wang, H., Yang, Y., Niu, Z., Wang, S., Zhang, H., Chen, X., & Yu, K. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890943 · Cited by 18 · see citing works
Jiang, X.-H., Ai, Y., Zheng, R.-C., Du, H.-P., Lu, Y.-X., & Ling, Z.-H. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832214 · Cited by 16 · see citing works
A comprehensive survey with critical analysis for deepfake speech detection
Pham, L. D., Lam, P., Tran, D., Tang, H., Nguyen, T., Schindler, A., Skopik, F., Polonsky, A., & Vu, H. C. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100757 · Cited by 23 · see citing works
Yet Another Generative Model for Room Impulse Response Estimation
Lee, S., Choi, H.-S., & Lee, K. (2023) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa58266.2023.10248189 · Cited by 15 · see citing works
Algorithmic Ventriloquism: The Contested State of Voice in AI Speech Generators
Ramati, I. (2024) · Social Media + Society · doi:10.1177/20563051231224401 · Cited by 15 · see citing works
Multi-Scale Sub-Band Constant-Q Transform Discriminator for High-Fidelity Vocoder
Gu, Y., Zhang, X., Xue, L., & Wu, Z. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448436 · Cited by 15 · see citing works
AnyEnhance: A Unified Generative Model With Prompt-Guidance and Self-Critic for Voice Enhancement
Zhang, J. S., Yang, J., Fang, Z., Wang, Y., Zhang, Z., Wang, Z., Fan, F., & Wu, Z. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3587393 · Cited by 14 · see citing works
Puvvada, K. C., Koluguri, N. R., Dhawan, K., Balam, J., & Ginsburg, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446998 · Cited by 15 · see citing works
Maskmark: Robust Neuralwatermarking for Real and Synthetic Speech
O’Reilly, P., Jin, Z., Su, J., & Pardo, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447253 · Cited by 14 · see citing works
Configurable EBEN: Extreme Bandwidth Extension Network to Enhance Body-Conducted Speech Capture
Hauret, J., Joubaud, T., Zimpfer, V., & Bavu, É. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3313433 · Cited by 13 · see citing works
SNR-Adaptive Multi-Layer Semantic Communication for Speech
Guo, J., Zhang, Y., Liu, C., Xu, W., & Bie, Z. (2023) · IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC) · doi:10.1109/pimrc56721.2023.10293992 · Cited by 13 · see citing works
Qiang, C., Li, H., Ni, H., Qu, H., Fu, R., Wang, T., Wang, L., & Dang, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446203 · Cited by 13 · see citing works
ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs For Audio, Music, and Speech
Shi, J., Tian, J., Wu, Y., Jung, J.-W., Yip, J. Q., Masuyama, Y., Chen, W., Wu, Y., Tang, Y., Baali, M., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832289 · Cited by 18 · see citing works
Zheng, R.-C., Du, H.-P., Jiang, X.-H., Ai, Y., & Ling, Z.-H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3579310 · Cited by 13 · see citing works
Low-Bitrate High-Quality Digital Semantic Communication Based on RVQGAN
Chen, X., Wang, J., Huang, J., Zeng, M., Zheng, Z., & Fei, Z. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3534462 · Cited by 12 · see citing works
Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR
Hu, Y., Chen, C., Zhu, Q., & Chng, E. S. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3332545 · Cited by 12 · see citing works
Coding Speech Through Vocal Tract Kinematics
Cho, C. J., Wu, P., Prabhune, T., Agarwal, D., & Anumanchipalli, G. K. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3497655 · Cited by 13 · see citing works
Boosting Large Language Model for Speech Synthesis: An Empirical Study
Hao, H., Zhou, L., Liu, S., Li, J., Hu, S., Wang, R., & Wei, F. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890588 · Cited by 12 · see citing works
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17 · see citing works
“Enodec [118] is employed to encode audio into a continuous tensor.”
MethodCornell, S., Wang, Z.-Q., Masuyama, Y., Watanabe, S., Pariente, M., Ono, N., & Squartini, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095961 · Cited by 11 · see citing works
Generative De-Quantization for Neural Speech Codec Via Latent Diffusion
Yang, H., Jang, I., & Kim, M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446556 · Cited by 11 · see citing works
High-Fidelity Neural Phonetic Posteriorgrams
Churchwell, C., Morrison, M., & Pardo, B. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10669905 · Cited by 11 · see citing works
Vec-Tok Speech: Speech Vectorization and Tokenization for Neural Speech Generation
Zhu, X., Lv, Y., Lei, Y., Li, T., He, W., Zhou, H., Lu, H., & Xie, L. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3546559 · Cited by 11 · see citing works
LightCodec: A High Fidelity Neural Audio Codec with Low Computation Complexity
Xu, L., Wang, J., Zhang, J., & Xie, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447532 · Cited by 11 · see citing works
Midi-Voice: Expressive Zero-Shot Singing Voice Synthesis via Midi-Driven Priors
Byun, D.-M., Lee, S.-H., Hwang, J.-S., & Lee, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447981 · Cited by 10 · see citing works
Yu, G., Zheng, X., Li, N., Han, R., Zheng, C., Zhang, C., Zhou, C., Huang, Q., & Yu, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446439 · Cited by 10 · see citing works
Lemos, M., Cardoso, P. J. S., & Rodrigues, J. M. F. (2024) · Multimodal Technologies and Interaction · doi:10.3390/mti8070056 · Cited by 3 · see citing works
AI-Generated Music Detection and its Challenges
Afchar, D., Meseguer-Brocal, G., & Hennequin, R. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890655 · Cited by 14 · see citing works
Learning Speech Representation from Contrastive Token-Acoustic Pretraining
Qiang, C., Li, H., Tian, Y., Fu, R., Wang, T., Wang, L., & Dang, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447797 · Cited by 10 · see citing works
SuperCodec: A Neural Speech Codec with Selective Back-Projection Network
Zheng, Y., Tu, W., Xiao, L., & Xu, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447744 · Cited by 9 · see citing works
Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment
Stahl, B., & Gamper, H. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888007 · Cited by 9 · see citing works
Learning Source Disentanglement in Neural Audio Codec
Bie, X., Liu, X., & Gaël Richard, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888065 · Cited by 9 · see citing works
Speech Enhancement Using Continuous Embeddings of Neural Audio Codec
Li, H., Yip, J. Q., Fan, T., & Chng, E. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890379 · Cited by 9 · see citing works
Schäfer, K., Choi, J.-E., & Neu, M. (2024) · The Automatic Speaker Verification Spoofing Countermeasures Workshop (ASVspoof 2024) · doi:10.21437/asvspoof.2024-9 · Cited by 8 · see citing works
“The most challenging codecs for our non-SSL based detectors were codec-4 (Encodec [26]), 7 (mp3+Encodec) and 10 (speex) with pooled min DCF of 0.675, 0.683 and 0.701.”
InfluentialCOCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representations
Ciranni, R., Mariani, G., Mancusi, M., Postolache, E., Fabbro, G., Rodolà, E., & Cosmo, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888146 · Cited by 8 · see citing works
GAN-Based Speech Enhancement for Low SNR Using Latent Feature Conditioning
Shetu, S. S., Habets, E. A. P., & Brendel, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890549 · Cited by 8 · see citing works
Salmon: A Suite for Acoustic Language Model Evaluation
Maimon, G., Roth, A., & Adi, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888561 · Cited by 8 · see citing works
Self-citationSLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs
Chen, W., Ma, Z., Li, X., Xu, X., Liang, Y., Zheng, Z., Yu, K., & Chen, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889071 · Cited by 9 · see citing works
Generation-Based Target Speech Extraction with Speech Discretization and Vocoder
Yu, L., Zhang, W., Du, C., Zhang, L., Zheng, L., & Qian, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446418 · Cited by 8 · see citing works
Codec-Superb @ SLT 2024: A Lightweight Benchmark For Neural Audio Codec Models
Wu, H., Chen, X., Lin, Y., Chang, K., Du, J., Lu, K.-H., Liu, A. H., Chung, H.-L., Wu, Y.-K., Yang, D., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832364 · Cited by 9 · see citing works
In-context Autoencoder for Context Compression in a Large Language Model
Ge, T., Hu, J., Wang, L., Wang, X., Chen, S.-Q., & Wei, F. (2023) · arXiv · doi:10.48550/arxiv.2307.06945 · Cited by 11 · see citing works
“We by default train the ICAE with bf16.”
MethodNeural Speech Coding for Real-Time Communications Using Constant Bitrate Scalar Quantization
Brendel, A., Pia, N., Gupta, K., Behringer, L., Fuchs, G., & Multrus, M. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3491575 · Cited by 7 · see citing works
Variable Bitrate Residual Vector Quantization for Audio Coding
Chae, Y., Choi, W., Takida, Y., Koo, J., Ikemiya, Y., Zhong, Z., Cheuk, K. W., Martínez-Ramírez, M. A., Lee, K., Liao, W., et al. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889508 · Cited by 7 · see citing works
LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport
Rho, K., Lee, H., Iverson, V., & Chung, J. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888241 · Cited by 7 · see citing works
A comprehensive study of audio profiling: Methods, applications, challenges, and future directions
Pudasaini, A., Al-Hawawreh, M. S., Bouadjenek, M. R., Hacid, H., & Aryal, S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130334 · Cited by 9 · see citing works
ASQ: An Ultra-Low Bit Rate ASR-Oriented Speech Quantization Method
Ye, L., Gao, C., Cheng, G., Luo, L., & Zhao, Q. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3347148 · Cited by 7 · see citing works
NoiseBandNet: Controllable Time-Varying Neural Synthesis of Sound Effects Using Filterbanks
Barahona-Ríos, A., & Collins, T. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3364616 · Cited by 7 · see citing works
TTSDS - Text-to-Speech Distribution Score
Minixhofer, C., Klejch, O., & Bell, P. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832178 · Cited by 7 · see citing works
Guo, H., Xie, F., Xie, K., Yang, D., Guo, D., Wu, X., & Meng, H. M. L. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832247 · Cited by 7 · see citing works
NOLACE: Improving Low-Complexity Speech Codec Enhancement Through Adaptive Temporal Shaping
Büthe, J., Mustafa, A., Valin, J.-M., Helwani, K., & Goodwin, M. M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448332 · Cited by 7 · see citing works
Liu, Y., Chen, L.-H., & Liu, H. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.11.18.567666 · Cited by 7 · see citing works
Wu, Y.-C., Marković, D., Krenn, S., Gebru, I. D., & Richard, A. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448371 · Cited by 6 · see citing works
- Authors
- Défossez, A., Copet, J., Synnaeve, G., & Adi, Y.
- Affiliations
- Meta (Israel) (IL)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2210.13438
- OpenAlex
- W4307323391
- Cited by
- 282
- References
- 0
- Citations by year
- 2026: 4 · 2025: 112 · 2024: 145 · 2023: 21
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Data Compression Techniques
- Keywords
- audio codec, encoder-decoder architecture, loss balancing, lightweight transformer, perceptual loss, MUSHRA, stereo audio
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
