PreprintCited by 290Speech and Audio ProcessingMusic and Audio ProcessingAdvanced Data Compression Techniques

The DOI matches its DataCite record.

High Fidelity Neural Audio Compression

Open access

Défossez, Alexandre, Copet, Jade, Synnaeve, Gabriel, Adi, Yossi·2022·arXiv

DOI 10.48550/arxiv.2210.13438arXiv 2210.13438

Open this paper in CoChat — free account Open PDF

Abstract

We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2210.13438

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Still being checked.

Sorted by citations

Edge Computing on IoT for Machine Signal Processing and Fault Diagnosis: A Review
Lu, S., Lu, J., An, K., Wang, X., & He, Q. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3239944 · Cited by 306
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.02111 · Cited by 250
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chen, S., Wang, C., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, T., Wang, H., Li, J., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3530270 · Cited by 162
ASVspoof 5: crowdsourced speech data, deepfakes, and adversarial attacks at scale
Wang, X., Delgado, H., Tak, H., Jung, J.-W., Shim, H.-J., Todisco, M., Kukanov, I., Liu, X., Sahidullah, M., Kinnunen, T. H., et al. (2024) · The Automatic Speaker Verification Spoofing Countermeasures Workshop (ASVspoof 2024) · doi:10.21437/asvspoof.2024-1 · Cited by 101
Stable Audio Open
Evans, Z., Parker, J. D., Carr, C., Zukowski, Z., Taylor, J., & Pons, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888461 · Cited by 236
InstructTTS: Modelling Expressive TTS in Discrete Latent Space With Natural Language Style Prompt
Yang, D., Liu, S., Huang, R., Weng, C., & Meng, H. M. L. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3402088 · Cited by 125
A Robust Audio Deepfake Detection System via Multi-View Feature
Yang, Y., Qin, H., Zhou, H., Wang, C., Guo, T., Han, K., & Wang, Y. (2024) · ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp48485.2024.10446560 · Cited by 61
“For audio neural codec models, we use EnCodec [12] and AudioDec [13] mainly consisted of autoencoder architecture and aimed to encode audio compactly.”
Method
Adapting Frechet Audio Distance for Generative Music Evaluation
Gui, A., Gamper, H., Braun, S., & Emmanouilidou, D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446663 · Cited by 85
The Codecfake Dataset and Countermeasures for the Universally Detection of Deepfake Audio
Xie, Y., Lu, Y., Fu, R., Wen, Z., Wang, Z., Tao, J., Qi, X., Wang, X., Liu, Y., Cheng, H., et al. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3525966 · Cited by 67
APCodec: A Neural Audio Codec With Parallel Amplitude and Phase Spectrum Encoding and Decoding
Ai, Y., Jiang, X.-H., Lu, Y.-X., Du, H.-P., & Ling, Z.-H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3417347 · Cited by 42
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
Wang, X., Thakker, M., Chen, Z., Kanda, N., Eskimez, Ş. E., Chen, S., Tang, M., Liu, S., Li, J., & Yoshioka, T. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3419418 · Cited by 83
FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec
Du, Z., Zhang, S., Hu, K., & Zheng, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447523 · Cited by 62
SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound
Liu, H., Xu, X., Yuan, Y., Wu, M., Wang, W., & Plumbley, M. D. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3506286 · Cited by 75
SafeEar: Content Privacy-Preserving Audio Deepfake Detection
Li, X., Li, K., Zheng, Y., Yan, C., Ji, X., & Xu, W. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670285 · Cited by 39
DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion
Choi, H.-Y., Lee, S.-H., & Lee, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29740 · Cited by 42
Expresso: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
Nguyen, T. A., Hsu, W.-N., D’Avirro, A., Shi, B., Gat, I., Fazel-Zarani, M., Remez, T., Copet, J., Synnaeve, G., Hassid, M., et al. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-1905 · Cited by 48
Self-citation
Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study
Chang, X., Yan, B., Choi, K., Jung, J.-W., Lu, Y., Maiti, S., Sharma, R., Shi, J., Tian, J., Watanabe, S., et al. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447929 · Cited by 37
Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation
Xue, J., Deng, Y., Gao, Y., & Li, Y. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3485485 · Cited by 71
Speaker Anonymization Using Neural Audio Codec Language Models
Panariello, M., Nespoli, F., Todisco, M., & Evans, N. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447871 · Cited by 34
VoxtLM: Unified Decoder-Only Models for Consolidating Speech Recognition, Synthesis and Speech, Text Continuation Tasks
Maiti, S., Peng, Y., Choi, S., Jung, J.-W., Chang, X., & Watanabe, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447112 · Cited by 53
ZMM-TTS: Zero-Shot Multilingual and Multispeaker Speech Synthesis Conditioned on Self-Supervised Discrete Speech Representations
Cheng, G., Wang, X., Cooper, E., Wells, D., Wang, L., Dang, J., Richmond, K., & Yamagishi, J. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3451951 · Cited by 34
ASVspoof 5: Design, collection and validation of resources for spoofing, deepfake, and adversarial attack detection using crowdsourced speech
Wang, X., Delgado, H., Tak, H., Jung, J.-W., Shim, H.-J., Todisco, M., Kukanov, I., Liu, X., Sahidullah, M., Kinnunen, T., et al. (2026) · Computer Speech & Language · doi:10.1016/j.csl.2025.101825 · Cited by 30
Music Generation Using Deep Learning and Generative AI: A Systematic Review
Mitra, R., & Zualkernan, I. A. (2025) · IEEE Access · doi:10.1109/access.2025.3531798 · Cited by 41
UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
Du, C., Guo, Y., Shen, F., Liu, Z., Zheng, L., Chen, X., Wang, S., Zhang, H., & Yu, K. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29747 · Cited by 27
Amphion: an Open-Source Audio, Music, and Speech Generation Toolkit
Zhang, X., Xue, L., Gu, Y., Wang, Y., Li, J., He, H., Wang, C., Liu, S., Chen, X., Zhang, J. S., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832255 · Cited by 26
A Foundation Model for Music Informatics
Won, M., Hung, Y.-N. A., & Le, D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448314 · Cited by 39
VioLA: Conditional Language Models for Speech Recognition, Synthesis, and Translation
Wang, T., Zhou, L., Zhang, Z., Wu, Y., Liu, S., Gaur, Y., Chen, Z., Li, J., & Wei, F. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3434425 · Cited by 135
Opportunities and Challenges of Foundation Models in Industrial Manufacturing
Ayyat, M., Osman, M., & Nadeem, T. (2025) · IEEE Access · doi:10.1109/access.2025.3589777 · Cited by 25
TextrolSpeech: A Text Style Control Speech Corpus with Codec Language Text-to-Speech Models
Ji, S., Zuo, J., Fang, M., Jiang, Z., Chen, F., Duan, X., Huai, B., & Zhao, Z. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10445879 · Cited by 24
Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS
Yang, Y., Shen, F., Du, C., Ma, Z., Yu, K., Povey, D., & Chen, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447751 · Cited by 24
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2311.07919 · Cited by 61
“…unify data format and tasks by directly feeding speech representation (Nachmani et al., 2023) or encoding continuous speech signals as discrete codes (Défossez et al., 2022; Zeghidour et al., 2022; Zhang et al., 2023c), and treating different human speech tasks as conditional generative tasks.”
Background
Audio Super-Resolution With Robust Speech Representation Learning of Masked Autoencoder
Kim, S.-B., Lee, S.-H., Choi, H.-Y., & Lee, S. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3349053 · Cited by 23
Latent-Domain Predictive Neural Speech Coding
Jiang, X., Peng, X., Xue, H., Zhang, Y., & Lu, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3277693 · Cited by 23
Textless Unit-to-Unit Training for Many-to-Many Multilingual Speech-to-Speech Translation
Kim, M., Choi, J., Kim, D., & Ro, Y. M. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3444470 · Cited by 34
TellMeTalk: Multimodal-driven talking face video generation
Li, P., Zhao, H., Liu, Q., Tang, P., & Zhang, L. (2024) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2023.109049 · Cited by 22
STEMGEN: A Music Generation Model That Listens
Parker, J. D., Spijkervet, J., Kosta, K., Yesiler, F., Кузнецов, Б. Н., Wang, J.-C., Avent, M., Chen, J., & Le, D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446088 · Cited by 33
Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesis
Jiang, X., Li, Y. A., Florea, A. N., Han, C., & Mesgarani, N. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889391 · Cited by 41
AV-RIR: Audio-Visual Room Impulse Response Estimation
Ratnarajah, A. J., Ghosh, S., Kumar, S., Chiniya, P., & Manocha, D. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02565 · Cited by 28
HiddenSinger: High-quality singing voice synthesis via neural audio codec and latent diffusion models
Hwang, J.-S., Lee, S.-H., & Lee, S.-W. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106762 · Cited by 26
EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning
Kim, J., Jung, J., Lee, J., & Woo, S. H. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446672 · Cited by 21
Fewer-Token Neural Speech Codec with Time-Invariant Codes
Ren, Y. S., Wang, T., Yi, J., Xu, L., Tao, J., Zhang, C. Y., & Zhou, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448454 · Cited by 19
Anomalous Sound Detection by Feature-Level Anomaly Simulation
Zavrtanik, V., Marolt, M., Kristan, M., & Skočaj, D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447941 · Cited by 19
V2Meow: Meowing to the Visual Beat via Video-to-Music Generation
Su, K., Li, J. Y., Huang, Q., Kuzmin, D., Lee, J., Donahue, C., Sha, F., Jansen, A., Wang, Y., Verzetti, M., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i5.28299 · Cited by 36
JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models
Li, P. P., Chen, B., Yao, Y., Wang, Y., Wang, A., & Wang, A. Y. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00146 · Cited by 50
Foleygen: Visually-Guided Audio Generation
Mei, X., Nagaraja, V., Lan, G. L., Ni, Z., Chang, E., Shi, Y., & Chandra, V. (2024) · Machine learning for signal processing ... · doi:10.1109/mlsp58920.2024.10734721 · Cited by 33
Translatotron 3: Speech to Speech Translation with Monolingual Data
Nachmani, E., Levkovitch, A., Ding, Y., Asawaroengchai, C., Zen, H., & Ramanovich, M. T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448426 · Cited by 18
Voice user interfaces for effortless navigation in medical virtual reality environments
Hombeck, J., Voigt, H., & Lawonn, K. (2024) · Computers & Graphics · doi:10.1016/j.cag.2024.104069 · Cited by 22
HILCodec: High-Fidelity and Lightweight Neural Audio Codec
Ahn, S., Woo, B. J., Han, M. H., Moon, C., & Kim, N. S. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3469530 · Cited by 18
VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech
Du, C., Guo, Y., Wang, H., Yang, Y., Niu, Z., Wang, S., Zhang, H., Chen, X., & Yu, K. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890943 · Cited by 31
MDCTCodec: A Lightweight MDCT-Based Neural Audio Codec Towards High Sampling Rate and Low Bitrate Scenarios
Jiang, X.-H., Ai, Y., Zheng, R.-C., Du, H.-P., Lu, Y.-X., & Ling, Z.-H. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832214 · Cited by 16
A comprehensive survey with critical analysis for deepfake speech detection
Pham, L. D., Lam, P., Tran, D., Tang, H., Nguyen, T., Schindler, A., Skopik, F., Polonsky, A., & Vu, H. C. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100757 · Cited by 28
Yet Another Generative Model for Room Impulse Response Estimation
Lee, S., Choi, H.-S., & Lee, K. (2023) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa58266.2023.10248189 · Cited by 15
Algorithmic Ventriloquism: The Contested State of Voice in AI Speech Generators
Ramati, I. (2024) · Social Media + Society · doi:10.1177/20563051231224401 · Cited by 15
Multi-Scale Sub-Band Constant-Q Transform Discriminator for High-Fidelity Vocoder
Gu, Y., Zhang, X., Xue, L., & Wu, Z. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448436 · Cited by 17
AnyEnhance: A Unified Generative Model With Prompt-Guidance and Self-Critic for Voice Enhancement
Zhang, J. S., Yang, J., Fang, Z., Wang, Y., Zhang, Z., Wang, Z., Fan, F., & Wu, Z. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3587393 · Cited by 37
Discrete Audio Representation as an Alternative to Mel-Spectrograms for Speaker and Speech Recognition
Puvvada, K. C., Koluguri, N. R., Dhawan, K., Balam, J., & Ginsburg, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446998 · Cited by 18
Maskmark: Robust Neuralwatermarking for Real and Synthetic Speech
O’Reilly, P., Jin, Z., Su, J., & Pardo, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447253 · Cited by 14
Configurable EBEN: Extreme Bandwidth Extension Network to Enhance Body-Conducted Speech Capture
Hauret, J., Joubaud, T., Zimpfer, V., & Bavu, É. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3313433 · Cited by 13
SNR-Adaptive Multi-Layer Semantic Communication for Speech
Guo, J., Zhang, Y., Liu, C., Xu, W., & Bie, Z. (2023) · IEEE International Symposium on Personal, Indoor and Mobile Radio Communications (PIMRC) · doi:10.1109/pimrc56721.2023.10293992 · Cited by 13
Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding
Qiang, C., Li, H., Ni, H., Qu, H., Fu, R., Wang, T., Wang, L., & Dang, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446203 · Cited by 16
ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs For Audio, Music, and Speech
Shi, J., Tian, J., Wu, Y., Jung, J.-W., Yip, J. Q., Masuyama, Y., Chen, W., Wu, Y., Tang, Y., Baali, M., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832289 · Cited by 18
ERVQ: Enhanced Residual Vector Quantization With Intra-and-Inter-Codebook Optimization for Neural Audio Codecs
Zheng, R.-C., Du, H.-P., Jiang, X.-H., Ai, Y., & Ling, Z.-H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3579310 · Cited by 13
Low-Bitrate High-Quality Digital Semantic Communication Based on RVQGAN
Chen, X., Wang, J., Huang, J., Zeng, M., Zheng, Z., & Fei, Z. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3534462 · Cited by 12
Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR
Hu, Y., Chen, C., Zhu, Q., & Chng, E. S. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3332545 · Cited by 15
Coding Speech Through Vocal Tract Kinematics
Cho, C. J., Wu, P., Prabhune, T., Agarwal, D., & Anumanchipalli, G. K. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3497655 · Cited by 26
Boosting Large Language Model for Speech Synthesis: An Empirical Study
Hao, H., Zhou, L., Liu, S., Li, J., Hu, S., Wang, R., & Wei, F. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890588 · Cited by 29
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17
“Enodec [118] is employed to encode audio into a continuous tensor.”
Method
Multi-Channel Speaker Extraction with Adversarial Training: The Wavlab Submission to The Clarity ICASSP 2023 Grand Challenge
Cornell, S., Wang, Z.-Q., Masuyama, Y., Watanabe, S., Pariente, M., Ono, N., & Squartini, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095961 · Cited by 11
Generative De-Quantization for Neural Speech Codec Via Latent Diffusion
Yang, H., Jang, I., & Kim, M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446556 · Cited by 12
High-Fidelity Neural Phonetic Posteriorgrams
Churchwell, C., Morrison, M., & Pardo, B. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10669905 · Cited by 11
Vec-Tok Speech: Speech Vectorization and Tokenization for Neural Speech Generation
Zhu, X., Lv, Y., Lei, Y., Li, T., He, W., Zhou, H., Lu, H., & Xie, L. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3546559 · Cited by 17
LightCodec: A High Fidelity Neural Audio Codec with Low Computation Complexity
Xu, L., Wang, J., Zhang, J., & Xie, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447532 · Cited by 11
Midi-Voice: Expressive Zero-Shot Singing Voice Synthesis via Midi-Driven Priors
Byun, D.-M., Lee, S.-H., Hwang, J.-S., & Lee, S. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447981 · Cited by 10
BAE-Net: a Low Complexity and High Fidelity Bandwidth-Adaptive Neural Network for Speech Super-Resolution
Yu, G., Zheng, X., Li, N., Han, R., Zheng, C., Zhang, C., Zhou, C., Huang, Q., & Yu, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446439 · Cited by 10
Harnessing AI and NLP Tools for Innovating Brand Name Generation and Evaluation: A Comprehensive Review
Lemos, M., Cardoso, P. J. S., & Rodrigues, J. M. F. (2024) · Multimodal Technologies and Interaction · doi:10.3390/mti8070056 · Cited by 14
AI-Generated Music Detection and its Challenges
Afchar, D., Meseguer-Brocal, G., & Hennequin, R. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890655 · Cited by 14
Learning Speech Representation from Contrastive Token-Acoustic Pretraining
Qiang, C., Li, H., Tian, Y., Fu, R., Wang, T., Wang, L., & Dang, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447797 · Cited by 10
SuperCodec: A Neural Speech Codec with Selective Back-Projection Network
Zheng, Y., Tu, W., Xiao, L., & Xu, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447744 · Cited by 9
Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment
Stahl, B., & Gamper, H. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888007 · Cited by 20
Learning Source Disentanglement in Neural Audio Codec
Bie, X., Liu, X., & Gaël Richard, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888065 · Cited by 9
Speech Enhancement Using Continuous Embeddings of Neural Audio Codec
Li, H., Yip, J. Q., Fan, T., & Chng, E. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890379 · Cited by 9
Robust audio deepfake detection: exploring front-/back-end combinations and data augmentation strategies for the ASVspoof5 Challenge
Schäfer, K., Choi, J.-E., & Neu, M. (2024) · The Automatic Speaker Verification Spoofing Countermeasures Workshop (ASVspoof 2024) · doi:10.21437/asvspoof.2024-9 · Cited by 10
“The most challenging codecs for our non-SSL based detectors were codec-4 (Encodec [26]), 7 (mp3+Encodec) and 10 (speex) with pooled min DCF of 0.675, 0.683 and 0.701.”
Influential
COCOLA: Coherence-Oriented Contrastive Learning of Musical Audio Representations
Ciranni, R., Mariani, G., Mancusi, M., Postolache, E., Fabbro, G., Rodolà, E., & Cosmo, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888146 · Cited by 14
GAN-Based Speech Enhancement for Low SNR Using Latent Feature Conditioning
Shetu, S. S., Habets, E. A. P., & Brendel, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890549 · Cited by 8
Salmon: A Suite for Acoustic Language Model Evaluation
Maimon, G., Roth, A., & Adi, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888561 · Cited by 24
Self-citation
SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMs
Chen, W., Ma, Z., Li, X., Xu, X., Liang, Y., Zheng, Z., Yu, K., & Chen, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889071 · Cited by 9
Generation-Based Target Speech Extraction with Speech Discretization and Vocoder
Yu, L., Zhang, W., Du, C., Zhang, L., Zheng, L., & Qian, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446418 · Cited by 8
Codec-Superb @ SLT 2024: A Lightweight Benchmark For Neural Audio Codec Models
Wu, H., Chen, X., Lin, Y., Chang, K., Du, J., Lu, K.-H., Liu, A. H., Chung, H.-L., Wu, Y.-K., Yang, D., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832364 · Cited by 16
In-context Autoencoder for Context Compression in a Large Language Model
Ge, T., Hu, J., Wang, L., Wang, X., Chen, S.-Q., & Wei, F. (2023) · arXiv · doi:10.48550/arxiv.2307.06945 · Cited by 11
“We by default train the ICAE with bf16.”
Method
Neural Speech Coding for Real-Time Communications Using Constant Bitrate Scalar Quantization
Brendel, A., Pia, N., Gupta, K., Behringer, L., Fuchs, G., & Multrus, M. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3491575 · Cited by 7
Variable Bitrate Residual Vector Quantization for Audio Coding
Chae, Y., Choi, W., Takida, Y., Koo, J., Ikemiya, Y., Zhong, Z., Cheuk, K. W., Martínez-Ramírez, M. A., Lee, K., Liao, W., et al. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889508 · Cited by 7
LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport
Rho, K., Lee, H., Iverson, V., & Chung, J. S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888241 · Cited by 7
A comprehensive study of audio profiling: Methods, applications, challenges, and future directions
Pudasaini, A., Al-Hawawreh, M. S., Bouadjenek, M. R., Hacid, H., & Aryal, S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130334 · Cited by 9
ASQ: An Ultra-Low Bit Rate ASR-Oriented Speech Quantization Method
Ye, L., Gao, C., Cheng, G., Luo, L., & Zhao, Q. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3347148 · Cited by 7
NoiseBandNet: Controllable Time-Varying Neural Synthesis of Sound Effects Using Filterbanks
Barahona-Ríos, A., & Collins, T. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3364616 · Cited by 7
TTSDS - Text-to-Speech Distribution Score
Minixhofer, C., Klejch, O., & Bell, P. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832178 · Cited by 12
SoCodec: A Semantic-Ordered Multi-Stream Speech Codec For Efficient Language Model Based Text-to-Speech Synthesis
Guo, H., Xie, F., Xie, K., Yang, D., Guo, D., Wu, X., & Meng, H. M. L. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832247 · Cited by 7
NOLACE: Improving Low-Complexity Speech Codec Enhancement Through Adaptive Temporal Shaping
Büthe, J., Mustafa, A., Valin, J.-M., Helwani, K., & Goodwin, M. M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448332 · Cited by 7
Diffusion in a quantized vector space generates non-idealized protein structures and predicts conformational distributionsPreprint
Liu, Y., Chen, L.-H., & Liu, H. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.11.18.567666 · Cited by 8
ScoreDec: A Phase-Preserving High-Fidelity Audio Codec with a Generalized Score-Based Diffusion Post-Filter
Wu, Y.-C., Marković, D., Krenn, S., Gebru, I. D., & Richard, A. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448371 · Cited by 13

Authors: Défossez, A., Copet, J., Synnaeve, G., & Adi, Y.
Affiliations: Meta (Israel) (IL)
ORCID: Synnaeve, Gabriel
Adi, Yossi
Venue: arXiv
Publisher: arXiv
Published: 2022
ISSN: 2331-8422 (ISSN-L)
DOI: 10.48550/arxiv.2210.13438
arXiv: 2210.13438
OpenAlex: W4307323391
Cited by: 282
References: 0
Citations by year: 2026: 4 · 2025: 112 · 2024: 145 · 2023: 21
Type: posted content
Language: en
Open access: green
arxiv.org
Licence: arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations: arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Topics: Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Data Compression Techniques
Keywords: audio codec, encoder-decoder architecture, loss balancing, lightweight transformer, perceptual loss, MUSHRA, stereo audio

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

High Fidelity Neural Audio Compression

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks