Vector-Quantized Autoregressive Predictive Coding
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Autoregressive Predictive Coding (APC), as a self-supervised objective, has enjoyed success in learning representations from large amounts of unlabeled data, and the learned representations are rich for many downstream tasks.However, the connection between low self-supervised loss and strong performance in downstream tasks remains unclear.In this work, we propose Vector-Quantized Autoregressive Predictive Coding (VQ-APC), a novel model that produces quantized representations, allowing us to explicitly control the amount of information encoded in the representations.By studying a sequence of increasingly limited models, we reveal the constituents of the learned representations.In particular, we confirm the presence of information with probing tasks, while showing the absence of information with mutual information, uncovering the model's preference in preserving speech information as its capacity becomes constrained.We find that there exists a point where phonetic and speaker information are amplified to maximize a selfsupervised objective.As a byproduct, the learned codes for a particular model capacity correspond well to English phones.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Clearly, with reducing codebook sizes, the model is forced to choose what information to discard and what to encode [51]. This process can result in higher performance as the network can more efficiently learn to ignore irrelevant information (such as noise) and picks up more discriminatory information.”
From Towards Learning Discrete Representations via Self-Supervision for Wearables-Based Human Activity Recognition (2024) · cites this paper as [51]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Neural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729
wav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
Cited by
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926
SUPERB: Speech Processing Universal PERformance Benchmark
Yang, S.-W., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1775 · Cited by 681
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372
“In VQ-APC [98], quantization is used with the APC”
MethodTERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
Liu, A. T., Li, S.-W., & Lee, H.-Y. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3095662 · Cited by 312
Kostas, D., Aroca-Ouellette, S., & Rudzicz, F. (2021) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2021.653659 · Cited by 272
Distilhubert: Speech Representation Learning by Layer-Wise Distillation of Hidden-Unit Bert
Chang, H.-J., Yang, S.-W., & Lee, H.-Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747490 · Cited by 246
Audio self-supervised learning: A survey
Liu, S., Mallol-Ragolta, A., Parada‐Cabaleiro, E., Qian, K., Jing, X., Kathan, A., Hu, B., & Schuller, B. W. (2022) · Patterns · doi:10.1016/j.patter.2022.100616 · Cited by 189
Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training
Chen, S., Wu, Y., Wang, C., Chen, Z., Chen, Z., Liu, S., Wu, J., Qian, Y., Wei, F., Li, J., et al. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747077 · Cited by 85
Tsai, H.-S., Chang, H.-J., Huang, W.-C., Huang, Z., Lakhotia, K., Yang, S., Dong, S., Liu, A. T., Lai, C.-I., Shi, J., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.580 · Cited by 149
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies
Liu, A. H., Chung, Y.-A., & Glass, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-349 · Cited by 75
Self-citationZhao, J. S., & Zhang, W.-Q. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3184480 · Cited by 127
MAESTRO: Matched Speech Text Representations through Modality Matching
Chen, Z., Zhang, Y., Rosenberg, A., Ramabhadran, B., Moreno, P. J., Bapna, A., & Zen, H. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10937 · Cited by 129
Toward understanding the communication in sperm whales
Andreas, J., Beguš, G., Bronstein, M. M., Diamant, R., Delaney, D., Gero, S., Goldwasser, S., Gruber, D. F., de Haas, S., Malkin, P., et al. (2022) · iScience · doi:10.1016/j.isci.2022.104393 · Cited by 110
Speaker Normalization for Self-Supervised Speech Emotion Recognition
Gat, I., Aronowitz, H., Zhu, W., Morais, E. D. S., & Hoory, R. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747460 · Cited by 86
Investigating Self-Supervised Learning for Speech Enhancement and Separation
Huang, Z., Watanabe, S., Yang, S.-W., Garcia, L. P., & Khudanpur, S. P. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746303 · Cited by 105
SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding
Chung, Y.-A., Zhu, C., & Zeng, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.152 · Cited by 88
Self-citationDeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization
Ling, S., & Liu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06659 · Cited by 56
Ochieng, P. J. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10612-2 · Cited by 77
An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition
Chang, X., Maekaku, T., Guo, P., Shi, J. Y., Lu, Y.-J., Subramanian, A. S., Wang, T., Yang, S.-W., Tsao, Y., Lee, H.-Y., et al. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688137 · Cited by 101
SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data
Zhang, Z., Chen, S., Zhou, L., Wu, Y., Ren, S., Liu, S., Yao, Z., Gong, X., Dai, L.-R., Li, J., et al. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3379877 · Cited by 90
SUPERB: Speech processing Universal PERformance Benchmark
Yang, S.-W., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.01051 · Cited by 52
ESPnet-SLU: Advancing Spoken Language Understanding Through ESPnet
Arora, S., Dalmia, S., Denisov, P., Chang, X., Ueda, Y., Peng, Y., Zhang, Y., Kumar, S., Ganesan, K., Yan, B., et al. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747674 · Cited by 86
A Large-Scale Evaluation of Speech Foundation Models
Yang, S.-W., Chang, H.-J., Huang, Z., Liu, A. T., Lai, C.-I., Wu, H., Shi, J., Chang, X., Tsai, H.-S., Huang, W.-C., et al. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3389631 · Cited by 102
Evaluating Self-Supervised Speech Representations for Speech Emotion Recognition
Atmaja, B. T., & Sasou, A. (2022) · IEEE Access · doi:10.1109/access.2022.3225198 · Cited by 77
Beguš, G. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.03.017 · Cited by 42
Exploration of a Self-Supervised Speech Model: A Study on Emotional Corpora
Li, Y., Mohamied, Y., Bell, P., & Lai, C. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023428 · Cited by 73
Similarity Analysis of Self-Supervised Speech Representations
Chung, Y.-A., Belinkov, Y., & Glass, J. R. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414321 · Cited by 56
Self-citationKamper, H., & van Niekerk, B. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-50 · Cited by 47
On the Utility of Self-Supervised Models for Prosody-Related Tasks
Lin, G.-T., Feng, C.-L., Huang, W., Tseng, Y., Lin, T.-H., Li, C.-A., Lee, H.-Y., & Ward, N. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023234 · Cited by 77
Lian, J., Baevski, A., Hsu, W.-N., & Auli, M. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389642 · Cited by 58
Applying machine learning to primate bioacoustics: Review and perspectives
Cauzinille, J., Favre, B., Marxer, R., & Rey, A. (2024) · American Journal of Primatology · doi:10.1002/ajp.23666 · Cited by 29
Feng, T.-H., Dong, A., Yeh, C.-F., Yang, S.-W., Lin, T.-Q., Shi, J., Chang, K., Huang, Z., Wu, H., Chang, X., et al. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022770 · Cited by 41
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model
Shih, Y.-J., Wang, H.-F., Chang, H.-J., Berry, L., Lee, H.-Y., & Harwath, D. F. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022954 · Cited by 54
A Comparative Study of Self-Supervised Speech Representation Based Voice Conversion
Huang, W.-C., Yang, S.-W., Hayashi, T., & Toda, T. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3193761 · Cited by 33
Autoregressive Predictive Coding: A Comprehensive Study
Yang, G.-P., Yeh, S.-L., Chung, Y.-A., Glass, J., & Tang, H. (2022) · IEEE Journal on Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3203608 · Cited by 26
Self-citationSelf-Supervised Pre-Training for Attention-Based Encoder-Decoder ASR Model
Gao, C., Cheng, G., Ta, L., Zhang, P., & Yan, Y. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3171967 · Cited by 18
Multi-view Self-supervised Learning and Multi-scale Feature Fusion for Automatic Speech Recognition
Zhao, J., Li, R., Tian, M., & An, W. (2024) · Neural Processing Letters · doi:10.1007/s11063-024-11614-z · Cited by 19
Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
Oota, S. R., Chen, Z., Gupta, M., Bapi, R. S., Jobard, G., Alexandre, F., & Hinaut, X. (2025) · Transactions on machine learning research · doi:10.48550/arxiv.2307.10246 · Cited by 43
Peng, Y., Arora, S., Higuchi, Y., Ueda, Y., Kumar, S., Ganesan, K., Dalmia, S., Chang, X., & Watanabe, S. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022399 · Cited by 33
Open-Emotion: A Reproducible EMO-Superb For Speech Emotion Recognition Systems
Wu, H., Chou, H.-C., Chang, K.-W., Goncalves, L., Du, J., Jang, J. R., Lee, C.-C., & Lee, H.-Y. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832296 · Cited by 17
MelHuBERT: A Simplified Hubert on Mel Spectrograms
Lin, T.-Q., Lee, H.-Y., & Tang, H. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389700 · Cited by 30
Self-citationDon't Speak Too Fast: The Impact of Data Bias on Self-Supervised Speech Models
Meng, Y., Chou, Y.-H., Liu, A. T., & Lee, H.-Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747897 · Cited by 41
Haresamudram, H., Essa, I., & Plötz, T. (2024) · Sensors · doi:10.3390/s24041238 · Cited by 14
Beguš, G. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101244 · Cited by 15
Self-Supervised Learning With Segmental Masking for Speech Representation
Yue, X., Lin, J., Ritter-Gutierrez, F., & Li, H. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3191845 · Cited by 13
Speech Separation With Pretrained Frontend to Minimize Domain Mismatch
Wang, W., Pan, Z., Li, X., Wang, S., & Li, H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3446242 · Cited by 18
Scaling Laws for Acoustic Models
Droppo, J., & Elibol, O. H. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1644 · Cited by 40
Interpreting Intermediate Convolutional Layers of Generative CNNs Trained on Waveforms
Beguš, G., & Zhou, A. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3209938 · Cited by 18
Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation ModelsPreprint
Tseng, L.-H., Fu, Y.-K., Chang, H.-J., & Lee, H.-Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03504 · Cited by 21
Fatehi, K., Torres, M. T., & Küçükyılmaz, A. (2024) · Speech Communication · doi:10.1016/j.specom.2024.103151 · Cited by 17
Can Self-Supervised Neural Representations Pre-Trained on Human Speech distinguish Animal Callers?
Sarkar, E., & Magimai.-Doss, M. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-1968 · Cited by 26
A Brief Overview of Unsupervised Neural Speech Representation LearningPreprint
Borgholt, L., Havtorn, J. D., Edin, J., Maaløe, L., & Igel, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.01829 · Cited by 16
Oota, S. R., Pahwa, K., Marreddy, M., Gupta, M., & Bapi, R. S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096248 · Cited by 18
Self-supervised Representation Learning for Speech Processing
Lee, H.-Y., Mohamed, A., Watanabe, S., Sainath, T. N., Livescu, K., Li, S.-W. D., Yang, S.-W., & Kirchhoff, K. (2022) · doi:10.18653/v1/2022.naacl-tutorials.2 · Cited by 14
Exploring Disentanglement with Multilingual and Monolingual VQ-VAE
Williams, J. A., Fong, J., Cooper, E., & Yamagishi, J. (2021) · ISCA Speech Synthesis Workshop (SSW) · doi:10.21437/ssw.2021-22 · Cited by 10
Connecting the Dots in Self-Supervised Learning: A Brief Survey for Beginners
Fang, P.-F., Li, X., Yan, Y., Zhang, S., Kang, Q., Li, X.-F., & Lan, Z. (2022) · Journal of Computational Science and Technology · doi:10.1007/s11390-022-2158-x · Cited by 9
Addressing Data Scarcity in Voice Disorder Detection with Self-Supervised Models
Gupta, R., Madill, C., Gunjawate, D. R., Nguyên, D. D., & Jin, C. T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446075 · Cited by 8
CTCBERT: Advancing Hidden-Unit Bert with CTC Objectives
Fan, R., Wang, Y., Gaur, Y., & Li, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094922 · Cited by 7
Towards Semi-Supervised Semantics Understanding from SpeechPreprint
Lai, C.-I., Cao, J. L., Bodapati, S., & Li, S.-W. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06195 · Cited by 7
Language Recognition Based on Unsupervised Pretrained Models
Yu, H., Zhao, J., Yang, S., Wu, Z., Nie, Y., & Zhang, W.-Q. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-807 · Cited by 5
Improved Representation Learning For Acoustic Event Classification Using Tree-Structured Ontology
Zharmagambetov, A. S., Tang, Q., Kao, C.-C., Zhang, Q., Sun, M., Rozgić, V., Droppo, J., & Wang, C. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746266 · Cited by 5
WhisPAr: Transferring pre-trained audio models to fine-grained classification via Prompt and Adapter
Shi, B., Wang, H., Lu, C., & Zhao, M. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112242 · Cited by 4
Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspective
Liu, A. H., Yeh, S.-L., & Glass, J. R. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447758 · Cited by 7
Self-citationOn Scaling Contrastive Representations for Low-Resource Speech Recognition
Borgholt, L., Tax, T. M. S., Havtorn, J. D., Maaløe, L., & Igel, C. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414310 · Cited by 4
Discrete representations in neural models of spoken language
Higy, B., Gelderloos, L., Alishahi, A., & Chrupała, G. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.11 · Cited by 4
Introducing Semantics into Speech Encoders
Xu, D. C., Dong, S., Wang, C., Kim, S., Lin, Z., Liu, B., Shrivastava, A., Li, S.-W. D., Tseng, L.-H., Lin, G.-T., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.639 · Cited by 11
On the Transferability of Large-Scale Self-Supervision to Few-Shot Audio Classification
Heggan, C., Budgett, S., Hospedales, T., & Yaghoobi, M. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10626094 · Cited by 4
Chen, C., & Huang, L. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.111439 · Cited by 5
Ahmadian, A., & Lindsten, F. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.07.011 · Cited by 3
Comparison of Self-Supervised Speech Pre-Training Methods on Flemish Dutch
Poncelet, J., & hamme, H. V. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688061 · Cited by 4
Text-Queried Audio Source Separation via Hierarchical Modeling
Yin, X., Peng, X., Jiang, X., Xiong, Z., & Lu, Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3633078 · Cited by 3
A Superb-Style Benchmark of Self-Supervised Speech Models for Audio Deepfake Detection
Ali, H., Adupa, N. S., Subramani, S., & Malik, H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460421 · Cited by 10
Learning Dependencies of Discrete Speech Representations with Neural Hidden Markov Models
Yeh, S.-L., & Tang, H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094772 · Cited by 2
Self-citationPatch-wise vector quantization for unsupervised medical anomaly detection
Kim, T., Lee, Y.-G., Jeong, I., Ham, S., & Woo, S. S. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.06.028 · Cited by 5
Estimating the Completeness of Discrete Speech Units
Yeh, S.-L., & Tang, H. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832198 · Cited by 11
Self-citationEfficient Training of Self-Supervised Speech Foundation Models on a Compute Budget
Liu, A. T., Lin, Y., Wu, H., WINKLER, S. S., & Lee, H.-Y. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832361 · Cited by 3
Empower Typed Descriptions by Large Language Models for Speech Emotion Recognition
Wu, H., Chou, H.-C., Chang, K.-W., Goncalves, L., Du, J., Jang, J. R., Lee, C.-C., & Lee, H.-Y. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10848758 · Cited by 2
Chou, H.-C., Wu, H., & Lee, C.-C. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890558 · Cited by 3
Discrete representations in neural models of spoken languagePreprint
Higy, B., Gelderloos, L., Alishahi, A., & Chrupała, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05582 · Cited by 4
Medical diagnosis using artificial neural networks
Begum, A., Rahman, M. M., & Jahan, S. (2024) · Mathematics in Applied Sciences and Engineering · doi:10.5206/mase/17138 · Cited by 1
Speaker-Invariant Speech-to-Intent Classification for Low-Resource Languages
Ignatius, A., & Thayasivam, U. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87802-3_26 · Cited by 1
Causal Self-Supervised Pretrained Frontend With Predictive Patterns for Speech Separation
Wang, W., Pan, Z., Li, X., Wang, S., & Li, H. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3640969 · Cited by 2
SVQ-MAE: an efficient speech pre-training framework with constrained computational resources
Zhuang, X., Qian, Y., & Wang, M. (2024) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-024-00375-1 · Cited by 1
Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio LearnersPreprint
Yadav, S., Theodoridis, S., Hansen, L. K., & Tan, Z.-H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.00561 · Cited by 12
Layer Reduction: Accelerating Conformer-Based Self-Supervised Model via Layer ConsistencyPreprint
Tian, J., Gu, R., Wang, H., & Zou, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.00812
Dual Learning for Large Vocabulary On-Device ASR
Peyser, C., Huang, R., Sainath, T. N., Prabhavalkar, R., Picheny, M., & Cho, K. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023407 · Cited by 1
Speech Taskonomy: Which Speech Tasks are the most Predictive of fMRI Brain Activity?
Oota, S. R., Agarwal, V., Marreddy, M., Gupta, M., & Bapi, R. S. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-121 · Cited by 7
Optimizations of Neural Audio Coder Toward Perceptual Transparency
Byun, J., Shin, S., Hwang, S.-R., Sung, J., Beack, S., & Park, Y. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3437155
Optimizing codebook training through control chart analysis
Wang, K., Shi, Q., Li, X., Wu, E., & Li, Z. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01555-x
Self-supervised Generative Network for Personalized Speech Enhancement
Yang, C., Yu, J., Zhang, L., & Wu, L. (2024) · International Conference on Signal Processing, Computer Networks and Communications (SPCNC) · doi:10.1145/3712335.3712419
DSSLNet: dual-stream self-supervised learning network for end-to-end speech recognition
Lyu, B., Fan, C., Ming, Y., & Hu, N. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06713-w
Machine Learning of Emotional Expressions In the Wild from Acoustic Signals and Text
Samir, S. A. (2023) · theses.fr (ABES) · doi:10.70675/d64c4bf2z99b1z4e18zbc76z736e91c6d559
Famularo, R. L., Aboelata, A., Schatz, T., & Feldman, N. H. (2026) · Developmental Science · doi:10.1111/desc.70220
Multilingual Speech Emotion Recognition Using End-to-End Multilingual Pre-trained Model
Das, A., & Haque, M. A. (2025) · IEEE International Conference on Signal Processing, Information, Communication and Systems (SPICSCON) · doi:10.1109/spicscon69221.2025.11504192
Metadata
- Authors
- Chung, Y.-A., Tang, H., & Glass, J.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Venue
- Interspeech 2020
- Publisher
- ISCA
- Published
- 2020-10-25
- Pages
- 3760-3764
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W3097286738
- Cited by
- 97
- References
- 27
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 21 · 2023: 18 · 2022: 24 · 2021: 22 · 2020: 3
- FWCI
- 6.31
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- public-domain
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- vector quantization, self-supervised learning, representation learning, probing tasks, mutual information, speech representation, phonetic information, speaker information, model capacity, downstream task performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.