Layer-Wise Analysis of a Self-Supervised Speech Representation Model
Open accessAbstract
Recently proposed self-supervised learning approaches have been successful for pre-training speech representation models. The utility of these learned representations has been observed empirically, but not much has been studied about the type or extent of information encoded in the pre-trained representations themselves. Developing such insights can help understand the capabilities and limits of these models and enable the research community to more efficiently develop their usage for downstream applications. In this work, we begin to fill this gap by examining one recent and successful pre-trained model (wav2vec 2.0), via its intermediate representation vectors, using a suite of analysis tools. We use the metrics of canonical correlation, mutual information, and performance on simple downstream tasks with non-parametric probes, in order to (i) query for acoustic and linguistic information content, (ii) characterize the evolution of information across model layers, and (iii) understand how fine-tuning the model for automatic speech recognition (ASR) affects these observations. Our findings motivate modifying the fine-tuning protocol for ASR, which produces improved word error rates in a low-resource setting.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Layer-wise analyses of self-supervised speech models show that acoustic and speaker-related cues concentrate in lower and intermediate layers while deeper layers drift toward lexical and semantic content, and that a learnable combination of all layers consistently outperforms the last-layer representation across diverse speech tasks.
“Layer-wise analyses of self-supervised speech models showthat acoustic and speaker-related cues concentrate in lower and intermediate layers while deeper layers drift toward lexical and semantic content [19, 20], and that a learnable combination of all layers consistently outperforms the last-layer representation across diverse speech tasks [21].”
From MOSS-Audio Technical Report (2026) · cites this paper as [19]
Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
RELATIONS BETWEEN TWO SETS OF VARIATES
Hotelling, H. (1936) · Biometrika · doi:10.1093/biomet/28.3-4.321 · Cited by 5,429 · see citing works
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370 · see citing works
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629 · see citing works
Relations Between Two Sets of Variates
Hotelling, H. (1936) · Biometrika · doi:10.2307/2333955 · Cited by 1,663 · see citing works
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307 · see citing works
wav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290 · see citing works
Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi
McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M., & Sonderegger, M. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1386 · Cited by 1,169 · see citing works
Ethayarajh, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1006 · Cited by 841 · see citing works
Similarity of Neural Network Representations Revisited
Kornblith, S., Norouzi, M., Lee, H., & Hinton, G. (2019) · arXiv · doi:10.48550/arxiv.1905.00414 · Cited by 724 · see citing works
SUPERB: Speech Processing Universal PERformance Benchmark
Yang, S.-W., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1775 · Cited by 681 · see citing works
Unsupervised Cross-Lingual Representation Learning for Speech Recognition
Conneau, A., Baevski, A., Collobert, R., Mohamed, A., & Auli, M. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-329 · Cited by 637 · see citing works
SpeechBrain: A General-Purpose Speech Toolkit
Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., Zhong, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.04624 · Cited by 512 · see citing works
Self-citationAnalysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 485 · see citing works
Libri-Light: A Benchmark for ASR with Limited or No Supervision
Kahn, J., Rivière, M., Zheng, W., Kharitonov, E., Xu, Q., Mazaré, P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9052942 · Cited by 475 · see citing works
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020) · neural information processing systems · Cited by 386 · see citing works
Raghu, M., Gilmer, J., Yosinski, J., & Sohl-Dickstein, J. (2017) · arXiv · doi:10.48550/arxiv.1706.05806 · Cited by 335 · see citing works
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
Liu, A. T., Yang, S.-W., Chi, P.-H., Hsu, P.-C., & Lee, H.-Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054458 · Cited by 335 · see citing works
Revisiting Few-sample BERT Fine-tuningPreprint
Zhang, T., Wu, F., Katiyar, A., Weinberger, K. Q., & Artzi, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05987 · Cited by 295 · see citing works
Speech Model Pre-Training for End-to-End Spoken Language Understanding
Lugosch, L., Ravanelli, M., Ignoto, P., Tomar, V. S., & Bengio, Y. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2396 · Cited by 280 · see citing works
Learning Problem-Agnostic Speech Representations from Multiple Self-Supervised Tasks
Pascual, S., Ravanelli, M., Serrà, J., Bonafonte, A., & Bengio, Y. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2605 · Cited by 222 · see citing works
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training
Hsu, W.-N., Sriram, A., Baevski, A., Likhomanenko, T., Xu, Q., Pratap, V., Kahn, J., Lee, A., Collobert, R., Synnaeve, G., et al. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-236 · Cited by 191 · see citing works
Unsupervised Speech RecognitionPreprint
Baevski, A., Hsu, W.-N., Conneau, A., & Auli, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.11084 · Cited by 191 · see citing works
Insights on representational similarity in neural networks with canonical correlation
Morcos, A. S., Raghu, M., & Bengio, S. (2018) · arXiv · doi:10.48550/arxiv.1806.05759 · Cited by 171 · see citing works
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
Chung, Y.-A., & Glass, J. R. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2341 · Cited by 163 · see citing works
Generative Pre-Training for Speech with Autoregressive Predictive Coding
Chung, Y.-A., & Glass, J. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054438 · Cited by 157 · see citing works
Voita, E., Sennrich, R., & Titov, I. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1448 · Cited by 147 · see citing works
Deep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition
Ling, S., Liu, Y., Salazar, J., & Kirchhoff, K. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053176 · Cited by 145 · see citing works
Representations of language in a model of visually grounded speech signal
Chrupała, G., Gelderloos, L., & Alishahi, A. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1057 · Cited by 128 · see citing works
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?
Hsu, W.-N., Tsai, Y.-H. H., Bolte, B., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414460 · Cited by 123 · see citing works
Unsupervised neural network based feature extraction using weak top-down constraints
Kamper, H., Elsner, M., Jansen, A., & Goldwater, S. J. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7179087 · Cited by 111 · see citing works
Community Evaluation and Exchange of Word Vectors at wordvectors.org
Faruqui, M., & Dyer, C. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5004 · Cited by 94 · see citing works
Rapid evaluation of speech representations for spoken term discovery
Carlin, M. A., Thomas, S., Jansen, A., & Hynek Hermansky, B. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-304 · Cited by 90 · see citing works
Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction
Wang, W., Tang, Q., & Livescu, K. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053541 · Cited by 81 · see citing works
Self-citationNguyen, T. A., de Seyssel, M., Rozé, P., Rivière, M., Kharitonov, E., Baevski, A., Dunbar, E., & Emmanuel Dupoux, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.11588 · Cited by 67 · see citing works
Shah, J., Singla, Y. K., Chen, C., & Shah, R. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.00387 · Cited by 45 · see citing works
Kumar, Y., Shah, J., Shah, R. R., & Chen, C. (2021) · Preprints.org · doi:10.20944/preprints202101.0081.v1 · Cited by 45 · see citing works
Probing Acoustic Representations for Phonetic Properties
Ma, D., Ryant, N., & Liberman, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414776 · Cited by 36 · see citing works
Similarity Analysis of Self-Supervised Speech Representations
Chung, Y.-A., Belinkov, Y., & Glass, J. R. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414321 · Cited by 35 · see citing works
Evaluating the Reliability of Acoustic Speech Embeddings
Algayres, R., Zaiem, S., Sagot, B., & Emmanuel Dupoux, A. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2362 · Cited by 30 · see citing works
Acoustically Grounded Word Embeddings for Improved Acoustics-to-word Speech Recognition
Settle, S., Audhkhasi, K., Livescu, K., & Picheny, M. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682903 · Cited by 30 · see citing works
Self-citationLarge-Scale Self- and Semi-Supervised Learning for Speech Translation
Wang, C., Wu, A., Pino, J., Baevski, A., Auli, M., & Conneau, A. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1912 · Cited by 23 · see citing works
How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems
Prasad, A., & Jyothi, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.345 · Cited by 22 · see citing works
Multilingual Jointly Trained Acoustic and Written Word Embeddings
Hu, Y., Settle, S., & Livescu, K. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2828 · Cited by 19 · see citing works
Self-citationLearned in Speech Recognition: Contextual Acoustic Word Embeddings
Palaskar, S., Raunak, V., & Metze, F. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683868 · Cited by 17 · see citing works
Large-Scale Self- and Semi-Supervised Learning for Speech TranslationPreprint
Wang, C., Wu, A., Pino, J., Baevski, A., Auli, M., & Conneau, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06678 · Cited by 7 · see citing works
Flagged first, then by citations
4 of the 55 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Carl Doersch, Abhinav Gupta, and Alexei A Efros, "Un- supervised visual representation learning by context pre- diction," in CVPR, 2015.
- Yushi Hu, Shane Settle, and Karen Livescu, "Multilin- gual jointly trained acoustic and written word embed- dings," in Interspeech, 2020.
Cited by
Sorted by citations
Audiolm: A language modeling approach to audio generation
Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., et al. (2023) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3288409 · Cited by 432 · see citing works
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372 · see citing works
“[68] further show that fine-tuning the pre-trained model with a character-level CTC training criterion changes the behavior of the last layers (especially the final two layers), breaking”
BackgroundSelf-citationInvestigating Self-Supervised Front Ends for Speech Spoofing Countermeasures
Wang, X., & Yamagishi, J. (2022) · The Speaker and Language Recognition Workshop (Odyssey 2022) · doi:10.21437/odyssey.2022-14 · Cited by 138 · see citing works
Distilhubert: Speech Representation Learning by Layer-Wise Distillation of Hidden-Unit Bert
Chang, H.-J., Yang, S.-W., & Lee, H.-Y. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747490 · Cited by 120 · see citing works
Large-Scale Self-Supervised Speech Representation Learning for Automatic Speaker Verification
Chen, Z., Chen, S., Wu, Y., Qian, Y., Wang, C., Liu, S., Qian, Y., & Zeng, M. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747814 · Cited by 112 · see citing works
Comparative Layer-Wise Analysis of Self-Supervised Speech Models
Pasad, A., Shi, B., & Livescu, K. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096149 · Cited by 103 · see citing works
Self-citationZhao, J. S., & Zhang, W.-Q. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3184480 · Cited by 78 · see citing works
Fusing ASR Outputs in Joint Training for Speech Emotion Recognition
Li, Y., Bell, P., & Lai, C. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746289 · Cited by 57 · see citing works
Chang, X., Maekaku, T., Fujita, Y., & Watanabe, S. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10839 · Cited by 54 · see citing works
A Large-Scale Evaluation of Speech Foundation Models
Yang, S.-W., Chang, H.-J., Huang, Z., Liu, A. T., Lai, C.-I., Wu, H., Shi, J., Chang, X., Tsai, H.-S., Huang, W.-C., et al. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3389631 · Cited by 49 · see citing works
Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learning
Kim, E., Jeon, J.-J., Seo, H., & Kim, H. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10245 · Cited by 44 · see citing works
Srinivasan, S., Huang, Z., & Kirchhoff, K. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747754 · Cited by 42 · see citing works
Boosting Self-Supervised Embeddings for Speech Enhancement
Hung, K.-H., Fu, S., Tseng, H.-H., Chiang, H.-T., Tsao, Y., & Lin, C. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10002 · Cited by 43 · see citing works
Exploration of a Self-Supervised Speech Model: A Study on Emotional Corpora
Li, Y., Mohamied, Y., Bell, P., & Lai, C. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023428 · Cited by 39 · see citing works
Self-Supervised Representations in Speech-Based Depression Detection
Wu, W., Zhang, C., & Woodland, P. C. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094910 · Cited by 47 · see citing works
What Do Self-Supervised Speech Models Know About Words?
Pasad, A., Chien, C.-M., Settle, S., & Livescu, K. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00656 · Cited by 36 · see citing works
Self-citationSharrab, Y. O., Attar, H. H., Eljinini, M. A. H., Al-Omary, Y., & Al-Momani, W. (2025) · IEEE Access · doi:10.1109/access.2025.3550855 · Cited by 35 · see citing works
Word Discovery in Visually Grounded, Self-Supervised Speech Models
Peng, P., & Harwath, D. F. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10652 · Cited by 35 · see citing works
Naderi, N., & Nasersharif, B. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110814 · Cited by 35 · see citing works
Cheng, G., Wang, X., Cooper, E., Wells, D., Wang, L., Dang, J., Richmond, K., & Yamagishi, J. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3451951 · Cited by 31 · see citing works
On the Utility of Self-Supervised Models for Prosody-Related Tasks
Lin, G.-T., Feng, C.-L., Huang, W., Tseng, Y., Lin, T.-H., Li, C.-A., Lee, H.-Y., & Ward, N. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023234 · Cited by 31 · see citing works
Probing speech emotion recognition transformers for linguistic knowledge
Triantafyllopoulos, A., Wagner, J., Wierstorf, H., Schmitt, M., Reichel, U. D., Eyben, F., Burkhardt, F., & Schuller, B. W. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10371 · Cited by 28 · see citing works
Pseudo Label Is Better Than Human Label
Hwang, D., Sim, K. C., Huo, Z., & Strohman, T. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-11034 · Cited by 29 · see citing works
Exploring Efficient-Tuning Methods in Self-Supervised Speech Models
Chen, Z.-C., Fu, C.-L., Liu, C.-Y., Li, S.-W. D., & Lee, H.-Y. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023274 · Cited by 26 · see citing works
Li, Y. A., Han, C., Raghavan, V., Mischler, G., & Mesgarani, N. (2023) · PubMed · doi:10.48550/arxiv.2306.07691 · Cited by 24 · see citing works
Language-Independent Speaker Anonymization Approach Using Self-Supervised Pre-Trained Models
Miao, X., Wang, X., Cooper, E., Yamagishi, J., & Tomashenko, N. (2022) · The Speaker and Language Recognition Workshop (Odyssey) · doi:10.21437/odyssey.2022-39 · Cited by 25 · see citing works
SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model
Shih, Y.-J., Wang, H.-F., Chang, H.-J., Berry, L., Lee, H.-Y., & Harwath, D. F. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022954 · Cited by 24 · see citing works
Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech
Cho, C. J., Wu, P., Mohamed, A., & Anumanchipalli, G. K. (2023) · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp49357.2023.10094711 · Cited by 21 · see citing works
Wu, Y., Zhang, Z., Peng, P., Zhao, Y., & Qin, B. (2022) · doi:10.1145/3551876.3554813 · Cited by 21 · see citing works
Analyzing Acoustic Word Embeddings from Pre-Trained Self-Supervised Speech Models
Sanabria, R., Tang, H., & Goldwater, S. J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096099 · Cited by 20 · see citing works
Boosting Cross-Domain Speech Recognition With Self-Supervision
Zhu, H., Cheng, G., Wang, J., Hou, W., Zhang, P., & Yan, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3301230 · Cited by 20 · see citing works
Leveraging Pretrained Representations With Task-Related Keywords for Alzheimer’s Disease Detection
Li, J., Song, K., Li, J., Zheng, B., Li, D., Wu, X., Liu, X., & Meng, H. M. L. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096205 · Cited by 18 · see citing works
Parameter Efficient Transfer Learning for Various Speech Processing Tasks
Otake, S., Kawakami, R., & Inoue, N. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096311 · Cited by 18 · see citing works
Fine-Tuning Strategies for Faster Inference Using Speech Self-Supervised Models: A Comparative Study
Zaiem, S., Algayres, R., Parcollet, T., Essid, S., & Ravanelli, M. (2023) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw59220.2023.10193042 · Cited by 17 · see citing works
Useful blunders: Can automated speech recognition errors improve downstream dementia classification?
Li, C., Xu, W., Cohen, T., & Pakhomov, S. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104598 · Cited by 21 · see citing works
Wav2vec-VC: Voice Conversion via Hidden Representations of Wav2vec 2.0
Lim, J., & Kim, K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447984 · Cited by 17 · see citing works
A Sidecar Separator Can Convert A Single-Talker Speech Recognition System to A Multi-Talker One
Meng, L., Kang, J., Cui, M., Wang, Y., Wu, X., & Meng, H. M. L. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095295 · Cited by 16 · see citing works
MelHuBERT: A Simplified Hubert on Mel Spectrograms
Lin, T.-Q., Lee, H.-Y., & Tang, H. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389700 · Cited by 15 · see citing works
Parcollet, T., Nguyen, H., Evain, S. V., Boito, M. Z., Pupier, A., Mdhaffar, S., Le, H., Alisamir, S., Tomashenko, N., Dinarelli, M., et al. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101622 · Cited by 15 · see citing works
AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
Tseng, Y., Berry, L., Chen, Y.-T., Chiu, I.-H., Lin, H.-H., Liu, M., Peng, P., Shih, Y.-J., Wang, H., Wu, H., et al. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10445941 · Cited by 15 · see citing works
Audio-visual representation learning via knowledge distillation from speech foundation models
Zhang, J.-X., Wan, G., Gao, J., & Ling, Z.-H. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111432 · Cited by 16 · see citing works
Pandey, R., Ren, R., Luo, Q., Liu, J., Rastrow, A., Gandhe, A., Filimonov, D., Strimel, G. P., Stolcke, A., & Bulyko, I. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096062 · Cited by 14 · see citing works
Anderson, A. J., Davis, C., & Lalor, E. C. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012537 · Cited by 13 · see citing works
ELP-Adapters: Parameter Efficient Adapter Tuning for Various Speech Processing Tasks
Inoue, N., Otake, S., Hirose, T., Ohi, M., & Kawakami, R. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3434445 · Cited by 15 · see citing works
Ultra-Low-Bitrate Speech Coding with Pretrained Transformers
Siahkoohi, A., Chinen, M., Denton, T., Kleijn, W. B., & Skoglund, J. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10988 · Cited by 12 · see citing works
Transfer Learning for Personality Perception via Speech Emotion Recognition
Li, Y., Bell, P., & Lai, C. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-2061 · Cited by 12 · see citing works
Liu, O. D., Tang, H., & Goldwater, S. J. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-871 · Cited by 12 · see citing works
Nasersharif, B., & Namvarpour, M. (2024) · The Journal of Supercomputing · doi:10.1007/s11227-024-06158-x · Cited by 13 · see citing works
Saliba, A., Li, Y., Sanabria, R., & Lai, C. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10627629 · Cited by 12 · see citing works
M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation
Zhao, J., Yang, H., Haffari, G., & Shareghi, E. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-592 · Cited by 12 · see citing works
A Study of Gender Impact in Self-supervised Models for Speech-to-Text Systems
Boito, M. Z., Besacier, L., Tomashenko, N., & Estève, Y. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-353 · Cited by 15 · see citing works
Vitale, V. N., Cutugno, F., Origlia, A., & Coro, G. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09435-1 · Cited by 11 · see citing works
Speech Separation With Pretrained Frontend to Minimize Domain Mismatch
Wang, W., Pan, Z., Li, X., Wang, S., & Li, H. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3446242 · Cited by 12 · see citing works
Angra, A., Muralikrishna, H., Dinesh, D. A., & Thenkanidiyoor, V. (2024) · IEEE Access · doi:10.1109/access.2024.3523951 · Cited by 12 · see citing works
Wav2vec-S: Semi-Supervised Pre-Training for Low-Resource ASR
Zhu, H., Wang, L., Cheng, G., Wang, J., Zhang, P., & Yan, Y. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-909 · Cited by 11 · see citing works
Prosody Modelling With Pre-Trained Cross-Utterance Representations for Improved Speech Synthesis
Zhang, Y.-J., Zhang, C., Song, W. G., Zhang, Z., Wu, Y., & He, X. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3278184 · Cited by 11 · see citing works
An Analysis of Semantically-Aligned Speech-Text Embeddings
Huzaifah, M., & Kukanov, I. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023147 · Cited by 11 · see citing works
Jing, Y., & Zhao, X. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681599 · Cited by 10 · see citing works
Cross-Lingual Speech Emotion Recognition: Humans vs. Self-Supervised Models
Han, Z., Geng, T., Feng, H., Yuan, J., Richmond, K., & Li, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889008 · Cited by 10 · see citing works
ON-TRAC Consortium Systems for the IWSLT 2022 Dialect and Low-resource Speech Translation Tasks
Boito, M. Z., Ortega, J., Riguidel, H., Laurent, A., Barrault, L., Bougares, F., Chaabani, F., Nguyen, H., Barbier, F., Gahbiche, S., et al. (2022) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2022.iwslt-1.28 · Cited by 13 · see citing works
Avoid Overthinking in Self-Supervised Models for Speech Recognition
Berrebbi, D., Yan, B., & Watanabe, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095335 · Cited by 10 · see citing works
Speech self-supervised representations benchmarking: A case for larger probing heads
Zaiem, S., Kemiche, Y., Parcollet, T., Essid, S., & Ravanelli, M. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101695 · Cited by 12 · see citing works
PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations
Lodagala, V. S., Ghosh, S., & Umesh, S. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022820 · Cited by 10 · see citing works
Basheer, N., Islam, S., Alwaheidi, M. K. S., Mouratidis, H., & Papastergiou, S. (2025) · Integrated Computer-Aided Engineering · doi:10.1177/10692509251368663 · Cited by 9 · see citing works
AdVerb: Visually Guided Audio Dereverberation
Chowdhury, S., Ghosh, S., Dasgupta, S., Ratnarajah, A. J., Tyagi, U., & Manocha, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00725 · Cited by 9 · see citing works
Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation ModelsPreprint
Tseng, L.-H., Fu, Y.-K., Chang, H.-J., & Lee, H.-Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.03504 · Cited by 11 · see citing works
Analysis of Self-Supervised Speech Models on Children’s Speech and Infant Vocalizations
Li, J., Hasegawa‐Johnson, M., & McElwain, N. L. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10626416 · Cited by 8 · see citing works
Wave to Syntax: Probing spoken language models for syntax
Shen, G., Alishahi, A., Bisazza, A., & Chrupała, G. (2023) · Interspeech · doi:10.21437/interspeech.2023-679 · Cited by 34 · see citing works
Contrastive representation learning with transformers for robust auditory EEG decoding
Bollens, L., Accou, B., hamme, H. V., & Francart, T. (2025) · Scientific Reports · doi:10.1038/s41598-025-13646-4 · Cited by 7 · see citing works
Zhao, R., Jiang, X., Yu, F. R., Leung, V. C. M., Wang, T., & Zhang, S. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3613687 · Cited by 10 · see citing works
Accidental Learners: Spoken Language Identification in Multilingual Self-Supervised Models
Bartley, T., Jia, F., Puvvada, K. C., Kriman, S., & Ginsburg, B. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096407 · Cited by 7 · see citing works
Hiformer: Sequence Modeling Networks With Hierarchical Attention Mechanisms
Wu, X., Lu, H., Li, K., Wu, Z., Liu, X., & Meng, H. M. L. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3313428 · Cited by 7 · see citing works
Self-supervised Representation Learning for Speech Processing
Lee, H.-Y., Mohamed, A., Watanabe, S., Sainath, T. N., Livescu, K., Li, S.-W. D., Yang, S.-W., & Kirchhoff, K. (2022) · doi:10.18653/v1/2022.naacl-tutorials.2 · Cited by 9 · see citing works
Self-citationHow to Estimate Model Transferability of Pre-Trained Speech Models?
Chen, Z.-C., Yang, C.-H. H., Li, B., Zhang, Y., Chen, N., Chang, S.-Y., Prabhavalkar, R., Lee, H.-Y., & Sainath, T. N. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-1079 · Cited by 7 · see citing works
Integrating Emotion Recognition with Speech Recognition and Speaker Diarisation for Conversations
Wu, W., Zhang, C., & Woodland, P. C. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-293 · Cited by 7 · see citing works
Li, Z., Graave, T., Liu, J., Lohrenz, T., Kunzmann, S., & Fingscheidt, T. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389795 · Cited by 7 · see citing works
Do Self-Supervised Speech and Language Models Extract Similar Representations as Human Brain?
Chen, P., He, L., Fu, L., Fan, L., Chang, E. F., & Li, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446334 · Cited by 6 · see citing works
Advancing Spanish Speech Emotion Recognition: A Comprehensive Benchmark of Pre-Trained Models
Mares, A., Diaz-Arango, G. U., Perez-Jacome-Friscione, J., Vázquez-Leal, H., Hernández-Martínez, L., Huerta-Chua, J., Alvarado, A. F. J., & Dominguez-Chavez, J. A. (2025) · Applied Sciences · doi:10.3390/app15084340 · Cited by 6 · see citing works
Park, D., Yu, Y., Katabi, D., & Kim, H. K. (2023) · IEEE Signal Processing Letters · doi:10.1109/lsp.2023.3298532 · Cited by 7 · see citing works
Zero-Shot Emotion Transfer for Cross-Lingual Speech Synthesis
Li, Y., Zhu, X., Lei, Y., Li, H. H., Liu, J., Xie, D., & Xie, L. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389638 · Cited by 6 · see citing works
PE-Wav2vec: A Prosody-Enhanced Speech Model for Self-Supervised Prosody Learning in TTS
Liu, Z.-C., Chen, L., Hu, Y.-J., Ling, Z.-H., & Pan, J. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3449148 · Cited by 6 · see citing works
Rethinking Mamba in Speech Processing by Self-Supervised Models
Zhang, X., Ma, J., Shahin, M. A., Ahmed, B., & Epps, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889111 · Cited by 6 · see citing works
Automatic Rating of Spontaneous Speech for Low-Resource Languages
Al-Ghezi, R., Getman, Y., Voskoboinik, E., Singh, M., & Kurimo, M. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022381 · Cited by 6 · see citing works
Relative Dynamic Time Warping Comparison for Pronunciation Errors
Richter, C., & Guðnason, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095033 · Cited by 6 · see citing works
de Oliveira, D., Prabhu, N. R., & Gerkmann, T. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-1758 · Cited by 6 · see citing works
Transferring Speech-Generic and Depression-Specific Knowledge for Alzheimer’s Disease Detection
Cui, Z., Wu, W., Zhang, W.-Q., Wu, J. Y., & Zhang, C. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389785 · Cited by 6 · see citing works
Self-Supervised Models of Speech Infer Universal Articulatory Kinematics
Cho, C. J., Mohamed, A., Black, A. W., & Anumanchipalli, G. K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447345 · Cited by 6 · see citing works
Learning Music Representations with wav2vec 2.0
Ragano, A., Benetos, E., & Hines, A. (2023) · Irish Conference on Artificial Intelligence and Cognitive Science (AICS) · doi:10.1109/aics60730.2023.10470644 · Cited by 6 · see citing works
Perception of Phonological Assimilation by Neural Speech Recognition Models
Pouw, C., Kloots, M. D. H., Alishahi, A., & Zuidema, W. H. (2024) · Computational Linguistics · doi:10.1162/coli_a_00526 · Cited by 6 · see citing works
Uncertainty as a Predictor: Leveraging Self-Supervised Learning for Zero-Shot MOS Prediction
Ravuri, A., Cooper, E., & Yamagishi, J. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10626267 · Cited by 6 · see citing works
A Study of Multi-Scale Feature Learning From Pre-Trained Models on Speaker Verification
Peng, S., Guo, W., Zhang, J., Li, Z., Yu, G., Gu, B., & Ai, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887949 · Cited by 5 · see citing works
Lim, H., Kim, Y., Yeom, K., Seo, E.-J., Lee, H., Choi, S. J., & Lee, H. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094623 · Cited by 5 · see citing works
Investigating Pre-trained Audio Encoders in the Low-Resource Condition
Yang, H., Zhao, J., Haffari, G., & Shareghi, E. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-343 · Cited by 5 · see citing works
Efficient Finetuning for Dimensional Speech Emotion Recognition in the Age of Transformers
Sampath, A., Tavernor, J., & Provost, E. M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889878 · Cited by 5 · see citing works
A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTS
Wang, S., Henter, G. E., Gustafson, J., & Székely, É. (2023) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw59220.2023.10193157 · Cited by 5 · see citing works
Han, C., Choudhari, V., Li, Y. A., & Mesgarani, N. (2023) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc40787.2023.10340191 · Cited by 5 · see citing works
SCORE: Self-Supervised Correspondence Fine-Tuning for Improved Content Representations
Meghanani, A., & Hain, T. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448060 · Cited by 5 · see citing works
Generic speech enhancement with self-supervised representation space loss
Satō, H., Ochiai, T., Delcroix, M., Moriya, T., Ashihara, T., & Masumura, R. (2025) · Frontiers in Signal Processing · doi:10.3389/frsip.2025.1587969 · Cited by 4 · see citing works
Efficient Utilization of Large Pre-Trained Models for Low Resource ASR
Vieting, P., Lüscher, C., Dierkes, J., Schlüter, R., & Ney, H. (2023) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw59220.2023.10193570 · Cited by 4 · see citing works
- Authors
- Pasad, A., Chou, J.-C., & Livescu, K.
- Affiliations
- Toyota Technological Institute at Chicago (US)
- ORCID
- Livescu, Karen
- Venue
- 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)
- Publisher
- IEEE
- Published
- 2021-12-13
- Pages
- 914-921
- OpenAlex
- W3179803166
- Cited by
- 262
- References
- 55
- Citations by year
- 2026: 42 · 2025: 65 · 2024: 61 · 2023: 62 · 2022: 29 · 2021: 3
- FWCI
- 17.90
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (IIS-1816627)
- Air Force Office of Scientific Research (FA9550-18-1-0166)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- self-supervised speech representation learning, wav2vec 2.0, layer-wise analysis, canonical correlation analysis, mutual information
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
