The DOI matches its Crossref record; 52 of the 55 references it cites were found in a registry or an index.
Layer-Wise Analysis of a Self-Supervised Speech Representation Model
Open accessAbstract
Recently proposed self-supervised learning approaches have been successful for pre-training speech representation models. The utility of these learned representations has been observed empirically, but not much has been studied about the type or extent of information encoded in the pre-trained representations themselves. Developing such insights can help understand the capabilities and limits of these models and enable the research community to more efficiently develop their usage for downstream applications. In this work, we begin to fill this gap by examining one recent and successful pre-trained model (wav2vec 2.0), via its intermediate representation vectors, using a suite of analysis tools. We use the metrics of canonical correlation, mutual information, and performance on simple downstream tasks with non-parametric probes, in order to (i) query for acoustic and linguistic information content, (ii) characterize the evolution of information across model layers, and (iii) understand how fine-tuning the model for automatic speech recognition (ASR) affects these observations. Our findings motivate modifying the fine-tuning protocol for ASR, which produces improved word error rates in a low-resource setting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Layer-wise analyses of self-supervised speech models showthat acoustic and speaker-related cues concentrate in lower and intermediate layers while deeper layers drift toward lexical and semantic content [19, 20], and that a learnable combination of all layers consistently outperforms the last-layer representation across diverse speech tasks [21].”
From MOSS-Audio Technical Report (2026) · cites this paper as [19]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
RELATIONS BETWEEN TWO SETS OF VARIATES
Hotelling, H. (1936) · Biometrika · doi:10.1093/biomet/28.3-4.321 · Cited by 6,041
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Unsupervised Visual Representation Learning by Context Prediction
Doersch, C., Gupta, A., & Efros, A. A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.167 · Cited by 2,629
Research works printed but not linked: 2
- 10.1109/SLT48900.2021.9383594 · DOI
- radford, 2018, Improving language understanding by generative pretraining
Cited by
Sorted by citations
Items 201–264 of 264
The Potential of Speech Features to Discriminate between Original and Machine-Translated Texts
Chen, Y., Farrús, M., & Toral, A. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887578
Automatic Text Pronunciation Correlation Generation and Application for Contextual Biasing
Cheng, G., Lu, H., Yang, C., Wang, X., Ta, L., & Yan, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888752
Lee, S., Kim, S.-H., & Chung, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889076
Learning Rich Speech Representations with Acoustic-Semantic Factorization
Niu, M., Sadoughi, N., Yanamandra, A., Wang, P., Liu, Z., Bhat, V., & Norred, S. E. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889923
Hierarchical Loss for Bi-Level Classification of Speech into Language and Dialects
Angra, A., Muralikrishna, H., Dileep, A. D., & Thenkanidiyoor, V. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887667
Exploring Prediction Targets in Masked Pre-Training for Speech Foundation Models
Chen, L., Higuchi, T., Bai, H., Abdelaziz, A. H., Watanabe, S., Rudnicky, A. I., Likhomanenko, T., Theobald, B.-J., & Aldeneh, Z. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889924
Neural Network Pruning for Invariance Learning
Xu, D., Chen, Y., Sun, Y., & Wang, W. W. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709262
Mesh-Agnostic Audio-Driven 3D Facial Animation Generation
Seo, K., Cha, S., Noh, J., Lee, I., & Noh, J. (2025) · Journal of the Korea Computer Graphics Society · doi:10.15701/kcgs.2025.31.3.87
An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training
Labrak, Y., Dufour, R., & Rouvier, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-02548-7_2
MMFA: Masked Multi-Layer Feature Aggregation for Speaker Verification Using WavLM
Lee, U., & Lee, S.-P. (2025) · Electronics · doi:10.3390/electronics14193857
Ananeva, A., Tomilov, A., & Volkova, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-07959-6_15
Liu, Y., Yang, X., & Qu, D. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04859-1
Audio Splicing Detection Using Self-Supervised Representations
Pascu, O., & Cucu, H. (2025) · International Conference on Speech Technology and Human-Computer Dialogue (SpeD) · doi:10.1109/sped67700.2025.11252446
Detecting Audio Deepfakes on the Edge: Lightweight SSL-Based Detection in a Browser Plugin
Pascu, O., Oneaţă, D., Cucu, H., & Müller, N. M. (2025) · International Conference on Speech Technology and Human-Computer Dialogue (SpeD) · doi:10.1109/sped67700.2025.11252312
Yamashita, N., Yamamoto, M., & Kawaguchi, Y. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249181
Dimension 414 and Minimal Embedding Dimensions for Phonetic Feature Encoding in WavLM
Sivalingam, N., & Thayasivam, U. (2025) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc65261.2025.11249033
Shibu, A., Angra, A., Muralikrishna, H., Chadaga, K., Dinesh, D. A., & Kannadaguli, P. (2025) · IEEE Access · doi:10.1109/access.2025.3639300
QR-VC: Leveraging Quantization Residuals for Linear Disentanglement in Zero-Shot Voice Conversion
Sim, Y., Yoon, J., Jeong, W., & Suh, Y.-J. (2025) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco63237.2025.11226445
Liu, Y.-L., Li, Y., Feng, R., Chen, J., Wang, Y., Chen, Y., Ding, N., Yuan, J., & Ling, Z.-H. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104968
Luan, Y., Speights, M. L., Dozier, G. V., & Seals, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-13187-4_13
Self-layer-wise Distillation for Light-Weighted Transformer Models in Speech Emotion Recognition
Yalavarthi, D. L., Bdeir, A., Landwehr, N., & Böck, R. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-4445-5_14
Brain-like dynamics in speech representations can emerge through self-supervised learningPreprint
Liu, O. D., Tang, H., Feldman, N. H., & Goldwater, S. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.16.700011
Petrache, M., Carvallo, A., Silva, V., Barceló, P., & Peña, M. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.01.30.702650
DHWANI - Domain Adapted Hybrid Whisper for Audio Transcription in Indic Languages
Sasmal, S., Nikam, H., Dhote, P., Bhargava, M., & Singh, L. (2025) · IEEE Pune Section International Conference (PuneCon) · doi:10.1109/punecon67554.2025.11377770
Toward a Realistic Encoding Model of Auditory Affective Understanding in the Brain
Pan, G., Yang, Y., Chen, S., Wang, X., Liu, L., Zheng, H., & Tang, S. (2026) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2026.3670015
Interpreting Speaker Characteristics in the Dimensions of Self-Supervised Speech Features
van Rensburg, K. J., van Niekerk, B., & Kamper, H. (2026) · IEEE Signal Processing Letters · doi:10.1109/lsp.2026.3716838
A CTC-Based Speech Recognition Network Fusing Local Convolution and Global Attention
Hu, H., Tang, C., Tan, P., & Xu, H. (2026) · Sensors · doi:10.3390/s26061865 · Cited by 1
Lin, J., Ao, J., Ge, M., Feng, M., & Li, H. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3675810
Man, D., Liu, X. H., Ran, K., Li, X., Niu, X., & Shi, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6478712
Do Self-Supervised Speech Models Exhibit the Critical Period Effects in Language Acquisition?
Koga, Y., Kando, S., & Miyao, Y. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434722
JOOCI: a Novel Method for Learning Comprehensive Speech Representations
Yadav, H. N., Sitaram, S., & Shah, R. R. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434653
Intermediate-Selective Feature Enhancement for Speech Emotion Recognition
Li, Y., Xing, X., Mai, J., Xing, J., & Xu, X. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434758
Evaluating Self-Supervised Speech Models Via Text-Based LLMs
Maekaku, T., Goto, K., Tian, J., Shinohara, Y., & Watanabe, S. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434740
Conscious Detection of Spoken Words Depends on Their Valence
Chen, G. R., Maswadeh, Z., Deouell, L., & Hassin, R. R. (2026) · Psychological Science · doi:10.1177/09567976261434113
Modelling non-native speech perception using automatic speech processing
Millet, J. (2022) · theses.fr (ABES) · doi:10.70675/86b4843az71c8z41d4za9dcz3db2229b1254
The Role of Prosodic and Lexical Cues in Turn-Taking with Self-Supervised Speech Representations
Russell, S. O., Charuau, D., & Harte, N. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464984
Han, S., Lee, S., & Lee, K. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462163
Mind the Shift: Using Delta SSL Embeddings to Enhance Child ASR
Wang, Z., Shankar, N. B., Zhang, K., Wang, Z., & Alwan, A. A. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462835
DistilMOS: Layer-Wise Self-Distillation for Self-Supervised Learning Model-Based MOS Prediction
Jianing, Y., Nakata, W., Saito, Y., & Saruwatari, H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464858 · Cited by 2
Shao-Hsuan, C., Hsieh, P.-C., Shen, Y.-L., Chi, T.-S., Liao, Y.-F., Lin, C.-H., & Xu, J.-W. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461375
Meghanani, A., & Hain, T. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462377
Ohnaka, H., Shirahata, Y., & Kawamura, M. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461239 · Cited by 1
Context-Aware Two-Stage Training for Domain Generalization in Speech Separation
Wang, W., Pan, Z., Lin, J., Wang, S., & Li, H. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3703253
MACA: Multi-Axis Covariance Alignment for Robust Distillation of Self-Supervised Speech Models
Kim, D.-H., Lee, J.-H., & Chang, J.-H. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3703215
Interpreting SSL Representations for Spoof Detection: A WavLM Study
Mallat, M., Panariello, M., Todisco, M., Evans, N., & Larcher, A. (2026) · International Workshop on Biometrics and Forensics (IWBF) · doi:10.1109/iwbf68042.2026.11558152
MSLA-XLS-R: Enhancing hierarchical SSL representations for audio deepfake detection
Meng, H., Tang, Y., Sun, Y., Deng, Y., Li, Z., & Li, Y. (2026) · Computer Speech & Language · doi:10.1016/j.csl.2026.102019
Group-Wise Layer Aggregation of Self-Supervised Representations for Speech Emotion Recognition
Park, S., Kim, H. S., & Kong, K. (2026) · IEEE Signal Processing Letters · doi:10.1109/lsp.2026.3707494
Voice, Speech, and Large Language Models in Neurology: From Acoustic Biomarkers to Conversational AI
Shelly, S. (2026) · Computation · doi:10.3390/computation14070160
VisG AV-HuBERT: Viseme-Guided AV-HuBERT
Papadopoulos, A., Jain, R., & Harte, N. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31335-5_45
Khanna, P., Sambangi, S., Vijaya Saraswathi, R., Ravuri, V., & Vuppala, A. K. (2026) · International Conference on Signal Processing and Communications (SPCOM) · doi:10.1109/spcom71105.2026.11622994
Tracking Phonetic Transitions in Self-Supervised Speech Representations
Kaustubh, K., Kachhi, A. B., Issac, A., & Sharma, A. K. (2026) · International Conference on Signal Processing and Communications (SPCOM) · doi:10.1109/spcom71105.2026.11622934
Are Speech Language Models More Noise Robust for Zero Shot Children's Speech Recognition
Das, A., Sinha, A., & Kathania, H. K. (2026) · International Conference on Signal Processing and Communications (SPCOM) · doi:10.1109/spcom71105.2026.11623001
QMOS: Qwen-Based MOS Prediction for TTS
Kolluru, R. S., Devarakonda, S. C., Salopanthula, S. R. S., & Vuppala, A. K. (2026) · International Conference on Signal Processing and Communications (SPCOM) · doi:10.1109/spcom71105.2026.11622925
Sanjay, B., Vijayalakshmi, P., Actlin Jeeva, M. P., Priyadharshini, M. K., & Nagarajan, T. (2026) · Speech Communication · doi:10.1016/j.specom.2026.103464
An Effective Structured Generation Approach to Sentence Stress Detection with Speech Language Models
Hung, T.-A., Hsieh, Y.-H., Lo, T.-H., Lin, H.-Y., Hsu, Y.-C., & Chen, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37249-9_50
On the Explainability of Speech Impairment Regarding Dysarthria Progression in Parkinson’s Disease
Gallo-Aristizábal, J. D., Ríos-Urrego, C. D., Escobar-Grisales, D., Mercado, J., & Orozco‐Arroyave, J. R. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37249-9_38
Neural encoding of auditory processing in schizophrenia
Vuyyuru, L. R., & Raju, A. (2026) · Frontiers in Neuroscience · doi:10.3389/fnins.2026.1893300
Chu, M.-N., Lin, C.-H., & Hsu, J.-L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37867-5_28
Uteuliev, N., Kudaybergenov, J., Kudaybergenov, T., Rakhimovich, M. A., & Khudaybergenov, K. K. (2026) · Information · doi:10.3390/info17101048
Roselló, E., López‐Espejo, I., Peinado, A. M., & Gomez, A. M. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3739196
Multimodal Depression Detection Varies with Cross-Dataset Differences
Bilalpur, M., Hinduja, S., Sivarajkumar, S., Wang, Y., Allen, N. B., Ertuğrul, I. Ö., & Cohn, J. F. (2026) · Companion Publication of the International Conference on Multimodal Interaction (ICMI Companion) · doi:10.1145/3776591.3836736
Rosero, K., Lee, C.-C., Hallac, R. R., & Busso, C. (2026) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3776574.3831166
Lillini, D., Aironi, C., Migliorelli, L., Gabrielli, L., & Squartini, S. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-42348-1_14
Shimizu, R., Antonello, R., Singh, C., & Mesgarani, N. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014906
Metadata
- Authors
- Pasad, A., Chou, J.-C., & Livescu, K.
- Affiliations
- Toyota Technological Institute at Chicago (US)
- ORCID
- Livescu, Karen
- Venue
- 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)
- Publisher
- IEEE
- Published
- 2021-12-13
- Pages
- 914-921
- OpenAlex
- W3179803166
- Cited by
- 262
- References
- 55
- Citations by year
- 2026: 42 · 2025: 65 · 2024: 61 · 2023: 62 · 2022: 29 · 2021: 3
- FWCI
- 17.90
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, public-domain)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- NSF (IIS-1816627)
- Air Force Office of Scientific Research (FA9550-18-1-0166)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- self-supervised speech representation learning, wav2vec 2.0, layer-wise analysis, canonical correlation analysis, mutual information
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review