Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech
Open accessThe DOI matches its Crossref record; 22 of the 27 references it cites were found in a registry or an index.
Abstract
Recent self-supervised learning (SSL) models have proven to learn rich representations of speech, which can readily be utilized by diverse downstream tasks. To understand such utilities, various analyses have been done for speech SSL models to reveal which and how information is encoded in the learned representations. Although the scope of previous analyses is extensive in acoustic, phonetic, and semantic perspectives, the physical grounding by speech production has not yet received full attention. To bridge this gap, we conduct a comprehensive analysis to link speech representations to articulatory trajectories measured by electromagnetic articulography (EMA). Our analysis is based on a linear probing approach where we measure articulatory score as an average correlation of linear mapping to EMA. We analyze a set of SSL models selected from the leaderboard of the SUPERB benchmark [1] and perform further layer-wise analyses on two most successful models, Wav2Vec 2.0 [2] and HuBERT [3]. Surprisingly, representations from the recent speech SSL models are highly correlated with EMA traces (best: r =0.81), and only 5 minutes are sufficient to train a linear model with high performance (r =0.77). Our findings suggest that SSL models learn to align closely with continuous articulations, and provide a novel insight into speech SSL.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Given the strong link between speech acoustics and underlying vocal-tract movements, articulation constitutes another relevant feature space (Fig. 3d) for speech decoding and speech synthesis128,129.”
From The speech neuroprosthesis (2024) · cites this paper as [129]
Citations
Cites
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · cited 1×
“2 To give a fair comparison, we tried to constrain the scope of analysis to the models pre-trained on LibriSpeech 960 hours corpus [12] or LibriLight 60K hours corpus [13], and we excluded some extension methods or variants.”
MethodHuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3122291 · Cited by 2,983
Self-citationwav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369 · cited 3×
“The final list includes: ( contrastive model ) Wav2Vec [17], VQ-Wav2Vec [18] and Wav2Vec 2.0 [2], ( predictive model ) HuBERT, Data2Vec [19] and WavLM, ( generative model ) TERA [20] and DeCoAR 2.0 [21], ( supervised model ) Fairseq S2T, and ( hand crafted ) filter bank, mel-spectrogram and MFCC…”
MethodResultInfluentialSelf-citationWavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3188113 · Cited by 1,926 · cited 6×
“Though WavLM [14] is an extension of HuBERT”
MethodInfluentialwav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
SUPERB: Speech Processing Universal PERformance Benchmark
Yang, S.-W., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-1775 · Cited by 681
Self-citationLibri-Light: A Benchmark for ASR with Limited or No Supervision
Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazare, P. E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9052942 · Cited by 458 · cited 1×
“2 To give a fair comparison, we tried to constrain the scope of analysis to the models pre-trained on LibriSpeech 960 hours corpus [12] or LibriLight 60K hours corpus [13], and we excluded some extension methods or variants.”
MethodSelf-citationvq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
Baevski, A., Schneider, S., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05453 · Cited by 388 · cited 1×
“The final list includes: ( contrastive model ) Wav2Vec [17], VQ-Wav2Vec [18] and Wav2Vec 2.0 [2], ( predictive model ) HuBERT, Data2Vec [19] and WavLM, ( generative model ) TERA [20] and DeCoAR 2.0 [21], ( supervised model ) Fairseq S2T, and ( hand crafted ) filter bank, mel-spectrogram and MFCC…”
MethodInfluentialSelf-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372 · cited 1×
“Recently proposed SSL methods are successful in diverse downstream spoken language tasks, surpassing supervised models especially in domain with small number of labeled data [4, 1].”
BackgroundSelf-citationTERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
Liu, A. T., Li, S.-W., & Lee, H.-Y. (2021) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2021.3095662 · Cited by 312 · cited 2×
“The final list includes: (contrastive model) Wav2Vec [17], VQ-Wav2Vec [18] and Wav2Vec 2.0 [2], (predictive model) HuBERT, Data2Vec [19] and WavLM, (generative model) TERA [20] and DeCoAR 2.0 [21], (supervised model) Fairseq S2T, and (hand crafted) filter bank, mel-spectrogram and MFCC (Fig.”
Influential
Cited by
A high-performance neuroprosthesis for speech decoding and avatar control
Metzger, S. L., Littlejohn, K. T., Silva, A. B., Moses, D. A., Seaton, M. P., Wang, R., Dougherty, M. E., Liu, J. R., Wu, P., Berger, M. A., et al. (2023) · Nature · doi:10.1038/s41586-023-06443-4 · Cited by 522
Self-citationA streaming brain-to-voice neuroprosthesis to restore naturalistic communication
Littlejohn, K. T., Cho, C. J., Liu, J. R., Silva, A. B., Yu, B., Anderson, V. R., Kurtz-Miott, C. M., Brosler, S., Kashyap, A., Hallinan, I. P., et al. (2025) · Nature Neuroscience · doi:10.1038/s41593-025-01905-6 · Cited by 131
“The target acoustic-speech units were extracted using HuBERT, a self-supervised speech representation learning model that encodes a speech waveform into a temporal sequence of units that capture the speech waveform’s underlying phonetic and articulatory characteristics 38,39 .”
Self-citationSilva, A. B., Littlejohn, K. T., Liu, J. R., Moses, D. A., & Chang, E. F. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-024-00819-9 · Cited by 103
“3d) for speech decoding and speech synthesis 128,129 .”
Coding Speech Through Vocal Tract Kinematics
Cho, C. J., Wu, P., Prabhune, T., Agarwal, D., & Anumanchipalli, G. K. (2024) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2024.3497655 · Cited by 50
“Furthermore, the recent findings by Cho et al. [9], [10] suggest that articulatory inversion naturally emerges from self-supervised learning (SSL) of speech.”
MethodResultInfluentialSelf-citationSD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in Hubert
Cho, C. J., Mohamed, A., Li, S.-W. D., Black, A. W., & Anumanchipalli, G. K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446062 · Cited by 26
“Recent studies show that speech SSL models are highly correlated with articulatory phonetics and their discretized units are fine-grained subphonemic units effectively tiling phonetic space [8, 9, 10].”
Self-citation
Metadata
- Authors
- Cho, C. J., Wu, P., Mohamed, A., & Anumanchipalli, G. K.
- Affiliations
- University of California, Berkeley (US)
- Meta (United States) (US)
- Venue
- ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2023-06-04
- Pages
- 1-5
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W4372348980
- Cited by
- 21
- References
- 27
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 9 · 2023: 1
- FWCI
- 3.73
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2106928)
- Noyce Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- self-supervised speech representation learning, articulatory trajectories, electromagnetic articulography, linear probing, wav2vec 2.0, HuBERT, speech production
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech. https://app.cochat.ai/papers/10.1109/icassp49357.2023.10094711
- The paper
- Cho, C. J., Wu, P., Mohamed, A., & Anumanchipalli, G. K. (2023). Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech. ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10.1109/icassp49357.2023.10094711
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.