The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Unsupervised Cross-Lingual Representation Learning for Speech Recognition
Open accessAbstract
This paper presents XLSR which learns cross-lingual speech representations by pretraining a single model from the raw waveform of speech in multiple languages. We build on wav2vec 2.0 which is trained by solving a contrastive task over masked latent speech representations and jointly learns a quantization of the latents shared across languages. The resulting model is fine-tuned on labeled data and experiments show that cross-lingual pretraining significantly outperforms monolingual pretraining. On the CommonVoice benchmark, XLSR shows a relative phoneme error rate reduction of 72% compared to the best known results. On BABEL, our approach improves word error rate by 16% relative compared to a comparable system. Our approach enables a single multilingual speech recognition model which is competitive to strong individual models. Analysis shows that the latent discrete speech representations are shared across languages with increased sharing for related languages. We hope to catalyze research in low-resource speech understanding by releasing XLSR-53, a large model pretrained in 53 languages.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Depending on the number of languages used for training, XLSR supports 53 languages (XLSR [124]) and 128 languages (XLS-R [125]).”
From Audio Deepfake Detection: What Has Been Achieved and What Lies Ahead (2025) · cites this paper as [124]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Momentum Contrast for Unsupervised Visual Representation LearningPreprint
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. B. (2019) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1911.05722 · Cited by 16,250
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
Self-citationGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Categorical Reparameterization with Gumbel-Softmax
Jang, E., Gu, S., & Poole, B. (2016) · arXiv · doi:10.48550/arxiv.1611.01144 · Cited by 3,616
Product Quantization for Nearest Neighbor Search
Jégou, H., Douze, M., & Schmid, C. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.57 · Cited by 2,956 · cited 1×
“The quantization is based on product quantization (Jegou et al., 2011; Baevski et al., 2020b) by choosing quantized representations from G = 2 codebooks with V = 320 entries each.”
Methodfairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-4009 · Cited by 2,475
Self-citationwav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020) · arXiv · doi:10.48550/arxiv.2006.11477 · Cited by 2,369
Self-citationCross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586
Self-citationwav2vec: Unsupervised Pre-Training for Speech Recognition
Schneider, S., Baevski, A., Collobert, R., & Auli, M. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-1873 · Cited by 1,290
Self-citationCommon Voice: A Massively-Multilingual Speech Corpus
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., & Weber, G. (2019) · arXiv · doi:10.48550/arxiv.1912.06670 · Cited by 905
Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers
Huang, J.-T., Li, J., Yu, D., Deng, L., & Gong, Y. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639081 · Cited by 612
vq-wav2vec: Self-Supervised Learning of Discrete Speech RepresentationsPreprint
Baevski, A., Schneider, S., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05453 · Cited by 500
Self-citationCCNet: Extracting High Quality Monolingual Datasets from Web Crawl DataPreprint
Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., & Grave, É. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.00359 · Cited by 473
Self-citationScalable Modified Kneser-Ney Language Model Estimation
Heafield, K., Pouzyrevsky, I., Clark, J. H., & Koehn, P. (2013) · Cited by 442
An Unsupervised Autoregressive Model for Speech Representation Learning
Chung, Y.-A., Hsu, W.-N., Tang, H., & Glass, J. R. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1473 · Cited by 410
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski, A., Zhou, Y., Mohamed, A., & Auli, M. (2020) · neural information processing systems · Cited by 386
Self-citationMLS: A Large-Scale Multilingual Dataset for Speech Research
Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., & Collobert, R. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2826 · Cited by 361
Self-citationMassively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
Arivazhagan, N., Bapna, A., Firat, O., Lepikhin, D., Johnson, M., Krikun, M., Chen, M. X., Cao, Y., Foster, G., Cherry, C., et al. (2019) · arXiv · doi:10.48550/arxiv.1907.05019 · Cited by 279
Multilingual acoustic models using distributed deep neural networks
Heigold, G., Vanhoucke, V., Senior, A., Nguyen, P., Ranzato, M., Devin, M., & Dean, J. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639348 · Cited by 277
Multilingual Speech Recognition with a Single End-to-End Model
Toshniwal, S., Sainath, T. N., Weiss, R. J., Li, B., Moreno, P. J., Weinstein, E., & Rao, K. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461972 · Cited by 228
Multilingual training of deep neural networks
Ghoshal, A. K., Świętojański, P., & Renals, S. J. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639084 · Cited by 197
Unsupervised Pretraining Transfers Well Across Languages
Riviere, M., Joulin, A., Mazare, P.-E., & Dupoux, E. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054548 · Cited by 186
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
Chung, Y.-A., & Glass, J. R. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2341 · Cited by 182
Multilingual acoustic modeling for speech recognition based on subspace Gaussian Mixture Models
Burget, L., Schwarz, P., Agarwal, M., Akyazi, P., Feng, K., Ghoshal, A. K., Glembek, O., Goel, N. K., Karafiát, M., Povey, D., et al. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5495646 · Cited by 162
Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model
Kannan, A., Datta, A., Sainath, T. N., Weinstein, E., Ramabhadran, B., Wu, Y., Bapna, A., Chen, Z., & Lee, S. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2858 · Cited by 159
Transformers with convolutional context for ASRPreprint
Mohamed, A., Okhonko, D., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11660 · Cited by 151
Self-citationEffectiveness of self-supervised pre-training for speech recognitionPreprint
Baevski, A., Auli, M., & Mohamed, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03912 · Cited by 142
Self-citationThe challenge of realistic music generation: modelling raw audio at scalePreprint
Dieleman, S., van den Oord, A., & Simonyan, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.10474 · Cited by 135
Speech recognition and keyword spotting for low-resource languages : Babel project research at CUED
Gales, M., Knill, K., Ragni, A., & Rath, S. P. (2014) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 123
Cho, J., Baskar, M. K., Li, R., Wiesner, M., Mallidi, S. H., Yalta, N., Karafiát, M., Watanabe, S., & Hori, T. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639655 · Cited by 113
Towards language independent acoustic modeling
Byrne, B., Beyerlein, P., Huerta, J. M., Khudanpur, S. P., Marthi, B., Morgan, J., Peterek, N., Picone, J., Vergyri, D., & Wang, T. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2000.859138 · Cited by 109
Improving Transformer-based Speech Recognition Using Unsupervised Pre-trainingPreprint
Jiang, D., Lei, X., Li, W., Luo, N., Hu, Y., Zou, W., & Li, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09932 · Cited by 94
Automatic Speech Recognition for Under-Resourced Languages: Application to Vietnamese Language
Le, V.-B., & Besacier, L. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2009.2021723 · Cited by 84
VQVAE Unsupervised Unit Discovery and Multi-Scale Code2Spec Inverter for Zerospeech Challenge 2019
Tjandra, A., Şişman, B., Zhang, M., Sakti, S., Li, H., & Nakamura, S. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3232 · Cited by 65
Multilingually trained bottleneck features in spoken language recognition
Fér, R., Matějka, P., Grézl, F., Plchot, O., Veselý, K., & Černocký, J. (2017) · Computer Speech & Language · doi:10.1016/j.csl.2017.06.008 · Cited by 59
Eloff, R., Nortje, A., van Niekerk, B., Govender, A., Nortje, L., Pretorius, A., van Biljon, E., van der Westhuizen, E., van Staden, L., & Kamper, H. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1518 · Cited by 54
An End-to-End Language-Tracking Speech Recognizer for Mixed-Language Speech
Seki, H., Watanabe, S., Hori, T., Roux, J. L., & Hershey, J. R. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462180 · Cited by 51
Transfer Learning of Language-independent End-to-end ASR with Language Model Fusion
Inaguma, H., Cho, J., Baskar, M. K., Kawahara, T., & Watanabe, S. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682918 · Cited by 49
An Unsupervised Autoregressive Model for Speech Representation LearningPreprint
Chung, Y.-A., Hsu, W.-N., Tang, H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03240 · Cited by 45
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from SpeechPreprint
Chung, Y.-A., & Glass, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08976 · Cited by 34
The 2016 BBN Georgian telephone speech keyword spotting system
Alumäe, T., Karakos, D., Hartmann, W., Hsiao, R. W.-H., Zhang, L., Nguyen, L. H. B., Tsakalidis, S., & Schwartz, R. M. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953259 · Cited by 30
Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networks
Ragni, A., Li, Q., Gales, M., & Wang, Y. (2018) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · doi:10.17863/cam.35236 · Cited by 20
Learning Robust and Multilingual Speech RepresentationsPreprint
Kawakami, K., Wang, L., Dyer, C., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.11128 · Cited by 13
Semi-supervised training in low-resource ASR and KWS
Metze, F., Gandhe, A., Miao, Y., Sheikh, Z., Wang, Y., Xu, D., Zhang, H., Kim, J., Lane, I. R., Lee, W. K., et al. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178862 · Cited by 12
VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019Preprint
Tjandra, A., Şişman, B., Zhang, M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11449 · Cited by 11
Ragni, A., Wu, C., Gales, M., Vasilakes, J., & Knill, K. M. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953074 · Cited by 8
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded SpeechPreprint
Harwath, D. F., Hsu, W.-N., & Glass, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09602 · Cited by 8
Flagged first, then by citations
Research works printed but not linked: 2
- Mark JF Gales, Kate M Knill, and Anton Ragni. Low-resource speech recognition and keyword- spotting. In Proc. of SPECOM, pp. 3-19. Springer, 2017.
- Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv, 2019.
Cited by
Sorted by citations
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
Babu, A., Wang, C., Tjandra, A., Lakhotia, K., Xu, Q., Goyal, N., Singh, K., von Platen, P., Saraf, Y., Pino, J., et al. (2022) · Interspeech 2022 · doi:10.21437/interspeech.2022-143 · Cited by 586
Self-citationFLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01519 · Cited by 678
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Witold Pedrycz, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 580
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372
“In [282], a wav2vec 2.0 model was trained on data from multiple different languages and different corpora (Babel, Common Voice, and multilingual LS) jointly, followed by fine-tuning for each individual language.”
MethodInfluentialSelf-citationWang, C., Rivière, M., Lee, A., Wu, A., Talnikar, C., Haziza, D., Williamson, M., Pino, J., & Emmanuel Dupoux, A. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.80 · Cited by 383
Metadata
- Authors
- Conneau, A., Baevski, A., Collobert, R., Mohamed, A., & Auli, M.
- Affiliations
- Meta (United Kingdom) (GB)
- ORCID
- Auli, Michael
- Venue
- Interspeech 2021
- Publisher
- ISCA
- Published
- 2021-08-30
- Pages
- 2426-2430
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W3037057938
- Cited by
- 637
- References
- 47
- Citations by year
- 2026: 60 · 2025: 136 · 2024: 115 · 2023: 155 · 2022: 103 · 2021: 61
- FWCI
- 50.12
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Keywords
- cross-lingual representation learning, multilingual speech recognition, XLS-R, wav2vec 2.0, contrastive learning, shared latent representation, Common Voice, Babel
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
