CoChat
Sign Up
PreprintCited by 2,369Parallel Computing and Optimization TechniquesDistributed systems and fault toleranceCloud Computing and Resource Management

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Open access
Baevski, Alexei, Zhou, Henry, Mohamed, Abdelrahman, Auli, Michael·2020·arXiv
DOI 10.48550/arxiv.2006.11477arXiv 2006.11477
Open this paper in CoChat — free accountOpen PDF

Abstract

We show for the first time that learning powerful representations from speech audio alone followed by fine-tuning on transcribed speech can outperform the best semi-supervised methods while being conceptually simpler. wav2vec 2.0 masks the speech input in the latent space and solves a contrastive task defined over a quantization of the latent representations which are jointly learned. Experiments using all labeled data of Librispeech achieve 1.8/3.3 WER on the clean/other test sets. When lowering the amount of labeled data to one hour, wav2vec 2.0 outperforms the previous state of the art on the 100 hour subset while using 100 times less labeled data. Using just ten minutes of labeled data and pre-training on 53k hours of unlabeled data still achieves 4.8/8.2 WER. This demonstrates the feasibility of speech recognition with limited amounts of labeled data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2006.11477

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A common approach in SER involves using self-supervised training methods to learn high-quality representations from unlabeled data, and then using these representations to replace hand-crafted features to complete downstream tasks.

    “Currently, a common approach involves using self-supervised training methods to learn high-quality representations from unlabeled data, and then using these representations to replace hand-crafted features to complete downstream tasks. This is a research hotspot in the field of speech emotion [49,50,51,52,53].”

    From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [49]

    The Introduction (p. 1) describes learning speech representations from unlabeled audio and fine-tuning them for downstream speech recognition. The cited paper does not establish that this is a common approach or research hotspot in speech emotion recognition, or that the representations replace hand-crafted features. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Empirical evidence underscored the superior performance and generalization of wav2vec 2.0 over its predecessor.

    “Empirical evidence underscored the superior performance and generalization of wav2vec 2.0 over its predecessor.”

    From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [55]

    The Conclusion (p. 8) reports that wav2vec 2.0 outperformed the previous best result on a LibriSpeech setup using 100 times less labeled data, supporting superior performance over earlier methods. The cited passages do not establish superior generalization specifically over its wav2vec predecessor; the comparison in Model (p. 2) describes an architectural difference, not a generalization result. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Fine-tuning is beneficial and can prevent overfitting, promoting better generalization.

    “Some studies have shown that fine-tuning is beneficial and can prevent overfitting, promoting better generalization [14].”

    From Voice Deepfake Detection Using the Self-Supervised Pre-Training Model HuBERT (2023) · cites this paper as [14]

    Section 3.3 (p. 4) reports that fine-tuning uses a masking strategy that delays overfitting and significantly improves final error rates, especially with few labeled examples. It attributes those effects to masking during fine-tuning, not to fine-tuning itself, and says overfitting is delayed rather than prevented. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Baevski et al. developed wav2vec 2.0, a new self-supervised learning framework for speech.

    “Subsequently, Baevski et al. [55] developed a new self-supervised learning framework for speech named wav2vec 2.0.”

    From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [55]

  • W2V2 models are trained with self-supervised learning methods with a large amount of unlabelled speech data, allowing them to learn high-level representations of the speech signal.

    “These models are trained with selfsupervised learning methods with a large amount of unlabelled speech data, allowing to learn high-level representations of the speech signal.”

    From The Vicomtech Audio Deepfake Detection System Based on Wav2vec2 for the 2022 ADD Challenge (2022) · cites this paper as [11]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. [1][1] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv , 2016. -
  2. [2][2] P. Bachman, R. D. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. In Proc. of NeurIPS , 2019. -
  3. [3][3] A. Baevski and M. Auli. Adaptive input representations for neural language modeling. In Proc. of ICLR , 2018. -
  4. [4][4] A. Baevski, M. Auli, and A. Mohamed. Effectiveness of self-supervised pre-training for speech recognition. arXiv , abs/1911.03912, 2019. - · arXiv
  5. [5][5] A. Baevski, S. Schneider, and M. Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In Proc. of ICLR , 2020. -
  6. [6][6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. arXiv , abs/2002.05709, 2020. - · arXiv
  7. [7][7] J. Chorowski, R. J. Weiss, S. Bengio, and A. van den Oord. Unsupervised speech representation learning using wavenet autoencoders. arXiv , abs/1901.08810, 2019. - · arXiv
  8. [8][8] Y. Chung, W. Hsu, H. Tang, and J. R. Glass. An unsupervised autoregressive model for speech representation learning. arXiv , abs/1904.03240, 2019. - · arXiv
  9. [9][9] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv , abs/1810.04805, 2018. - · arXiv
  10. [10][10] S. Dieleman, A. van den Oord, and K. Simonyan. The challenge of realistic music generation: modelling raw audio at scale. arXiv , 2018. -
  11. [11][11] R. Eloff, A. Nortje, B. van Niekerk, A. Govender, L. Nortje, A. Pretorius, E. Van Biljon, E. van der Westhuizen, L. van Staden, and H. Kamper. Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks. arXiv , abs/1904.07556, 2019. - · arXiv
  12. [12][12] A. Fan, E. Grave, and A. Joulin. Reducing transformer depth on demand with structured dropout. In Proc. of ICLR , 2020. -
  13. [17][17] W. Han, Z. Zhang, Y. Zhang, J. Yu, C.-C. Chiu, J. Qin, A. Gulati, R. Pang, and Y . Wu. Contextnet: Improving convolutional neural networks for automatic speech recognition with global context. arXiv , 2020. -
  14. [18][18] D. Harwath, W.-N. Hsu, and J. Glass. Learning hierarchical discrete linguistic units from visually-grounded speech. In Proc. of ICLR , 2020. -
  15. [19][19] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. arXiv , abs/1911.05722, 2019. - · arXiv
  16. [20][20] O. J. Hénaff, A. Razavi, C. Doersch, S. M. A. Eslami, and A. van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv , abs/1905.09272, 2019. - · arXiv
  17. [21][21] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv , 2016. -
  18. [22][22] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Weinberger. Deep networks with stochastic depth. arXiv , 2016. -
  19. [23][23] M. G. A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. of AISTATS , 2010. -
  20. [24][24] E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. arXiv , abs/1611.01144, 2016. - · arXiv
  21. [26][26] D. Jiang, X. Lei, W. Li, N. Luo, Y. Hu, W. Zou, and X. Li. Improving transformer-based speech recognition using unsupervised pre-training. arXiv , abs/1910.09932, 2019. - · arXiv
  22. [27][27] J. Kahn et al. Libri-light: A benchmark for asr with limited or no supervision. In Proc. of ICASSP , 2020. -
  23. [28][28] K. Kawakami, L. Wang, C. Dyer, P. Blunsom, and A. van den Oord. Learning robust and multilingual speech representations. arXiv , 2020. -
  24. [29][29] D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In Proc. of ICLR , 2015. -
  25. [30][30] A. Laptev, R. Korostik, A. Svischev, A. Andrusenko, I. Medennikov, and S. Rybin. You do not need more data: Improving end-to-end speech recognition by text-to-speech data augmentation. arXiv , abs/2005.07157, 2020. - · arXiv
  26. [31][31] M. P. Lewis, G. F. Simon, and C. D. Fennig. Ethnologue: Languages of the world, nineteenth edition. Online version: http://www.ethnologue.com , 2016. -
  27. [32][32] A. H. Liu, T. Tu, H. yi Lee, and L. shan Lee. Towards unsupervised speech recognition and synthesis with quantized speech representation learning. arXiv , 2019. -
  28. [35][35] C. J. Maddison, D. Tarlow, and T. Minka. A* sampling. In Advances in Neural Information Processing Systems , pages 3086-3094, 2014. -
  29. [36][36] I. Misra and L. van der Maaten. Self-supervised learning of pretext-invariant representations. arXiv , 2019. -
  30. [37][37] A. Mohamed, D. Okhonko, and L. Zettlemoyer. Transformers with convolutional context for ASR. arXiv , abs/1904.11660, 2019. - · arXiv
  31. [38][38] M. Ott, S. Edunov, D. Grangier, and M. Auli. Scaling neural machine translation. In Proc. of WMT , 2018. -
  32. [39][39] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations , 2019. -
  33. [41][41] D. S. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. D. Cubuk, and Q. V. Le. Specaugment: A simple data augmentation method for automatic speech recognition. In Proc. of Interspeech , 2019. -
  34. [42][42] D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le. Improved noisy student training for automatic speech recognition. arXiv , abs/2005.09629, 2020. - · arXiv
  35. [44][44] V. Pratap, A. Hannun, Q. Xu, J. Cai, J. Kahn, G. Synnaeve, V. Liptchinsky, and R. Collobert. Wav2letter++: A fast open-source speech recognition system. In Proc. of ICASSP , 2019. -
  36. [46][46] M. Ravanelli, P. Brakel, M. Omologo, and Y. Bengio. Light gated recurrent units for speech recognition. IEEE Transactions on Emerging Topics in Computational Intelligence , 2(2):92-102, 2018. -
  37. [47][47] M. Ravanelli, J. Zhong, S. Pascual, P. Swietojanski, J. Monteiro, J. Trmal, and Y. Bengio. Multi-task self-supervised learning for robust speech recognition. arXiv , 2020. -
  38. [48][48] M. Rivière, A. Joulin, P.-E. Mazaré, and E. Dupoux. Unsupervised pretraining transfers well across languages. arXiv , abs/2002.02848, 2020. - · arXiv
  39. [50][50] M. Schuster and K. Nakajima. Japanese and korean voice search. In Proc. of ICASSP , 2012. -
  40. [51][51] G. Synnaeve, Q. Xu, J. Kahn, T. Likhomanenko, E. Grave, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert. End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures. arXiv , abs/1911.08460, 2020. - · arXiv
  41. [52][52] A. Tjandra, B. Sisman, M. Zhang, S. Sakti, H. Li, and S. Nakamura. Vqvae unsupervised unit discovery and multi-scale code2spec inverter for zerospeech challenge 2019. arXiv , 1905.11449, 2019. - · arXiv
  42. [53][53] A. van den Oord, O. Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems , pages 6306-6315, 2017. -
  43. [55][55] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Proc. of NIPS , 2017. -
  44. [56][56] W. Wang, Q. Tang, and K. Livescu. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. arXiv , 2020. -
  45. [57][57] F. Wu, A. Fan, A. Baevski, Y. N. Dauphin, and M. Auli. Pay less attention with lightweight and dynamic convolutions. In Proc. of ICLR , 2019. -
  46. [58][58] Q. Xu, T. Likhomanenko, J. Kahn, A. Hannun, G. Synnaeve, and R. Collobert. Iterative pseudo-labeling for speech recognition. arXiv , 2020. -
  47. [59][59] N. Zeghidour, N. Usunier, I. Kokkinos, T. Schaiz, G. Synnaeve, and E. Dupoux. Learning filterbanks from raw speech for phone recognition. In Proc. of ICASSP , 2018. -
  48. [60][60] Q. Zhang, H. Lu, H. Sak, A. Tripathi, E. McDermott, S. Koo, and S. Kumar. Transformer transducer: A streamable speech recognition model with transformer encoders and rnn-t loss. arXiv , 2020.

Cited by

Sorted by citations

Authors
Baevski, A., Zhou, H., Mohamed, A., & Auli, M.
Affiliations
Meta (United Kingdom) (GB)
ORCID
Auli, Michael
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2006.11477
arXiv
2006.11477
OpenAlex
W3036601975
Cited by
2,369
References
9
Citations by year
2026: 35 · 2025: 252 · 2024: 512 · 2023: 870 · 2022: 499 · 2021: 175 · 2020: 22 · 2019: 1 · 2017: 1 · 2013: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (2530909)
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Computer Networks and Communications › Distributed systems and fault tolerance
  • Physical Sciences › Computer Science › Information Systems › Cloud Computing and Resource Management
Keywords
eBPF, distributed communication, virtual memory, MapReduce, gradient aggregation, protocol design, resource-constrained systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review