CoChat
Sign Up
Cited by 1,290Speech Recognition and SynthesisSpeech and Audio ProcessingMusic and Audio Processing

wav2vec: Unsupervised Pre-Training for Speech Recognition

Open access
Schneider, Steffen, Baevski, Alexei, Collobert, Ronan, Auli, Michael·2019·Interspeech 2019
DOI 10.21437/interspeech.2019-1873
Open this paper in CoChat — free accountOpen PDF

Abstract

We explore unsupervised pre-training for speech recognition by learning representations of raw audio. wav2vec is trained on large amounts of unlabeled audio data and the resulting representations are then used to improve acoustic model training. We pre-train a simple multi-layer convolutional neural network optimized via a noise contrastive binary classification task. Our experiments on WSJ reduce WER of a strong character-based log-mel filterbank baseline by up to 36% when only a few hours of transcribed data is available. Our approach achieves 2.43% WER on the nov92 test set. This outperforms Deep Speech 2, the best reported character-based system in the literature while using two orders of magnitude less labeled training data.

Integrity signals

Registry

Crossref · 10.21437/interspeech.2019-1873

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 9 cited sources checked · 0 supported · 0 flagged

  • Findingreasoning

    Beyond ten negative samples, performance levels off even as training time grows.

    “Thereafter, performance plateaus while training time increases.”
    Quote in the paperNot a substantial claim
  • Findingown data

    More negative samples improve results only until ten samples are used.

    “Table 3 shows that increasing the number of negative samples only helps up to ten samples.”
    Quote in the paperNot a substantial claim
  • Findingown data

    On the WSJ benchmark, greater amounts of pre-training data yield higher accuracy.

    “Table 1 shows that pre-training on more data leads to better accuracy on the WSJ benchmark.”
    Quote in the paperSubstantial claim
  • Findingown data

    On TIMIT, wav2vec pre-training on Librispeech and WSJ audio can match the best reported result in the literature.

    “On TIMIT, pre-training enables us to match the best reported result in the literature.”
    • Also stated in: § 4.2 PRE-TRAINING FOR TIMIT · plain
      “Table 2 shows that wav2vec pre-training on Librispeech and WSJ audio data can lead to results matching the state of the art.”
    Quote in the paperSubstantial claim
  • Findingown data

    The pre-trained representations markedly outperform the character-based baseline that uses log-mel filterbank features.

    “Pretrained representations can substantially improve performance over our character-based baseline which is trained on log-mel filterbank features.”
    Quote in the paperSubstantial claim
  • May combine two claims. 8
  • The quote only partly states this claim. 1
  • Classified as probably not a substantial claim. 3

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    On the WSJ test set, wav2vec reaches 2.43% WER, beating Deep Speech 2, the best reported character-based system, by 0.67 WER (from 3.1% to 2.43%) while using two orders of magnitude less transcribed training data.

    “Our approach achieves 2.43 % WER on the nov92 test set.This outperforms Deep Speech 2, the best reported character-based system in the literature while using two orders of magnitude less labeled training data.”
    • Also stated in: § 5 CONCLUSIONS · plain
      “Our approach achieves 2.43 % WER on the test set of WSJ, a result that outperforms the next best known character-based speech recognition model in the literature (Amodei et al., 2016) while using two orders of magnitude less transcribed training data.”
    • Also stated in: § 4.1 PRE-TRAINING FOR THE WSJ BENCHMARK · boosted
      “This shows that pre-training on unlabeled audio data can improve over the best character-based approach, Deep Speech 2 (Amodei et al., 2016), by 0.67 WER on nov92.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Amodei et al., 2016

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Adding skip connections in the aggregator aids convergence for the larger wav2vec variant.

    “We found it important to introduce skip connections in the aggregator to help convergence in this case.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Against Hadian et al. (2018), wav2vec matches their phoneme-based model, while wav2vec large exceeds it by 0.37 WER.

    “In comparison to Hadian et al. (2018), wav2vec performs as well as their phoneme-based model and wav2vec large outperforms it by 0.37 WER.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hadian et al., 2018

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    The method helps both low-resource conditions and the setting that uses all WSJ training data.

    “We show that more data for pre-training improves performance and that this approach not only improves resource-poor setups, but also settings where all WSJ training data is used.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Unlabeled audio data is much easier to collect than labeled data.

  • May combine two claims.

    Findingown data

    Forecasting beyond 12 future steps does not improve performance and raises training time.

    “Table 5 also shows that predicting more than 12 steps ahead in the future does not result in better performance and increasing the number of steps increases training time.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    Fine-tuning the embedding network gives no meaningful performance gain but greatly lengthens acoustic model training.

    “Similar to Hénaff et al. (2019), we noticed that fine-tuning the embedding network does not meaningfully improve performance while substantially increasing the acoustic model training time.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hénaff et al., 2019

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    TIMIT accuracy rises steadily as pre-training data grows, peaking with the largest pre-training set.

    “Accuracy steadily increases with more data for pre-training and the best accuracy is achieved with the largest amount of data for pre-training.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Leaving maximum length unrestricted yields an average sequence length near 207k frames and the poorest accuracy.

    “Not restricting the maximum length (None) gives an average sequence length of about 207k frames and results in the worst accuracy.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The wav2vec design uses a fully convolutional network, which allows easier time-wise parallelization on modern hardware than the recurrent models of earlier work.

    “wav2vec relies on a fully convolutional architecture which can be easily parallelized over time on modern hardware compared to recurrent models used in previous work ( §2).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The negative-sample experiment held all factors constant except the count of negative samples.

    “In this experiment, everything is kept equal except for the number of negative samples.”
    Quote in the paperSubstantial claim
10 of 10

  • The paper relies on this without support.

    Unlabeled audio data is much easier to collect than labeled data.

    “This enables exploiting unlabeled audio data which is much easier to collect than labeled data.”

  • The paper relies on this without support.

    Accurately modeling the data distribution p(x) is a common and challenging problem for these approaches.

    “A common problem with these approaches is the requirement to accurately model the data distribution p(x), which is challenging.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Utilizing the causal convolutional aggregator results in a fully convolutional model, increasing the parallelizability of pre-training and resulting in faster training times.

    “As in [15], utilizing the causal convolutional aggregator results in a fully convolutional model, increasing the parallelizability of the pre-training, resulting in faster training times.”

    From Investigating Enhancements to Contrastive Predictive Coding for Human Activity Recognition (2023) · cites this paper as [15]

    Section 2.1 describes causal convolutions in both the encoder and context networks, and the Introduction says the fully convolutional architecture is easier to parallelize over time than recurrent models. The paper does not state or demonstrate that this architecture results in faster pre-training times. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Utilizing multiple causal convolution blocks for autoregressive summarization is simpler, faster to train, and easier to parallelize than recurrent models.

    “we propose to instead utilize multiple causal convolution blocks to perform the autoregressive summarization, due to its simplicity, faster training times, and ease of parallelization that recurrent models do not enjoy.”

    From Investigating Enhancements to Contrastive Predictive Coding for Human Activity Recognition (2023) · cites this paper as [15]

    The Introduction says wav2vec’s fully convolutional architecture can be easily parallelized over time compared with recurrent models. Section 2.1 describes a nine-layer context network using causal convolutions to combine earlier representations. The paper describes the model as simple, but our copy does not establish faster training than recurrent models. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • wav2vec directly trains on speech's raw waveform to extract deep features that represent speech signals.

    “Note that wav2vec directly trains on speech’s raw waveform to extract deep features that represent speech signals.”

    From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [54]

  • Schneider et al. proposed wav2vec, an unsupervised speech pre-training method that uses only unlabeled speech data for self-supervised training to obtain generalized speech features applicable to multiple tasks.

    “For example, Schneider et al. [54] proposed an unsupervised speech pre-training method named wav2vec that only uses unlabeled speech data for self-supervised training to obtain generalized speech features that can be applied to multiple tasks.”

    From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [54]

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. [4]4. Jan Chorowski, Ron J. Weiss, Samy Bengio, and Aäron van den Oord. Unsupervised speech representation learning using wavenet autoencoders. arXiv, abs/1901.08810, 2019. · arXiv
  2. [32]32. Vineel Pratap, Awni Hannun, Qiantong Xu, Jeff Cai, Jacob Kahn, Gabriel Synnaeve, Vitaliy Liptchinsky, and Ronan Collobert. wav2letter++: The fastest open-source speech recognition system. arXiv, abs/1812.07625, 2018. · arXiv
  3. [39]39. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: Lessons learned from the 2015 MS COCO image captioning challenge. arXiv, abs/1609.06647, 2016. · arXiv
  4. [41]41. Yuxin Wu and Kaiming He. Group normalization. arXiv, abs/1803.08494, 2018. · arXiv

Cited by

Sorted by citations

Related

Authors
Schneider, S., Baevski, A., Collobert, R., & Auli, M.
Affiliations
  • Technical University of Munich (DE)
  • Meta (Israel) (IL)
ORCID
Venue
Interspeech 2019
Publisher
ISCA
Published
2019-09-15
Pages
3465-3469
ISSN
2308-457X (ISSN-L)
DOI
10.21437/interspeech.2019-1873
OpenAlex
W2939710050
Cited by
1,290
References
32
Citations by year
2026: 86 · 2025: 169 · 2024: 222 · 2023: 262 · 2022: 221 · 2021: 225 · 2020: 97 · 2019: 7
FWCI
52.61
Type
conference paper
Language
en
Open access
Licence
public-domain
Locations
  • Interspeech (USB) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
Keywords
wav2vec, unsupervised pre-training, speech recognition, audio representation learning, noise contrastive estimation, acoustic model training, convolutional neural networks, Wall Street Journal

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-09-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-09-28.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review