CoChat
Sign Up
PreprintCited by 79Speech Recognition and SynthesisSpeech and Audio ProcessingSpeech and dialogue systems

SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network

Open access
Chan, William, Park, Daniel, Lee, Chris, Zhang, Yu, Le, Quoc, Norouzi, Mohammad·2021·arXiv
DOI 10.48550/arxiv.2104.02133arXiv 2104.02133
Open this paper in CoChat — free accountOpen PDF

Abstract

We present SpeechStew, a speech recognition model that is trained on a combination of various publicly available speech recognition datasets: AMI, Broadcast News, Common Voice, LibriSpeech, Switchboard/Fisher, Tedlium, and Wall Street Journal. SpeechStew simply mixes all of these datasets together, without any special re-weighting or re-balancing of the datasets. SpeechStew achieves SoTA or near SoTA results across a variety of tasks, without the use of an external language model. Our results include 9.0\% WER on AMI-IHM, 4.7\% WER on Switchboard, 8.3\% WER on CallHome, and 1.3\% on WSJ, which significantly outperforms prior work with strong external language models. We also demonstrate that SpeechStew learns powerful transfer learning representations. We fine-tune SpeechStew on a noisy low resource speech dataset, CHiME-6. We achieve 38.9\% WER without a language model, which compares to 38.6\% WER to a strong HMM baseline with a language model.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2104.02133

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Training a model on multiple public speech datasets results in comparable accuracy but higher robustness than models trained on a single source.

    “Chan et al. [2021] show that training a model on multiple public speech datasets results in a comparable accuracy but higher robustness than models trained on a single source.”

    From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Chan et al. 2021

    The Introduction (p. 1) describes training one model on combined speech datasets and reports results that compare favourably with prior work; the Experiments (p. 3) include single-domain results for comparison. The paper also reports strong transfer to a new dataset, but it does not establish that the mixed-data model is more robust than models trained on a single source. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Transformer and recurrent neural-network transducer models containing up to one billion parameters have produced a marked decrease in word error rate for automated transcription of spoken English-language sentences.

    “In particular, transformer1 and recurrent neural-network transducer (RNNT)12,13,16 models containing up to one billion parameters2 have produced a marked decrease in word error rate (WER) (and therefore much better accuracy) for the automated transcription of spoken English-language sentences, as shown in Fig. 1a for two widely used datasets, Librispeech17 and SwitchBoard18.”

    From An analog-AI chip for energy-efficient speech recognition and transcription (2023) · cites this paper as [2]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. [1][1] A. Graves and N. Jaitly, 'Towards End-to-End Speech Recognition with Recurrent Neural Networks,' in ICML , 2014. -
  2. [2][2] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, 'Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition,' in ICASSP , 2016. -
  3. [3][3] W. Chan, C. Saharia, G. Hinton, M. Norouzi, and N. Jaitly, 'Imputer: Sequence Modelling via Imputation and Dynamic Programming,' in ICML , 2020. -
  4. [4][4] D. Park, W. Chan, Y. Zhang, C.-C. Chiu, B. Zoph, E. Cubuk, and Q. Le, 'SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,' in INTERSPEECH , 2019. -
  5. [5][5] Y. Zhang, J. Qin, D. S. Park, W. Han, C.-C. Chiu, R. Pang, Q. V. Le, and Y. Wu, 'Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition,' in arXiv:2010.10504 , 2020. - · arXiv
  6. [6][6] C.-C. Chiu, T. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina, N. Jaitly, B. Li, J. Chorowski, and M. Bacchiani, 'State-of-the-art Speech Recognition With Sequence-to-Sequence Models,' in ICASSP , 2018. -
  7. [7][7] S. Watanabe, M. Mandel, J. Barker, E. Vincent, A. Arora, X. Chang, S. Khudanpur, V. Manohar, D. Povey, D. Raj, D. Snyder, A. S. Subramanian, J. Trmal, B. B. Yair, C. Boeddeker, Z. Ni, Y. Fujita, S. Horiguchi, N. Kanda, T. Yoshioka, and N. Ryant, 'CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings,' in arXiv , 2020. -
  8. [8][8] A. Andrusenko, A. Laptev, and I. Medennikov, 'Towards a Competitive End-to-End Speech Recognition for CHiME-6 Dinner Party Transcription,' in arXiv , 2020. -
  9. [9][9] I. Medennikov, M. Korenevsky, T. Prisyach, Y. Khokhlov, M. Korenevskaya1, I. Soroki, T. Timofeeva, A. Mitrofanov, A. Andrusenko, I. Podluzhny, A. Laptev, and A. Romanenko, 'The STC System for the CHiME-6 Challenge,' in CHiME Workshop , 2020. -
  10. [10][10] L. Burget, P. Schwarz, M. Agarwal, P. Akyazi, K. Feng, A. Ghoshal, O. Glembek, N. Goel, M. Karafi´ at, D. Povey, A. Rastrowand, R. C. Rose, and S. Thomas, 'Multilingual Acoustic Modeling for Speech Recognition based on Subspace Gaussian Mixture Models,' in ICASSP , 2010. -
  11. [11][11] G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean, 'Multilingual Acoustic Models Using Distributed Deep Neural Networks,' in ICASSP , 2013. -
  12. [15][15] A. M. M. A. Alexei Baevski, Henry Zhou, 'wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,' in arXiv:2006.11477 , 2020. - · arXiv
  13. [16][16] G. Synnaeve, Q. Xu, J. Kahn, E. Grave, T. Likhomanenko, V. Pratap, A. Sriram, V. Liptchinsky, and R. Collobert, 'Endto-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures,' in ICML: Workshop on Self-supervision in Audio and Speech , 2020. -
  14. [17][17] D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le, 'Improved Noisy Student Training for Automatic Speech Recognition,' in INTERSPEECH , 2020. -
  15. [19][19] S. Kornblith, J. Shlens, and Q. Le, 'Do Better ImageNet Models Transfer Better?' in CVPR , 2019. -
  16. [20][20] B. Li, Y. Zhang, T. Sainath, Y. Wu, and W. Chan, 'Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes,' in ICASSP , 2019. -
  17. [22][22] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby, 'Big Transfer (BiT): General Visual Representation Learning,' in arXiv , 2019. -
  18. [25][25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, 'Attention Is All You Need,' in NIPS , 2017. -
  19. [26][26] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, 'Language Models are Few-Shot Learners,' in NeurIPS , 2020. -
  20. [27][27] P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever, 'Deep Double Descent: Where Bigger Models and More Data Hurt,' in ICLR , 2020. -
  21. [28][28] Y. Zhang, W. Chan, and N. Jaitly, 'Very Deep Convolutional Networks for End-to-End Speech Recognition,' in ICASSP , 2017. -
  22. [30][30] A. Graves, 'Sequence Transduction with Recurrent Neural Networks,' in ICML Representation Learning Workshop , 2012. -
  23. [31][31] J. Carletta, S. Ashby, S. Bourban, M. Flynn, M. Guillemot, T. Hain, J. Kadlec, V. Karaiskos, W. Kraaij, M. Kronenthal, G. Lathoud, M. Lincoln, A. Lisowska, I. McCowan, W. Post, D. Reidsma, and P. Wellner, 'The AMI Meeting Corpus: A PreAnnouncement,' in International Workshop on Machine Learning for Multimodal Interaction , 2005. -
  24. [32][32] R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, 'Common Voice: A Massively-Multilingual Speech Corpus,' in LREC , 2020. -
  25. [34][34] A. Rousseau, P. Del´ eglise, and Y . Est` eve, 'TED-LIUM: an Automatic Speech Recognition dedicated corpus,' in LREC , 2012. -
  26. [36][36] T. Likhomanenko, Q. Xu, V. Pratap, P. Tomasello, J. Kahn, G. Avidov, R. Collobert, and G. Synnaeve, 'Rethinking Evaluation in ASR: Are Our Models Robust Enough?' arXiv preprint arXiv:2010.11745 , 2020. - · arXiv
  27. [37][37] S. Sabour, W. Chan, and M. Norouzi, 'Optimal Completion Distillation for Sequence Learning,' in ICLR , 2019. -
  28. [38][38] N. Kanda, G. Ye, Y. Wu, Y. Gaur, X. Wang, Z. Meng, Z. Chen, and T. Yoshioka, 'Large-Scale Pre-Training of End-to-End MultiTalker ASR for Meeting Transcription with Single Distant Microphone,' in arXiv , 2021. -
  29. [39][39] G. Sun, C. Zhang, and P. C. Woodland, 'Transformer Language Models with LSTM-based Cross-utterance Information Representation,' in arXiv , 2021. -
  30. [40][40] N. Kanda, Y. Fujita, S. Horiguchi, R. Ikeshita, K. Nagamatsu, and S. Watanabe, 'Acoustic Modeling for Distant Multi-talker Speech Recognition with Single- and Multi-channel Branches,' in ICASSP , 2019. -
  31. [41][41] W. Wang, G. Wang, A. Bhatnagar, Y. Zhou, C. Xiong, and R. Socher, 'An investigation of phone-based subword units for end-to-end speech recognition,' in INTERSPEECH , 2020. -
  32. [42][42] W. Zhou, W. Michel, K. Irie, M. Kitza, R. Schl¨ uter, and H. Ney, 'The RWTH ASR System for TED-LIUM Release 2: Improving Hybrid HMM with SpecAugment,' in ICASSP , 2020. -
  33. [43][43] H. Hadian, H. Sameti, D. Povey, and S. Khudanpur, 'End-to-end speech recognition using lattice-free MMI,' in INTERSPEECH , 2018. -
  34. [45][45] A. Graves, 'Practical Variational Inference for Neural Networks,' in NIPS , 2011. -
  35. [46][46] D. S. Park, Y. Zhang, C.-C. Chiu, Y. Chen, B. Li, W. Chan, Q. V. Le, and Y. Wu, 'Specaugment on large scale datasets,' in ICASSP , 2020. -
  36. [47][47] J. Kahn, M. Rivi` ere, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazar´ e, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux, 'Libri-Light: A Benchmark for ASR with Limited or No Supervision,' in arXiv , 2019. -
  37. [48][48] N. Shazeer and M. Stern, 'Adafactor: Adaptive learning rates with sublinear memory cost,' in ICML , 2018. -
  38. [49][49] C. Boeddecker, J. Heitkaemper, J. Schmalenstroeer, L. Drude, J. Heymann, and R. Haeb-Umbach, 'Front-End Processing for the CHiME-5 Dinner Party Scenario,' in CHiME Workshop , 2018. -
  39. [50][50] R. Chojnacka, J. Pelecanos, Q. Wang, and I. L. Moreno, 'SpeakerStew: Scaling to Many Languages with a Triaged Multilingual Text-Dependent and Text-Independent Speaker Verification System,' in arXiv , 2021. -
  40. [52][52] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton, 'Big Self-Supervised Models are Strong Semi-Supervised Learners,' in NeurIPS , 2020. -
  41. [53][53] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, Q. Le, and A. Ng, 'Large Scale Distributed Deep Networks,' in NIPS , 2012. -
  42. [54][54] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, 'Caffe: Convolutional Architecture for Fast Feature Embedding,' in International Conference on Multimedia , 2014.

Cited by

Sorted by citations

Authors
Chan, W., Park, D., Lee, C., Zhang, Y., Le, Q., & Norouzi, M.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2104.02133
arXiv
2104.02133
OpenAlex
W3139918052
Cited by
79
References
8
Citations by year
2025: 3 · 2024: 16 · 2023: 31 · 2022: 14 · 2021: 15
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
speech recognition, transfer learning, low-resource speech, noisy speech, word error rate

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review