CoChat
Sign Up
PreprintCited by 13Speech Recognition and SynthesisSpeech and Audio ProcessingSpeech and dialogue systems

ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition

Open access
Gandhi, Sanchit, von Platen, Patrick, Rush, Alexander M.·2022·arXiv
DOI 10.48550/arxiv.2210.13352arXiv 2210.13352
Open this paper in CoChat — free accountOpen PDF

Abstract

Speech recognition applications cover a range of different audio and text distributions, with different speaking styles, background noise, transcription punctuation and character casing. However, many speech recognition systems require dataset-specific tuning (audio filtering, punctuation removal and normalisation of casing), therefore assuming a-priori knowledge of both the audio and text distributions. This tuning requirement can lead to systems failing to generalise to other datasets and domains. To promote the development of multi-domain speech systems, we introduce the End-to-end Speech Benchmark (ESB) for evaluating the performance of a single automatic speech recognition (ASR) system across a broad set of speech datasets. Benchmarked systems must use the same data pre- and post-processing algorithm across datasets - assuming the audio and text data distributions are a-priori unknown. We compare a series of state-of-the-art (SoTA) end-to-end (E2E) systems on this benchmark, demonstrating how a single speech system can be applied and evaluated on a wide range of data distributions. We find E2E systems to be effective across datasets: in a fair comparison, E2E systems achieve within 2.6% of SoTA systems tuned to a specific dataset. Our analysis reveals that transcription artefacts, such as punctuation and casing, pose difficulties for ASR systems and should be included in evaluation. We believe E2E benchmarking over a range of datasets promotes the research of multi-domain speech recognition systems. ESB is available at https://huggingface.co/esb.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2210.13352

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Evaluating ASR systems on multiple public speech datasets results in an average End-to-End Speech Benchmark (ESB) score.

    “Respectively [Gandhi et al. 2022] propose to evaluate ASR systems on multiple public speech datasets resulting in an average End-to-End Speech Benchmark (ESB) score.”

    From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Gandhi et al. 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 69

  1. Al¨ ena Aks¨ enova, Daan van Esch, James Flynn, and Pavel Golik. How Might We Create Better Benchmarks for Speech Recognition? In Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future , pp. 22-34, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.bppf-1.4. URL https://aclanthology.org/2021. bppf-1.4 . · DOI · Paper page
  2. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common Voice: A Massively-Multilingual Speech Corpus. In Proceedings of the 12th Language Resources and Evaluation Conference , pp. 4218-4222, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4. URL https://www.aclweb.org/ anthology/2020.lrec-1.520 .
  3. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale. arXiv e-prints , art. arXiv:2111.09296, November 2021. · arXiv
  4. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems , volume 33, pp. 12449-12460. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf .
  5. Doug Beeferman, Adam Berger, and John Lafferty. Cyberpunc: A lightweight punctuation annotation system for speech. In In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing , pp. 689-692, 1998.
  6. Jean Carletta. Unleashing the killer corpus: experiences in creating the multi-everything AMI Meeting Corpus. Language Resources and Evaluation , 41(2):181-190, 2007. ISSN 1574-020X. doi: 10.1007/s10579-007-9040-x. · DOI
  7. William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 4960-4964, 2016. doi: 10.1109/ICASSP.2016.7472621. · DOI · Paper page
  8. William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, and Mohammad Norouzi. SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network. arXiv e-prints , art. arXiv:2104.02133, April 2021. · arXiv
  9. C Julian Chen. Speech recognition with automatic punctuation. In Sixth European Conference on Speech Communication and Technology , 1999.
  10. Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Yujun Wang, Zhao You, and Zhiyong Yan. GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio. arXiv e-prints , art. arXiv:2106.06909, June 2021. · arXiv
  11. Jack Choquette, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. NVIDIA A100 Tensor Core GPU: Performance and Innovation. IEEE Micro , 41(2):29-35, 2021. doi: 10.1109/MM.2021.3061394. · DOI · Paper page
  12. Alexis Conneau, Ankur Bapna, Yu Zhang, Min Ma, Patrick von Platen, Anton Lozhkov, Colin Cherry, Ye Jia, Clara Rivera, Mihir Kale, Daan Van Esch, Vera Axelrod, Simran Khanuja, Jonathan H. Clark, Orhan Firat, Michael Auli, Sebastian Ruder, Jason Riesa, and Melvin Johnson. XTREME-S: Evaluating Cross-lingual Speech Representations. arXiv e-prints , art. arXiv:2203.10752, March 2022. · arXiv
  13. Miguel Del Rio, Peter Ha, Quinten McNamara, Corey Miller, and Shipra Chandra. Earnings-22: A Practical Benchmark for Accents in the Wild. arXiv e-prints , art. arXiv:2203.15591, March 2022. · arXiv
  14. Jun Du, Yan-Hui Tu, Lei Sun, Feng Ma, Hai kun Wang, Jia Pan, Cong Liu, Jing-Dong Chen, and Chin-Hui Lee. The USTC-iFlytek system for CHiME-4 challenge. 4th International Workshop on Speech Processing in Everyday Environments , 2016.
  15. Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, and Shyam Ratan (eds.). Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022) , Seattle, United States, July 2022. Association for Computational Linguistics. URL https://aclanthology.org/2022.semeval-1.0 .
  16. Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cer´ on, Keith Achorn, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, and Vijay Janapa Reddi. The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage. arXiv e-prints , art. arXiv:2111.09344, November 2021. · arXiv
  17. Gonc ¸al V. Garc´ es D´ ıaz-Mun´ ıo, Joan Albert Silvestre-Cerd` a, Javier Jorge, Adri` a Gim´ enez, Javier Iranzo-S´ anchez, Pau Baquero-Arnal, Nahuel Rosell´ o, Alejandro P´ erez-Gonz´ alez de Martos, Jorge Civera, Albert Sanchis, and Alfons Juan. Europarl-ASR: A Large Corpus of Parliamentary Debates for Streaming ASR Benchmarking and Speech Data Filtering/Verbatimization. In Proc. Interspeech 2021 , pp. 3695-3699, Brno (Czech Republic), 2021. doi: 10.21437/Interspeech. 2021-1905. · DOI
  18. J.J. Godfrey, E.C. Holliman, and J. McDaniel. SWITCHBOARD: telephone speech corpus for research and development. In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing , volume 1, pp. 517-520 vol.1, 1992. doi: 10.1109/ ICASSP.1992.225858.
  19. Agustin Gravano, Martin Jansche, and Michiel Bacchiani. Restoring punctuation and capitalization in transcribed speech. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing , pp. 4741-4744, 2009. doi: 10.1109/ICASSP.2009.4960690. · DOI
  20. Alex Graves. Sequence Transduction with Recurrent Neural Networks. arXiv e-prints , art. arXiv:1211.3711, November 2012. · arXiv
  21. Alex Graves and Navdeep Jaitly. Towards End-to-End Speech Recognition with Recurrent Neural Networks. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32 , ICML'14, pp. II-1764-II-1772. JMLR.org, 2014.
  22. Alex Graves, Santiago Fern´ andez, Faustino Gomez, and J¨ urgen Schmidhuber. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. In Proceedings of the 23rd International Conference on Machine Learning , ICML '06, pp. 369-376, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 10.1145/1143844.1143891. URL https://doi.org/10.1145/1143844.1143891 . · DOI · Paper page
  23. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmented Transformer for Speech Recognition. arXiv e-prints , art. arXiv:2005.08100, May 2020. · arXiv
  24. Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng. Deep Speech: Scaling up end-to-end speech recognition. arXiv e-prints , art. arXiv:1412.5567, December 2014. · arXiv
  25. Kenneth Heafield. KenLM: Faster and Smaller Language Model Queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation , pp. 187-197, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL https://aclanthology.org/W11-2123 .
  26. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. Scalable Modified Kneser-Ney Language Model Estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pp. 690-696, Sofia, Bulgaria, August 2013. Association for Computational Linguistics. URL https://www.aclweb. org/anthology/P13-2121 .
  27. Franc ¸ois Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Est` eve. TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation. In Speech and Computer , pp. 198-208. Springer International Publishing, 2018.
  28. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 328-339, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1031. URL https://aclanthology.org/ P18-1031 . · DOI · Paper page
  29. Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, and David Patterson. A Domain-Specific Supercomputer for Training Deep Neural Networks. Commun. ACM , 63(7):67-78, jun 2020. ISSN 0001-0782. doi: 10.1145/3360307. URL https://doi.org/10.1145/3360307 . · DOI
  30. Chanwoo Kim and Richard Stern. Robust signal-to-noise ratio estimation based on waveform amplitude distribution analysis. pp. 2598-2601, 09 2008. doi: 10.21437/Interspeech.2008-644. · DOI
  31. Chanwoo Kim and Richard M. Stern. Power-Normalized Cepstral Coefficients (PNCC) for Robust Speech Recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 24 (7):1315-1329, 2016. doi: 10.1109/TASLP.2016.2545928. · DOI
  32. Ji-Hwan Kim and P. C. Woodland. The use of prosody in a combined system for punctuation generation and speech recognition. In Proc. 7th European Conference on Speech Communication and Technology (Eurospeech 2001) , pp. 2757-2760, 2001. doi: 10.21437/Eurospeech.2001-645. · DOI
  33. Ji-Hwan Kim and Philip C Woodland. A combined punctuation generation and speech recognition system and its performance enhancement using prosody. Speech Communication , 41(4):563-577, 2003. ISSN 0167-6393. doi: https://doi.org/10.1016/S0167-6393(03)00049-9. URL https: //www.sciencedirect.com/science/article/pii/S0167639303000499 . · DOI
  34. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR (Poster) , 2015. URL http://arxiv.org/abs/1412.6980 . · arXiv
  35. Kate Knill, Mark Gales, Konstantinos Kyriakopoulos, Andrey Malinin, Anton Ragni, Yu Wang, and Andrew Caines. Impact of ASR performance on free speaking language assessment. In Proc. Interspeech 2018 , pp. 1641-1645. International Speech Communication Association (ISCA), 2018.
  36. Jia Koh, Aqilah Mislan, Kevin Khoo, Brian Ang, Wilson Ang, Charmaine Ng, and Ying-Ying Tan. Building the Singapore English National Speech Corpus. pp. 321-325, 09 2019. doi: 10.21437/ Interspeech.2019-1525.
  37. Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, Patrice Castonguay, Mariya Popova, Jocelyn Huang, and Jonathan M. Cohen. NeMo: a toolkit for building AI applications using Neural Modules. arXiv e-prints , art. arXiv:1909.09577, September 2019. · arXiv
  38. Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pp. 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-2012. URL https://aclanthology.org/D18-2012 . · DOI
  39. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising Sequence-to-Sequence Pretraining for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pp. 7871-7880, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.703. URL https://aclanthology.org/2020.acl-main.703 . · DOI · Paper page
  40. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario ˇ Saˇ sko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Cl´ ement Delangue, Th´ eo Matussi` ere, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, Franc ¸ois Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A Community Library for Natural Language Processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. URL https://aclanthology.org/2021.emnlp-demo.21 .
  41. Xian Li, Changhan Wang, Yun Tang, Chau Tran, Yuqing Tang, Juan Pino, Alexei Baevski, Alexis Conneau, and Michael Auli. Multilingual Speech Translation with Efficient Finetuning of Pretrained Models. arXiv e-prints , art. arXiv:2010.12829, October 2020. · arXiv
  42. Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. Rethinking Evaluation in ASR: Are Our Models Robust Enough? arXiv e-prints , art. arXiv:2010.11745, October 2020. · arXiv
  43. Lucian Vlad Lita, Abe Ittycheriah, Salim Roukos, and Nanda Kambhatla. TRuEcasIng. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1 , ACL '03, pp. 152-159, USA, 2003. Association for Computational Linguistics. doi: 10.3115/1075096.1075116. URL https://doi.org/10.3115/1075096.1075116 . · DOI
  44. Y. Lu, Mark J.F. Gales, Kate M. Knill, P. Manakul, L. Wang, and Y. Wang. Impact of ASR Performance on Spoken Grammatical Error Detection. In Proc. Interspeech 2019 , pp. 1876-1880. International Speech Communication Association (ISCA), 2019. doi: 10.21437/Interspeech. 2019-1706. · DOI
  45. Colin McIntosh and Cambridge University Press. Cambridge Advanced Learner's Dictionary . Cambridge University Press, Cambridge, England, 4 edition, 2015.
  46. Hermann Ney, Ute Essen, and Reinhard Kneser. On structuring probabilistic dependences in stochastic language modelling. Computer Speech & Language , 8(1):1-38, 1994. ISSN 08852308. doi: https://doi.org/10.1006/csla.1994.1001. URL https://www.sciencedirect. com/science/article/pii/S0885230884710011 . · DOI · Paper page
  47. Patrick K. O'Neill, Vitaly Lavrukhin, Somshubra Majumdar, Vahid Noroozi, Yuekai Zhang, Oleksii Kuchaiev, Jagadeesh Balam, Yuliya Dovzhenko, Keenan Freyberg, Michael D. Shulman, Boris Ginsburg, Shinji Watanabe, and Georg Kucsko. SPGISpeech: 5,000 Hours of Transcribed Financial Audio for Fully Formatted End-to-End Speech Recognition. In Proc. Interspeech 2021 , pp. 1434-1438, 2021. doi: 10.21437/Interspeech.2021-1860. · DOI
  48. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pp. 5206-5210, 2015. doi: 10.1109/ICASSP.2015. 7178964. · DOI
  49. Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition. arXiv e-prints , art. arXiv:1904.08779, April 2019. · arXiv
  50. Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ e-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32 , pp. 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf .
  51. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. The Kaldi Speech Recognition Toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and Understanding . IEEE Signal Processing Society, December 2011. IEEE Catalog No.: CFP11SRW-USB.
  52. Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. MLS: A large-scale multilingual dataset for speech research. In Interspeech 2020 . ISCA, oct 2020. doi: 10. 21437/interspeech.2020-2826. URL https://doi.org/10.21437%2Finterspeech. 2020-2826 . · DOI
  53. Steve Renals, Thomas Hain, and Herve Bourlard. Recognition and understanding of meetings the AMI and AMIDA projects. In 2007 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) , pp. 238-247, 2007. doi: 10.1109/ASRU.2007.4430116. · DOI
  54. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Franc ¸ois Crespo, and Dan Dennison. Hidden Technical Debt in Machine Learning Systems. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ 86df7dcfd896fcaf2674f757a2463eba-Paper.pdf .
  55. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1162. URL https: //aclanthology.org/P16-1162 . · DOI · Paper page
  56. Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, Mia X. Chen, Ye Jia, Anjuli Kannan, Tara Sainath, Yuan Cao, Chung-Cheng Chiu, Yanzhang He, Jan Chorowski, Smit Hinsu, Stella Laurenzo, James Qin, Orhan Firat, Wolfgang Macherey, Suyog Gupta, Ankur Bapna, Shuyuan Zhang, Ruoming Pang, Ron J. Weiss, Rohit Prabhavalkar, Qiao Liang, Benoit Jacob, Bowen Liang, HyoukJoong Lee, Ciprian Chelba, S´ ebastien Jean, Bo Li, Melvin Johnson, Rohan Anil, Rajat Tibrewal, Xiaobing Liu, Akiko Eriguchi, Navdeep Jaitly, Naveen Ari, Colin Cherry, Parisa Haghani, Otavio Good, Youlong Cheng, Raziel Alvarez, Isaac Caswell, Wei-Ning Hsu, Zongheng Yang, Kuan-Chieh Wang, Ekaterina Gonina, Katrin Tomanek, Ben Vanik, Zelin Wu, Llion Jones, Mike Schuster, Yanping Huang, Dehao Chen, Kazuki Irie, George Foster, John Richardson, Klaus Macherey, Antoine Bruguier, Heiga Zen, Colin Raffel, Shankar Kumar, Kanishka Rao, David Rybach, Matthew Murray, Vijayaditya Peddinti, Maxim Krikun, Michiel A. U. Bacchiani, Thomas B. Jablin, Rob Suderman, Ian Williams, Benjamin Lee, Deepti Bhatia, Justin Carlson, Semih Yavuz, Yu Zhang, Ian McGraw, Max Galkin, Qi Ge, Golan Pundak, Chad Whipkey, Todd Wang, Uri Alon, Dmitry Lepikhin, Ye Tian, Sara Sabour, William Chan, Shubham Toshniwal, Baohua Liao, Michael Nirschl, and Pat Rondon. Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling. arXiv e-prints , art. arXiv:1902.08295, February 2019. · arXiv
  57. Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. 'end-to-end asr: from supervised to semi-supervised learning with modern architectures. In ICML 2020 Workshop on Self-supervision in Audio and Speech , 2020. URL https://openreview.net/forum? id=OSVxDDc360z .
  58. Zolt´ an T¨ uske, George Saon, and Brian Kingsbury. On the limit of English conversational speech recognition. arXiv e-prints , art. arXiv:2105.00982, May 2021. · arXiv
  59. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .
  60. Emmanuel Vincent, Shinji Watanabe, Aditya Arie Nugraha, Jon Barker, and Ricard Marxer. An analysis of environment, microphone and data simulation mismatches in robust speech recognition. Comput. Speech Lang. , 46(C):535-557, nov 2017. ISSN 0885-2308. doi: 10.1016/j.csl. 2016.11.005. URL https://doi.org/10.1016/j.csl.2016.11.005 . · DOI
  61. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ e-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019a. URL https://proceedings.neurips.cc/paper/2019/ file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf .
  62. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019b. URL https://openreview.net/forum?id= rJ4km2R5t7 .
  63. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pp. 993-1003, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.acl-long.80. URL https://aclanthology.org/2021.acl-long.80 . · DOI
  64. Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung yi Lee. SUPERB: Speech Processing Universal PERformance Benchmark. In Proc. Interspeech 2021 , pp. 1194-1198, 2021. doi: 10.21437/Interspeech.2021-1775. · DOI · Paper page
  65. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´ emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pp. 38-45, Online, October 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/ 2020.emnlp-demos.6 .
  66. Zheng Yuan and Ted Briscoe. Grammatical error correction using neural machine translation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for
  67. Computational Linguistics: Human Language Technologies , pp. 380-386, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v1/N16-1042. URL https://aclanthology.org/N16-1042 . · DOI
  68. Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V. Le, and Yonghui Wu. Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition. arXiv e-prints , art. arXiv:2010.10504, October 2020. · arXiv
  69. Yu Zhang, Daniel S. Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang, Shibo Wang, Zongwei Zhou, Bo Li, Min Ma, William Chan, Jiahui Yu, Yongqiang Wang, Liangliang Cao, Khe Chai Sim, Bhuvana Ramabhadran, Tara N. Sainath, Franc ¸oise Beaufays, Zhifeng Chen, Quoc V. Le, Chung-Cheng Chiu, Ruoming Pang, and Yonghui Wu. BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition. IEEE Journal of Selected Topics in Signal Processing , pp. 1-14, 2022. doi: 10.1109/JSTSP.2022.3182537. · DOI · Paper page

Cited by

Sorted by citations

Authors
Gandhi, S., von Platen, P., & Rush, A. M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2210.13352
arXiv
2210.13352
OpenAlex
W4307322847
Cited by
13
References
0
Citations by year
2025: 1 · 2024: 9 · 2023: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
end-to-end speech recognition, benchmark, data preprocessing, data post-processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review