On the Limit of English Conversational Speech Recognition
Open accessAbstract
In our previous work we demonstrated that a single headed attention encoder-decoder model is able to reach state-of-the-art results in conversational speech recognition. In this paper, we further improve the results for both Switchboard 300 and 2000. Through use of an improved optimizer, speaker vector embeddings, and alternative speech representations we reduce the recognition errors of our LSTM system on Switchboard-300 by 4% relative. Compensation of the decoder model with the probability ratio approach allows more efficient integration of an external language model, and we report 5.9% and 11.5% WER on the SWB and CHM parts of Hub5'00 with very simple LSTM models. Our study also considers the recently proposed conformer, and more advanced self-attention based language models. Overall, the conformer shows similar performance to the LSTM; nevertheless, their combination and decoding with an improved LM reaches a new record on Switchboard-300, 5.0% and 10.0% WER on SWB and CHM. Our findings are also confirmed on Switchboard-2000, and a new state of the art is reported, practically reaching the limit of the benchmark.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The state-of-the-art for English automated speech recognition (e.g., smartphone dictation) has an approximate 5% word error rate.
“the state-of-the-art for English automated speech recognition (e.g., smartphone dictation) has an approximate 5% word error rate 28”
From An Accurate and Rapidly Calibrating Speech Neuroprosthesis (2024) · cites this paper as [28]
The Abstract reports a state-of-the-art 5.0% word error rate on the Switchboard portion of a conversational-speech benchmark. The Experimental setup specifies English conversational telephony speech, and the Abstract reports 10.0% on another benchmark portion. These results do not establish an approximate 5% rate for English ASR generally or smartphone dictation. (LLM, conf=0.90)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382 · see citing works
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · see citing works
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,312 · see citing works
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,837 · see citing works
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
Conformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3015 · Cited by 3,150 · see citing works
Perceptual linear predictive (PLP) analysis of speech
Heřmanský, H. (1990) · The Journal of the Acoustical Society of America · doi:10.1121/1.399423 · Cited by 2,550 · see citing works
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294 · see citing works
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226 · see citing works
Deep Speech 2: End-to-End Speech Recognition in English and MandarinPreprint
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,191 · see citing works
(2013) · Cited by 2,056 · see citing works
Attention-Based Models for Speech Recognition
Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1506.07503 · Cited by 2,021 · see citing works
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621 · see citing works
(1992) · Cited by 1,405 · see citing works
Audio augmentation for speech recognition
Ko, T., Peddinti, V., Povey, D., & Khudanpur, S. P. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-711 · Cited by 1,130 · see citing works
Connectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115 · see citing works
Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition
Dong, L., Xu, S., & Xu, B. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462506 · Cited by 1,100 · see citing works
Modeling Coverage for Neural Machine Translation
Tu, Z., Lu, Z., Liu, Y., Liu, X., & Li, H. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1008 · Cited by 702 · see citing works
Speaker adaptation of neural network acoustic models using i-vectors
Saon, G., Soltau, H., Nahamoo, D., & Picheny, M. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707705 · Cited by 586 · see citing works
Self-citationThe Microsoft 2017 Conversational Speech Recognition System
Xiong, W., Wu, L., Alleva, F., Droppo, J., Huang, X., & Stolcke, A. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461870 · Cited by 472 · see citing works
On Using Monolingual Corpora in Neural Machine TranslationPreprint
Gülçehre, Ç., Fırat, O., Xu, K., Cho, K., Barrault, L., Lin, H.-C., Bougares, F., Schwenk, H., & Bengio, Y. (2015) · HAL · doi:10.48550/arxiv.1503.03535 · Cited by 470 · see citing works
Zhang, Q., Han, L., Sak, H., Tripathi, A., McDermott, E., Koo, S., & Kumar, S. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053896 · Cited by 462 · see citing works
English Conversational Telephone Speech Recognition by Humans and Machines
Saon, G., Kurata, G., Sercu, T., Audhkhasi, K., Thomas, S., Dimitriadis, D., Cui, X., Ramabhadran, B., Picheny, M., Lim, L.-L., et al. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-405 · Cited by 364 · see citing works
Self-citationTowards Better Decoding and Language Model Integration in Sequence to Sequence Models
Chorowski, J., & Jaitly, N. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-343 · Cited by 358 · see citing works
Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training
Murray, A. F., & Edwards, P. J. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.317730 · Cited by 201 · see citing works
Self-Attentional Acoustic Models
Sperber, M., Niehues, J., Neubig, G., et al. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1910 · Cited by 174 · see citing works
A Time-Restricted Self-Attention Layer for ASR
Povey, D., Hadian, H., Ghahremani, P., Li, K., & Khudanpur, S. P. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462497 · Cited by 162 · see citing works
Gammatone Features and Feature Combination for Large Vocabulary Speech Recognition
Schlüter, R., Bezrukov, I., Wagner, H., & Ney, H. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366996 · Cited by 158 · see citing works
Elastic spectral distortion for low resource speech recognition with deep neural networks
Kanda, N., Takeda, R., & Obuchi, Y. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707748 · Cited by 115 · see citing works
A Density Ratio Approach to Language Model Fusion in End-to-End Automatic Speech Recognition
McDermott, E., Sak, H., & Variani, E. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003790 · Cited by 106 · see citing works
Advancing RNN Transducer Technology for Speech Recognition
Saon, G., Tuske, Z., Bolaños, D., & Kingsbury, B. E. D. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414716 · Cited by 103 · see citing works
Self-citationThe CAPIO 2017 Conversational Speech Recognition SystemPreprint
Han, K. J., Chandrashekaran, A., Kim, J., & Lane, I. R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.00059 · Cited by 73 · see citing works
Character-Level Language Modeling with Deeper Self-Attention
Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013159 · Cited by 56 · see citing works
Towards better decoding and language model integration in sequence to sequence modelsPreprint
Chorowski, J., & Jaitly, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.02695 · Cited by 56 · see citing works
Single Headed Attention Based Sequence-to-Sequence Model for State-of-the-Art Results on Switchboard
Tüske, Z., Saon, G., Audhkhasi, K., & Kingsbury, B. E. D. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1488 · Cited by 55 · see citing works
Self-citationLearning Multiscale Features Directly from Waveforms
Zhu, Z., Engel, J., & Hannun, A. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-256 · Cited by 51 · see citing works
Sequence Noise Injected Training for End-to-end Speech Recognition
Saon, G., Tüske, Z., Audhkhasi, K., & Kingsbury, B. E. D. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683706 · Cited by 41 · see citing works
Self-citationSemi-supervised maximum mutual information training of deep neural network acoustic models
Manohar, V., Povey, D., & Khudanpur, S. P. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-561 · Cited by 34 · see citing works
Cumulative Adaptation for BLSTM Acoustic Models
Kitza, M., Golik, P., Schlüter, R., & Ney, H. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2162 · Cited by 29 · see citing works
Combining TDNN and HMM in a hybrid system for improved continuous-speech recognition
Dugast, C., Devillers, L. Y., & Aubert, X. L. (1994) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.260364 · Cited by 28 · see citing works
Acoustic Modeling of Speech Waveform Based on Multi-Resolution, Neural Network Signal Processing
Tüske, Z., Schlüter, R., & Ney, H. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461871 · Cited by 26 · see citing works
Self-citationInvestigation on LSTM Recurrent N-gram Language Models for Speech Recognition
Tüske, Z., Schlüter, R., & Ney, H. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2476 · Cited by 21 · see citing works
Self-citationEnd-to-End Spoken Language Understanding Without Full Transcripts
Kuo, H.-K. J., Tüske, Z., Thomas, S., et al. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2924 · Cited by 19 · see citing works
Self-citationCOMBINATION AND JOINT TRAINING OF ACOUSTIC CLASSIFIERS FOR SPEECH RECOGNITION
Kirchhoff, K., & Bilmes, J. (2000) · Cited by 16 · see citing works
Small Energy Masking for Improved Neural Network Training for End-To-End Speech Recognition
Kim, C., Kim, K., & Indurthi, S. R. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054382 · Cited by 7 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," in ICLR, 2015.
- D. Krueger et al., "Zoneout: regularizing RNNs by randomly preserving hidden activations," in ICLR, 2017.
- I. Loshchilov and F. Hutter, "Decoupled weight decay regulariza- tion," in ICLR, 2019.
- K. Irie et al., "Training language models for long-span cross- sentence evaluation," in ASRU, 2019, pp. 419-426.
- A. Hannun et al., "Deep Speech: Scaling up end-to- end speech recognition," 2014. [Online]. Available: http: //arxiv.org/abs/1412.5567
- Y. Nesterov, "A method of solving a convex programming problem with convergence rate O(1/k 2 )," in Soviet Mathematics Doklady, vol. 27, no. 2, 1983, pp. 372-376.
Cited by
Sorted by citations
An Accurate and Rapidly Calibrating Speech Neuroprosthesis
Card, N. S., Wairagkar, M., Iacobacci, C., Hou, X., Singer-Clark, T., Willett, F. R., Kunz, E. M., Fan, C., Vahdati Nia, M., Deo, D. R., et al. (2024) · New England Journal of Medicine · doi:10.1056/nejmoa2314132 · Cited by 238 · see citing works
Zhang, Y., Park, D. S., Han, W., Qin, J., Gulati, A., Shor, J., Jansen, A., Xu, Y., Huang, Y., Wang, S., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3182537 · Cited by 161 · see citing works
Geng, M., Xie, X., Ye, Z., Wang, T., Li, G., Hu, S., Liu, X., & Meng, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3195113 · Cited by 48 · see citing works
An accurate and rapidly calibrating speech neuroprosthesis
Card, N. S., Wairagkar, M., Iacobacci, C., Hou, X., Singer-Clark, T., Willett, F. R., Kunz, E. M., Fan, C., Vahdati Nia, M., Deo, D. R., et al. (2023) · doi:10.1101/2023.12.26.23300110 · Cited by 244 · see citing works
- Authors
- Tüske, Z., Saon, G., & Kingsbury, B.
- Affiliations
- IBM (United States) (US)
- Venue
- Interspeech 2021
- Publisher
- ISCA
- Published
- 2021-08-30
- Pages
- 2062-2066
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W3157428800
- Cited by
- 41
- References
- 51
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 6 · 2023: 15 · 2022: 13 · 2021: 1
- FWCI
- 4.04
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- conversational speech recognition, switchboards, conformational isomers, LSTM, speech representation, word error rate
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
