State-of-the-Art Speech Recognition with Sequence-to-Sequence Models
Open accessThe DOI matches its Crossref record; 32 of the 37 references it cites were found in a registry or an index, and 1 are not checked yet; none of the 11 sources its claims cite is checked for support yet.
Abstract
Attention-based encoder-decoder architectures such as Listen, Attend, and Spell (LAS), subsume the acoustic, pronunciation and language model components of a traditional automatic speech recognition (ASR) system into a single neural network. In previous work, we have shown that such architectures are comparable to state-of-the-art ASR systems on dictation tasks, but it was not clear if such architectures would be practical for more challenging tasks such as voice search. In this work, we explore a variety of structural and optimization improvements to our LAS model which significantly improve performance. On the structural side, we show that word piece models can be used instead of graphemes. We also introduce a multi-head attention architecture, which offers improvements over the commonly-used single-head attention. On the optimization side, we explore synchronous training, scheduled sampling, label smoothing, and minimum word error rate optimization, which are all shown to improve accuracy. We present results with a unidirectional LSTM encoder for streaming recognition. On a 12, 500 hour voice search task, we find that the proposed changes improve the WER from 9.2% to 5.6%, while the best conventional system achieves 6.7%; on a dictation task our model achieves a WER of 4.1% compared to 5% for the conventional system.
Checked 2026-10-11 · 1 reference not checked yet
Overview
CoChat Summary
## TL;DR
This paper shows that a LAS sequence-to-sequence ASR system can beat a strong conventional system on large-scale voice search by combining word pieces, multi-head attention, several optimization tricks, and second-pass language-model rescoring. On voice search, WER improved from 9.2% to 5.6%, and on dictation the model reached 4.1% vs. 5.0% for the conventional system.
## Key findings
- Word piece models (WPM) slightly improved over graphemes: WER went from 9.2% to 9.0% (2.2% relative WERR).
- Adding multi-head attention on top of WPM reduced WER from 9.0% to 8.0% (11.1% relative WERR).
- Optimization changes further improved WER: synchronous training to 7.7%, scheduled sampling to 7.1%, label smoothing to 6.7%, and MWER training to 5.8%; the paper reports an overall 27.5% gain from these optimizations, moving from 8.0% to 5.8%.
- Second-pass rescoring with an external LM improved WER from 5.8% to 5.6% (3.4% relative improvement); overall, the proposed model achieved 5.6% WER on voice search and 4.1% on dictation, compared with 6.7% and 5.0% for the conventional system.
## Methods
- Sequence-to-sequence LAS model with an encoder-attender-decoder architecture; experiments used 5-layer LSTM encoders and a 2-layer LSTM decoder, with both unidirectional and bidirectional variants.
- Data: about 12,500 hours of training data from 15 million English utterances, artificially corrupted with noise/reverberation; evaluation on about 14.8K voice-search utterances and 15.7K dictation utterances.
- Analysis: compared graphemes vs. word pieces, single-head vs. multi-head attention, and several training/decoding strategies including synchronous training, scheduled sampling, label smoothing, MWER training, and second-pass LM rescoring.
## Limitations
- The paper states that the unidirectional LAS system requires the entire utterance to be seen by the encoder before any labels can be decoded, even though encoding is done in a streaming fashion.
- The paper says the second-pass LM is very large: 80 GB, and it still dominates model size.
- Evident from the text: the strongest reported comparison is on voice search and dictation only, so generalization beyond these tasks is not established in this paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 1 Not valid · 9 Not checked · 30 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while a slightly smaller improvement of 28 . 4 %for a bidirectional system.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“including scheduled sampling ( E5 ) gives an additional 7 . 8 %relative improvement in WER”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“label smoothing gives an additional 5 . 6 % relative improvement.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the proposed changes give a 37 . 8 %relative reduction in WER for a unidirectional system”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, we get 13% relative improvement in WER with these structure improvements.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
18 more claims
Create a free account to see all 40 claims and to verify the 10 references still open.
- It may state two separate claims as one. 12
- The quote only partly states this claim. 2
- Probably not a substantial claim. 6
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Speech recognition [23] is also very popular that typically uses sound and linguistic models, e.g., Google Assistant, Cortana, Siri, Alexa, etc. [67], where machine learning methods are used.”
From Machine learning: Algorithms, real-world applications and research directions (2021) · cites this paper as [23]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Self-citationLarge Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., & He, K. (2017) · arXiv · doi:10.48550/arxiv.1706.02677 · Cited by 2,294
Attention-Based Models for Speech Recognition
Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1506.07503 · Cited by 2,021
Self-citationScheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621
Self-citationSequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732 · Cited by 1,426
Sequence Transduction with Recurrent Neural Networks
Graves, A. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.3711 · Cited by 1,170
End-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472618 · Cited by 1,096
Self-citationJapanese and Korean voice search
Schuster, M., & Nakajima, K. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6289079 · Cited by 862
Minimum Phone Error and I-smoothing for improved discriminative training
Povey, D., & Woodland, P. C. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5743665 · Cited by 622
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2016) · International Conference on Learning Representations · Cited by 572
Fast and accurate recurrent neural network acoustic models for speech recognition
Sak, H., Senior, A., Rao, K., & Beaufays, F. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-350 · Cited by 385
Self-citationTowards Better Decoding and Language Model Integration in Sequence to Sequence Models
Chorowski, J., & Jaitly, N. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-343 · Cited by 358
Self-citationRao, K., Sak, H., & Prabhavalkar, R. (2017) · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru.2017.8268935 · Cited by 341
Self-citationA Comparison of Sequence-to-Sequence Models for Speech Recognition
Prabhavalkar, R., Rao, K., Sainath, T. N., Li, B., Johnson, L., & Jaitly, N. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-233 · Cited by 289
Self-citationChan, W., Jaitly, N., Le, Q. V., & Vinyals, O. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.01211 · Cited by 284
Self-citationAn Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model
Kannan, A., Wu, Y., Nguyen, P., Sainath, T. N., Chen, Z., & Prabhavalkar, R. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462682 · Cited by 253
Self-citationLattice-based optimization of sequence classification criteria for neural-network acoustic modeling
Kingsbury, B. (2009) · 2009 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2009.4960445 · Cited by 232
Minimum Word Error Rate Training for Attention-Based Sequence-to-Sequence Models
Prabhavalkar, R., Sainath, T. N., Wu, Y., Nguyen, P., Chen, Z., Chiu, C., & Kannan, A. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461809 · Cited by 201
Self-citationOnline and Linear-Time Attention by Enforcing Monotonic Alignments
Raffel, C., Luong, M.-T., Liu, P. J., Weiss, R. J., & Eck, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.00784 · Cited by 199
Self-citationLower Frame Rate Neural Network Acoustic Models
Pundak, G., & Sainath, T. N. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-275 · Cited by 128
Self-citationRecurrent Neural Aligner: An Encoder-Decoder Neural Network Model for Sequence to Sequence Mapping
Sak, H., Shannon, M., Rao, K., & Beaufays, F. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1705 · Cited by 128
Self-citationAn Online Sequence-to-Sequence Model Using Partial ConditioningPreprint
Jaitly, N., Le, Q. V., Vinyals, O., Sutskever, I., Sussillo, D., & Samy Bengio, A. (2015) · arXiv (Cornell University) · Cited by 68
Self-citationLatent Sequence DecompositionsPreprint
Chan, W., Zhang, Y., Le, Q. V., & Jaitly, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03035 · Cited by 61
Self-citationImproving the Performance of Online Neural Transducer Models
Sainath, T. N., Chiu, C., Prabhavalkar, R., Kannan, A., Wu, Y., Nguyen, P., & Chen, Z. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462366 · Cited by 51
Self-citationNo Need for a Lexicon? Evaluating the Value of the Pronunciation Lexica in End-to-End Models
Sainath, T. N., Prabhavalkar, R., Kumar, S., Lee, S., Kannan, A., Rybach, D., Schogol, V., Nguyen, P., Li, B., Wu, Y., et al. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462380 · Cited by 48
Self-citationBayesian language model interpolation for mobile speech input
Allauzen, C., & Riley, M. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-249 · Cited by 48
Research works printed but not linked: 1
- abadi, 2015, Tensorflow Large-scale machine learning on heterogeneous distributed systems
Cited by
Machine learning: Algorithms, real-world applications and research directions
Sarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00592-x · Cited by 5,583
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558
Self-citationTabNet: Attentive Interpretable Tabular Learning
Arik, S. Ö., & Pfister, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16826 · Cited by 1,771
(2018) · doi:10.21437/interspeech.2018-1456 · Cited by 1,366
Deep Learning for Audio Signal Processing
Purwins, H., Li, B., Virtanen, T., Schluter, J., Chang, S.-Y., & Sainath, T. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2908700 · Cited by 855
Self-citation
Metadata
- Authors
- Chiu, C.-C., Sainath, T. N., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss, R. J., Rao, K., Gonina, E., Jaitly, N., Li, B., Chorowski, J., & Bacchiani, M.
- Affiliations
- Google (United States) (US)
- ORCID
- Venue
- 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2018-04
- Pages
- 4774-4778
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2775304348
- Cited by
- 1,043
- References
- 37
- Citations by year
- 2026: 10 · 2025: 42 · 2024: 53 · 2023: 102 · 2022: 143 · 2021: 230 · 2020: 215 · 2019: 180 · 2018: 67
- FWCI
- 70.66
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Keywords
- speech recognition, sequence-to-sequence models, attention-based encoder-decoder, multi-head attention, scheduled sampling, label smoothing, synchronous training, voice search, dictation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.