Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
The DOI matches its Crossref record; 28 of the 31 references it cites were found in a registry or an index.
Abstract
We present Listen, Attend and Spell (LAS), a neural speech recognizer that transcribes speech utterances directly to characters without pronunciation models, HMMs or other components of traditional speech recognizers. In LAS, the neural network architecture subsumes the acoustic, pronunciation and language models making it not only an end-to-end trained system but an end-to-end model. In contrast to DNN-HMM, CTC and most other models, LAS makes no independence assumptions about the probability distribution of the output character sequences given the acoustic sequence. Our system has two components: a listener and a speller. The listener is a pyramidal recurrent network encoder that accepts filter bank spectra as inputs. The speller is an attention-based recurrent network decoder that emits each character conditioned on all previous characters, and the entire acoustic sequence. On a Google voice search task, LAS achieves a WER of 14.1% without a dictionary or an external language model and 10.3% with language model rescoring over the top 32 beams. In comparison, the state-of-the-art CLDNN-HMM model achieves a WER of 8.0% on the same set.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“To overcome the conditional independent issue of CTC, a commonly used modified version of the CTC model is the RNN-Transducer (RNN-T) [141] in speech recognition.”
From Deep learning for visual speech analysis: A survey (2024) · cites this paper as [141]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“In this section, we will formally describe LAS which accepts acoustic features as inputs and emits English characters as outputs.”
MethodGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“Other neural network architectures have been described in literature with similar motivations, including the hierarchical RNN [17], clockwork RNN [18] and CNN [19].”
MethodLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 1×
“This method is based on the sequence-tosequence learning framework with attention [8, 9, 10, 11, 12, 13].”
MethodSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“Unlike previous approaches, LAS does not make independence assumptions in the label sequence and it does not rely on HMMs. LAS is based on the sequence to sequence learning framework with attention [17, 18, 16, 14, 15].”
MethodSelf-citationNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 2×
“Unlike previous approaches, LAS does not make independence assumptions in the label sequence and it does not rely on HMMs. LAS is based on the sequence to sequence learning framework with attention [17, 18, 16, 14, 15].”
MethodHinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · cited 1×
“Recent work in this area attempts to rectify this disjoint training issue by designing models that are trained end-to-end – from speech directly to transcripts [10, 11, 12, 13, 14, 15].”
BackgroundSelf-citationSpeech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 1×
“Recent work in this area attempts to rectify this disjoint training issue by designing models that are trained end-to-end – from speech directly to transcripts [10, 11, 12, 13, 14, 15].”
BackgroundShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 1×
“The sequence to sequence framework has been used extensively for many applications: machine translation [24, 25], image captioning [26, 27], parsing [28] and conversational modeling [29].”
MethodKaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,609
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Self-citationTowards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 2,557
Self-citationAttention-Based Models for Speech Recognition
Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1506.07503 · Cited by 2,021 · cited 3×
“Unlike previous approaches, LAS does not make independence assumptions in the label sequence and it does not rely on HMMs. LAS is based on the sequence to sequence learning framework with attention [17, 18, 16, 14, 15].”
MethodHybrid speech recognition with Deep Bidirectional LSTM
Graves, A., Jaitly, N., & Mohamed, A.-R. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707742 · Cited by 1,736
Self-citationConvolutional, Long Short-Term Memory, fully connected Deep Neural Networks
Sainath, T. N., Vinyals, O., Senior, A., & Sak, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178838 · Cited by 1,678 · cited 5×
“This increased the amount of audio data by 20 times with a SNR between 5dB and 30dB [22].”
BackgroundMethodInfluentialSelf-citationScheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621 · cited 4×
“ssing and the predictions can suffer because the model was not trained to be resilient to feeding in bad predictions at some time steps. To ameliorate this effect, we use a trick that was proposed in [19]. During training, instead of always feeding in the ground truth transcript for next step prediction, we sometimes…”
BackgroundMethodInfluentialSelf-citationGradientBased Learning Applied to Document Recognition
(2009) · Intelligent Signal Processing · doi:10.1109/9780470544976.ch9 · Cited by 1,135 · cited 1×
“Other neural network architectures have been described in literature with similar motivations, including the hierarchical RNN [17], clockwork RNN [18] and CNN [19].”
MethodEnd-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472618 · Cited by 1,096 · cited 1×
“In our experiments, we find that these components are necessary for LAS to work well.”
MethodEESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding
Miao, Y., Gowayyed, M., & Metze, F. (2015) · 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404790 · Cited by 707 · cited 3×
“CTC has also been applied to end-to-end training with phoneme targets and n-gram language models using FSTs in [26, 27, 6].”
BackgroundMethodSequence-discriminative training of deep neural networks
Veselý, K., Ghoshal, A., Burget, L., & Povey, D. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-548 · Cited by 657
Fast and accurate recurrent neural network acoustic models for speech recognition
Sak, H., Senior, A., Rao, K., & Beaufays, F. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-350 · Cited by 385 · cited 1×
“CTC has also been applied to end-to-end training with phoneme targets and n-gram language models using FSTs in [26, 27, 6].”
MethodEnd-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
Chorowski, J., Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.1602 · Cited by 362 · cited 3×
“At each step the RNN uses its internal state to guide an attention mechanism [10, 11, 12] to compute a “context” vector from the high level features of the listener.”
MethodHierarchical Recurrent Neural Networks for Long-Term Dependencies
Hihi, S. E., & Bengio, Y. (1995) · Cited by 325
Koutník, J., Greff, K., Gomez, F. J., & Schmidhuber, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.3511 · Cited by 262 · cited 1×
“Other neural network architectures have been described in literature with similar motivations, including the hierarchical RNN [17], clockwork RNN [18] and CNN [19].”
MethodLearning acoustic frame labeling for speech recognition with recurrent neural networks
Sak, H., Senior, A., Rao, K., Irsoy, O., Graves, A., Beaufays, F., & Schalkwyk, J. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178778 · Cited by 171 · cited 1×
“CTC has also been applied to end-to-end training with phoneme targets and n-gram language models using FSTs in [26, 27, 6].”
MethodLexicon-Free Conversational Speech Recognition with Neural Networks
Maas, A. L., Xie, Z., Jurafsky, D., & Ng, A. Y. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1038 · Cited by 145 · cited 2×
“There has recently been an explosion in methods for end-to-end trained speech models because of their inherent simplicity compared to current speech recognition systems [2, 24, 6, 25, 14].”
ResultSequence discriminative distributed training of long short-term memory recurrent neural networks
Sak, H., Vinyals, O., Heigold, G., Senior, A., McDermott, E., Monga, R., & Mao, M. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-305 · Cited by 141 · cited 1×
“Two main approaches for this are Connectionist Temporal Classification (CTC) [10] and sequence to sequence models with attention [16].”
MethodSelf-citationKubo, Y., Hori, T., & Nakamura, A. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-14 · Cited by 14 · cited 1×
“In these models, acoustic models are updated based on a WER proxy, while the pronunciation and language models are rarely updated [7], if at all.”
Method(2014) · doi:10.4230/oasics.slate.2021.16 · Cited by 1 · cited 3×
“There has recently been an explosion in methods for end-to-end trained speech models because of their inherent simplicity compared to current speech recognition systems [2, 24, 6, 25, 14].”
BackgroundResult
Cited by
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558
“The three networks LAS-4-1024, LAS-6-1024 and LAS-6-1280 are trained on LibriSpeech 960h with a combination of augmentation policies (None, LB, LD) and training schedules (B/D).”
BackgroundMethodInfluentialSelf-citationA review on the attention mechanism of deep learning
Niu, Z., Zhong, G., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.091 · Cited by 3,379
Tacotron: towards end-to-end speech synthesis
Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., et al. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1452 · Cited by 1,746
“Moreover, unlike end-to-end speech recognition [4] or machine translation [5], TTS outputs are continuous, and output sequences are usually much longer than those of the input.”
BackgroundSelf-citation(2018) · doi:10.21437/interspeech.2018-1456 · Cited by 1,366
Metadata
- Authors
- Chan, W., Jaitly, N., Le, Q., & Vinyals, O.
- Affiliations
- Carnegie Mellon University (US)
- Google (United States) (US)
- Venue
- 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2016-03
- Pages
- 4960-4964
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2327501763
- Cited by
- 2,226
- References
- 31
- Citations by year
- 2026: 46 · 2025: 132 · 2024: 174 · 2023: 260 · 2022: 261 · 2021: 408 · 2020: 387 · 2019: 308 · 2018: 173 · 2017: 62 · 2016: 15
- FWCI
- 100.22
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- end-to-end speech recognition, attention-based decoder, word error rate
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.