The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
End-to-end attention-based large vocabulary speech recognition
Open accessAbstract
Many state-of-the-art Large Vocabulary Continuous Speech Recognition (LVCSR) Systems are hybrids of neural networks and Hidden Markov Models (HMMs). Recently, more direct end-to-end methods have been investigated, in which neural architectures were trained to model sequences of characters [1,2]. To our knowledge, all these approaches relied on Connectionist Temporal Classification [3] modules. We investigate an alternative method for sequence modelling based on an attention mechanism that allows a Recurrent Neural Network (RNN) to learn alignments between sequences of input frames and output labels. We show how this setup can be applied to LVCSR by integrating the decoding RNN with an n-gram language model and by speeding up its operation by constraining selections made by the attention mechanism and by reducing the source sequence lengths by pooling information over time. Recognition accuracies similar to other HMM-free RNN-based approaches are reported for the Wall Street Journal corpus.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 6×
“In this work we use GRU for it has a simpler architecture and is easier to implement efficiently.”
BackgroundMethodInfluentialSelf-citationSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Self-citationHinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · cited 6×
“RNN Transducers have led to state-of-the-art results in phoneme recognition on the TIMIT dataset (Graves et al., 2013) which were recently matched by an ASRG network (Chorowski et al., 2015).”
BackgroundMethodInfluentialShow, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Self-citationImproving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317
Self-citationGenerating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894
Recurrent Models of Visual Attention
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 2,568
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 2,557
Sequence Transduction with Recurrent Neural Networks
Graves, A. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.3711 · Cited by 1,170
Theano: A CPU and GPU Math Compiler in Python
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., & Bengio, Y. (2010) · Proceedings of the Python in Science Conference · doi:10.25080/majora-92bf1922-003 · Cited by 1,052
Self-citationTheano: new features and speed improvements
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I., Bergeron, A., Bouchard, N., Warde-Farley, D., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5590 · Cited by 1,005 · cited 2×
“The experiments were conducted using Theano [26, 27], Blocks and Fuel [28] libraries.”
MethodSelf-citationWeighted finite-state transducers in speech recognition
Mohri, M., Pereira, F., & Riley, M. (2002) · Computer Speech & Language · doi:10.1006/csla.2001.0184 · Cited by 885
EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding
Miao, Y., Gowayyed, M., & Metze, F. (2015) · 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404790 · Cited by 707
Gated Feedback Recurrent Neural Networks
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.02367 · Cited by 583 · cited 2×
“In the spirit of he Clockwork RNN (Koutnik et al., 2014) and hierarchical gating RNN (Chung et al., 2015), we introduce a recurrent architecture that successively reduces source sequence length by pooling frames neighboring in time.”
MethodSelf-citationOpenFst: A General and Efficient Weighted Finite-State Transducer Library
Allauzen, C., Riley, M., Schalkwyk, J., Skut, W., & Mohri, M. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-76336-9_3 · Cited by 539
Review of Neural Networks for Speech Recognition
Lippmann, R. P. (1989) · Neural Computation · doi:10.1162/neco.1989.1.1.1 · Cited by 477
On Using Monolingual Corpora in Neural Machine Translation
Gülçehre, Ç., Fırat, O., Xu, K., Cho, K., Barrault, L., Lin, H.-C., Bougares, F., Schwenk, H., & Bengio, Y. (2015) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.48550/arxiv.1503.03535 · Cited by 466
Self-citationEnd-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
Chorowski, J., Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.1602 · Cited by 362
Self-citationKoutník, J., Greff, K., Gomez, F. J., & Schmidhuber, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.3511 · Cited by 262
First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNsPreprint
Hannun, A., Maas, A. L., Jurafsky, D. S., & Ng, A. Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.2873 · Cited by 204
Blocks and Fuel: Frameworks for deep learningPreprint
van Merriënboer, B., Bahdanau, D., Dumoulin, V., Serdyuk, D., Warde-Farley, D., Chorowski, J., & Bengio, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00619 · Cited by 171
Self-citationHigh-dimensional sequence transduction
Boulanger-Lewandowski, N., Bengio, Y., & Vincent, P. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638244 · Cited by 100
Self-citation
Cited by
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110
Norouzzadeh, M. S., Nguyen, A., Kosmala, M., Swanson, A., Palmer, M. S., Packer, C., & Clune, J. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1719367115 · Cited by 1,230
Metadata
- Authors
- Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y.
- Venue
- 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2016-03
- Pages
- 4945-4949
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2109886035
- Cited by
- 1,096
- References
- 28
- FWCI
- 79.87
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.