The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Improved Training of End-to-end Attention Models for Speech Recognition
Open accessAbstract
Sequence-to-sequence attention-based models on subword units allow simple open-vocabulary end-to-end speech recognition. In this work, we show that such models can achieve competitive results on the Switchboard 300h and LibriSpeech 1000h tasks. In particular, we report the state-of-the-art word error rates (WER) of 3.54% on the dev-clean and 3.82% on the test-clean evaluation subsets of LibriSpeech. We introduce a new pretraining scheme by starting with a high time reduction factor and lowering it during training, which is crucial both for convergence and final performance. In some experiments, we also use an auxiliary CTC loss function to help the convergence. In addition, we train long short-term memory (LSTM) language models on subword units. By shallow fusion, we report up to 27% relative improvements in WER over the attention baseline without a language model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
(2014) · Cited by 33,419
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
SRILM - an extensible language modeling toolkit
Stolcke, A. (2002) · 7th International Conference on Spoken Language Processing (ICSLP 2002) · doi:10.21437/icslp.2002-303 · Cited by 4,326
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition
Chan, W., Jaitly, N., Le, Q., & Vinyals, O. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472621 · Cited by 2,226
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110
LSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963
Self-citationTowards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 1,861
Improved backing-off for M-gram language modeling
Kneser, R., & Ney, H. (2002) · 1995 International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1995.479394 · Cited by 1,473
Self-citationConnectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models
Chiu, C.-C., Sainath, T. N., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss, R. J., Rao, K., Gonina, E., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462105 · Cited by 1,043
Multiple Object Recognition with Visual Attention
Ba, J., Mnih, V., & Kavukcuoglu, K. (2014) · arXiv · doi:10.48550/arxiv.1412.7755 · Cited by 808
Purely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI
Povey, D., Peddinti, V., Galvez, D., Ghahremani, P., Manohar, V., Na, X., Wang, Y., & Khudanpur, S. (2016) · Interspeech 2016 · doi:10.21437/interspeech.2016-595 · Cited by 740
EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding
Miao, Y., Gowayyed, M., & Metze, F. (2015) · 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404790 · Cited by 707
Modeling Coverage for Neural Machine Translation
Tu, Z., Lu, Z., Liu, Y., Liu, X., & Li, H. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1008 · Cited by 692
Tree-based state tying for high accuracy acoustic modelling
Young, S. J., Odell, J. J., & Woodland, P. C. (1994) · Proceedings of the workshop on Human Language Technology - HLT '94 · doi:10.3115/1075812.1075885 · Cited by 664
On Using Monolingual Corpora in Neural Machine Translation
Gülçehre, Ç., Fırat, O., Xu, K., Cho, K., Barrault, L., Lin, H.-C., Bougares, F., Schwenk, H., & Bengio, Y. (2015) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.48550/arxiv.1503.03535 · Cited by 466
An application of recurrent nets to phone probability estimation
Robinson, A. J. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279192 · Cited by 441
Fast and accurate recurrent neural network acoustic models for speech recognition
Sak, H., Senior, A., Rao, K., & Beaufays, F. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-350 · Cited by 385
End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
Chorowski, J., Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.1602 · Cited by 362
Rao, K., Sak, H., & Prabhavalkar, R. (2017) · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru.2017.8268935 · Cited by 341
Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
Soltau, H., Liao, H., & Sak, H. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1566 · Cited by 304
Hori, T., Watanabe, S., Zhang, Y., & Chan, W. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1296 · Cited by 294
A Comparison of Sequence-to-Sequence Models for Speech Recognition
Prabhavalkar, R., Rao, K., Sainath, T. N., Li, B., Johnson, L., & Jaitly, N. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-233 · Cited by 289
Wav2Letter: an End-to-End ConvNet-based Speech Recognition System
Collobert, R., Puhrsch, C., & Synnaeve, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.03193 · Cited by 230
Exploring neural transducers for end-to-end speech recognition
Battenberg, E., Chen, J., Child, R., Coates, A., Li, Y. G. Y., Liu, H., Satheesh, S., Sriram, A., & Zhu, Z. (2017) · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru.2017.8268937 · Cited by 227
Online and Linear-Time Attention by Enforcing Monotonic Alignments
Raffel, C., Luong, M.-T., Liu, P. J., Weiss, R. J., & Eck, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.00784 · Cited by 199
A comprehensive study of deep bidirectional LSTM RNNS for acoustic modeling in speech recognition
Zeyer, A., Doetsch, P., Voigtlaender, P., Schluter, R., & Ney, H. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952599 · Cited by 170
Self-citationGammatone Features and Feature Combination for Large Vocabulary Speech Recognition
Schlüter, R., Bezrukov, I., Wagner, H., & Ney, H. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366996 · Cited by 158
Self-citationMorphological Inflection Generation with Hard Monotonic Attention
Aharoni, R., & Goldberg, Y. (2016) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p17-1183 · Cited by 134
Lower Frame Rate Neural Network Acoustic Models
Pundak, G., & Sainath, T. N. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-275 · Cited by 128
Recurrent Neural Aligner: An Encoder-Decoder Neural Network Model for Sequence to Sequence Mapping
Sak, H., Shannon, M., Rao, K., & Beaufays, F. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1705 · Cited by 128
Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition
Toshniwal, S., Tang, H., Lu, L., & Livescu, K. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1118 · Cited by 127
Lu, L., Zhang, X., & Renals, S. J. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472641 · Cited by 125
Lexicon-Free Conversational Speech Recognition with Neural Networks
Maas, A. L., Xie, Z., Jurafsky, D., & Ng, A. Y. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1038 · Cited by 120
Direct Acoustics-to-Word Models for English Conversational Speech Recognition
Audhkhasi, K., Ramabhadran, B., Saon, G., Picheny, M., & Nahamoo, D. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-546 · Cited by 118
Advances in all-neural speech recognition
Zweig, G., Yu, C., Droppo, J., & Stolcke, A. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953069 · Cited by 91
The CAPIO 2017 Conversational Speech Recognition SystemPreprint
Han, K. J., Chandrashekaran, A., Kim, J., & Lane, I. R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.00059 · Cited by 81
RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
Zeyer, A., Alkhouli, T., & Ney, H. (2018) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p18-4022 · Cited by 77
Self-citationReturnn: The RWTH extensible training framework for universal recurrent neural networks
Doetsch, P., Zeyer, A., Voigtlaender, P., Kulikov, I., Schlüter, R., & Ney, H. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7953177 · Cited by 65
Self-citationLetter-Based Speech Recognition with Gated ConvNetsPreprint
Liptchinsky, V., Synnaeve, G., & Collobert, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.09444 · Cited by 63
Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence LabellingPreprint
Liu, H., Zhu, Z., Li, X., & Satheesh, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00096 · Cited by 55
RASR/NN: The RWTH neural network toolkit for speech recognition
Wiesler, S., Richard, A., Golik, P., Schlüter, R., & Ney, H. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854207 · Cited by 52
Self-citationCTC in the Context of Generalized Full-Sum HMM Training
Zeyer, A., Beck, E., Schlüter, R., & Ney, H. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1073 · Cited by 45
Self-citationLocal Monotonic Attention Mechanism for End-to-End Speech and Language ProcessingPreprint
Tjandra, A., Sakti, S., & Nakamura, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08091 · Cited by 41
Gaussian Prediction Based Attention for Online End-to-End Speech Recognition
Hou, J., Zhang, S., & Dai, L.-R. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-751 · Cited by 39
Improving End-to-End Speech Recognition with Policy Learning
Zhou, Y., Xiong, C., & Socher, R. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462361 · Cited by 39
An Analysis of “Attention” in Sequence-to-Sequence Models
Prabhavalkar, R., Sainath, T. N., Li, B., Rao, K., & Jaitly, N. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-232 · Cited by 35
The RWTH Aachen Machine Translation System for WMT 2010
Huck, M., Wuebker, J., Schmidt, C., Freitag, M., Peitz, S., Stein, D. J., Dagnelies, A., Mansour, S., Leusch, G., & Ney, H. (2010) · Cited by 30
Self-citationThe RWTH Aachen Machine Translation Systems for IWSLT 2017
Bahar, P., Rosendahl, J., Rossenbach, N., & Ney, H. (2017) · RWTH Publications (RWTH Aachen) · doi:10.18154/rwth-conv-220570 · Cited by 30
Self-citationMcFee, B., McVicar, M., Nieto, O., et al. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.774759 · Cited by 30
Bidirectional Decoder Networks for Attention-Based End-to-End Offline Handwriting Recognition
Doetsch, P., Zeyer, A., & Ney, H. (2016) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr.2016.0074 · Cited by 26
Self-citationInverted Alignments for End-to-End Automatic Speech Recognition
Doetsch, P., Hannemann, M., Schlüter, R., & Ney, H. (2017) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2017.2752691 · Cited by 11
Self-citation(2014) · doi:10.4230/oasics.slate.2021.16 · Cited by 1
Cited by
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558
A Comparative Study on Transformer vs RNN in Speech Applications
Karita, S., Chen, N., Hayashi, T., Hori, T., Inaguma, H., Jiang, Z., Someki, M., Soplin, N. E. Y., Yamamoto, R., Wang, X., et al. (2019) · 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru46091.2019.9003750 · Cited by 727
Recent Advances in End-to-End Automatic Speech Recognition
Li, J. (2022) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000050 · Cited by 395
Attention, please! A survey of neural attention models in deep learning
de Santana Correia, A., & Colombini, E. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10148-x · Cited by 298
Metadata
- Authors
- Zeyer, A., Irie, K., Schlüter, R., & Ney, H.
- Venue
- Interspeech 2018
- Publisher
- ISCA
- Published
- 2018-09-02
- Pages
- 7-11
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2799800213
- Cited by
- 257
- References
- 59
- FWCI
- 20.81
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.