The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
Hybrid CTC/Attention Architecture for End-to-End Speech Recognition
Abstract
Conventional automatic speech recognition (ASR) based on a hidden Markov model (HMM)/deep neural network (DNN) is a very complicated system consisting of various modules such as acoustic, lexicon, and language models. It also requires linguistic resources, such as a pronunciation dictionary, tokenization, and phonetic context-dependency trees. On the other hand, end-to-end ASR has become a popular alternative to greatly simplify the model-building process of conventional ASR systems by representing complicated modules with a single deep network architecture, and by replacing the use of linguistic resources with a data-driven learning method. There are two major types of end-to-end architectures for ASR; attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC) uses Markov assumptions to efficiently solve sequential problems by dynamic programming. This paper proposes hybrid CTC/attention end-to-end ASR, which effectively utilizes the advantages of both architectures in training and decoding. During training, we employ the multiobjective learning framework to improve robustness and achieve fast convergence. During decoding, we perform joint decoding by combining both attention-based and CTC scores in a one-pass beam search algorithm to further eliminate irregular alignments. Experiments with English (WSJ and CHiME-4) tasks demonstrate the effectiveness of the proposed multiobjective learning over both the CTC and attention-based encoder-decoder baselines. Moreover, the proposed method is applied to two large-scale ASR benchmarks (spontaneous Japanese and Mandarin Chinese), and exhibits performance that is comparable to conventional DNN/HMM ASR systems based on the advantages of both multiobjective learning and joint decoding without linguistic resources.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 1×
“…th-order Markov assumption) as an m - gram model, i.e., Although recurrent neural network language models (RNNLMs) can avoid this conditional independence assumption issue [23], it makes the decoding complex, and RNNLMs are often combined with m -gram language models based on a rescoring technique.”
MethodGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,609
NLTK: the Natural Language Toolkit
Loper, E., & Bird, S. (2002) · Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics - · doi:10.3115/1118108.1118117 · Cited by 3,218
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
Amodei, D., Anubhai, R., Battenberg, E., Case, C. T., Casper, J., Catanzaro, B., Chen, J., Chrzanowski, M., Coates, A., Diamos, G., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.02595 · Cited by 2,110
Attention-Based Models for Speech Recognition
Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1506.07503 · Cited by 2,021
Towards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 1,861
Hybrid speech recognition with Deep Bidirectional LSTM
Graves, A., Jaitly, N., & Mohamed, A.-R. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707742 · Cited by 1,736
Connectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115
End-to-end attention-based large vocabulary speech recognition
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., & Bengio, Y. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472618 · Cited by 1,096
Continuous speech recognition by statistical methods
Jelinek, F. (1976) · Proceedings of the IEEE · doi:10.1109/proc.1976.10159 · Cited by 920
(1997) · Cited by 914
Joint CTC-attention based end-to-end speech recognition using multi-task learning
Kim, S., Hori, T., & Watanabe, S. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953075 · Cited by 844
Self-citationNLTK: The Natural Language Toolkit
Bird, S. (2006) · Proceedings of the COLING/ACL on Interactive presentation sessions - · doi:10.3115/1225403.1225421 · Cited by 841 · cited 1×
“Finally, some languages do not explicitly have a word boundary and need tokenization modules [3], [4].”
BackgroundPurely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI
Povey, D., Peddinti, V., Galvez, D., Ghahremani, P., Manohar, V., Na, X., Wang, Y., & Khudanpur, S. (2016) · Interspeech 2016 · doi:10.21437/interspeech.2016-595 · Cited by 740
Applying Conditional Random Fields to Japanese Morphological Analysis
Kudo, T., Yamamoto, K., & Matsumoto, Y. (2004) · Cited by 715
EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding
Miao, Y., Gowayyed, M., & Metze, F. (2015) · 2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2015.7404790 · Cited by 707
Connectionist Speech Recognition
Bourlard, H. A., & Morgan, N. (1994) · Springer eBooks · doi:10.1007/978-1-4615-3210-1 · Cited by 585
End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
Chorowski, J., Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.1602 · Cited by 362
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models
Chorowski, J., & Jaitly, N. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-343 · Cited by 358
Chinese Word Segmentation as Character Tagging
Xue, N. (2003) · doi:10.30019/ijclclp.200302.0002 · Cited by 358
An analysis of environment, microphone and data simulation mismatches in robust speech recognition
Vincent, E., Watanabe, S., Nugraha, A. A., Barker, J., & Marxer, R. (2017) · Computer Speech & Language · doi:10.1016/j.csl.2016.11.005 · Cited by 351
Self-citationNeural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
Soltau, H., Liao, H., & Sak, H. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1566 · Cited by 304
Hori, T., Watanabe, S., Zhang, Y., & Chan, W. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1296 · Cited by 294
Self-citationChan, W., Jaitly, N., Le, Q. V., & Vinyals, O. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1508.01211 · Cited by 284
Joint CTC/attention decoding for end-to-end speech recognition
Hori, T., Watanabe, S., & Hershey, J. R. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1048 · Cited by 131
Self-citationLu, L., Zhang, X., & Renals, S. J. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472641 · Cited by 125
HKUST/MTS: A Very Large Scale Mandarin Telephone Speech Corpus
Liu, Y., Fung, P., Yongsheng, Y., Cieri, C., Huang, S., & Graff, D. (2006) · Lecture notes in computer science · doi:10.1007/11939993_73 · Cited by 119
An empirical exploration of CTC acoustic models
Miao, Y., Gowayyed, M. A., Na, X., Ko, T., Metze, F., & Waibel, A. H. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472152 · Cited by 78
Latent Sequence DecompositionsPreprint
Chan, W., Zhang, Y., Le, Q. V., & Jaitly, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03035 · Cited by 61
Maximum a posteriori Based Decoding for CTC Acoustic Models
Kanda, N., Lu, X., & Kawai, H. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-71 · Cited by 41
Multitask Learning with CTC and Segmental CRF for Speech Recognition
Lu, L., Kong, L., Dyer, C., & Smith, N. A. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-71 · Cited by 23
Cited by
SpeechBrain: A General-Purpose Speech Toolkit
Ravanelli, M., Parcollet, T., Plantinga, P., Rouhe, A., Cornell, S., Lugosch, L., Subakan, C., Dawalatabad, N., Heba, A., Zhong, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.04624 · Cited by 512
Recent Advances in End-to-End Automatic Speech Recognition
Li, J. (2022) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000050 · Cited by 395
Attention, please! A survey of neural attention models in deep learning
de Santana Correia, A., & Colombini, E. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10148-x · Cited by 298
The Fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, Task and Baselines
Barker, J., Watanabe, S., Vincent, E., & Trmal, J. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1768 · Cited by 279
Self-citationWang, P.-Y., Chen, C.-T., Su, J.-W., Wang, T.-Y., & Huang, S.-H. (2021) · IEEE Access · doi:10.1109/access.2021.3071306 · Cited by 105
“Watanabe et al. [25] proposed a hybrid model combining CTC and an attention mechanism and imported a multiobjective learning model to increase robustness and convergence.”
Method
Metadata
- Authors
- Watanabe, S., Hori, T., Kim, S., Hershey, J. R., & Hayashi, T.
- Venue
- IEEE Journal of Selected Topics in Signal Processing
- Published
- 2017-12
- Volume
- 11
- Issue
- 8
- Pages
- 1240-1253
- ISSN
- 1932-4553 (ISSN-L)
- 1941-0484
- OpenAlex
- W2766219058
- Cited by
- 800
- References
- 38
- FWCI
- 32.64
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.