Towards Better Decoding and Language Model Integration in Sequence to Sequence Models
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.
Abstract
The recently proposed Sequence-to-Sequence (seq2seq) framework advocates replacing complex data processing pipelines, such as an entire automatic speech recognition system, with a single neural network trained in an end-to-end fashion.In this contribution, we analyse an attention-based seq2seq speech recognition system that directly transcribes recordings into characters.We observe two shortcomings: overconfidence in its predictions and a tendency to produce incomplete transcriptions when language models are used.We propose practical solutions to both problems achieving competitive speaker independent word error rates on the Wall Street Journal dataset: without separate language models we reach 10.6% WER, while together with a trigram language model, we reach 6.7% WER.
Overview
CoChat Summary
TL;DR
This paper analyzes two failure modes of an attention-based seq2seq speech recognizer: overconfident character predictions and incomplete transcripts when using language models. It proposes label smoothing plus decode-time coverage handling, reaching 10.6% WER without a separate language model and 6.7% WER with a trigram language model on WSJ.
Key findings
- The authors report that the baseline seq2seq model on WSJ achieved 14.2% WER on eval92, while unigram label smoothing reduced this to 10.6% and temporal/neighborhood smoothing to 10.7% on eval92.
- With an extended trigram language model and coverage, the best reported result was 6.7% WER on eval92; the corresponding dev93 score was 9.7% for temporal label smoothing + coverage.
- They found that model overconfidence makes beam search less effective and that increasing SoftMax temperature can help beam search, but without changing greedy decoding accuracy.
- For incomplete transcripts, simply constraining EOS was not sufficient at large beam widths; promoting coverage helped, while maximizing transcript length alone caused more word insertions and worse overall WER.
Methods
- Attention-based seq2seq speech recognition model built on Listen, Attend and Spell: a multilayer Bi-LSTM listener, an LSTM speller, and attention.
- Experiments were run on the Wall Street Journal dataset, training on si284, validating on dev93, and evaluating on eval92; inputs were 80-dimensional mel-scale filterbanks with first and second differences and per-speaker normalization.
- Main analysis compared decoding strategies and regularizers: SoftMax temperature changes, label smoothing variants (uniform/unigram and neighborhood/temporal), beam search settings, and a decode-time coverage penalty with an extended trigram language model.
Limitations
- The paper states that adding a language model introduces two challenges: model overconfidence makes balancing the beam-search terms difficult, and incomplete transcripts occur unless a coverage term is added.
- The authors note that simply maximizing transcript length yields more word insertion errors and worse overall WER.
- Evident from the text: results are reported on a single dataset (WSJ) with a particular ASR setup, so generalization beyond this setting is not established in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 7 Not checked · 37 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe two shortcomings: overconfidence in its predictions and a tendency to produce incomplete transcriptions when language models are used.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A width 100 beam search with a trigram language model finds the second transcript, which misses the beginning of the utterance.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we didn't observe consistent gains from this extension of the beam search criterion.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Temporal LS | 6.6M | 14.1 | 10.7 |”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, incomplete transcripts were produced unless a recording coverage term was added.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 46 claims and to verify the 9 references still open.
- It may state two separate claims as one. 16
- Probably not a substantial claim. 5
- The quote only partly states this claim. 2
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“Deep learning [1] has led to many breakthroughs including speech and image recognition [2, 3, 4, 5, 6, 7].”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“Networks were trained using 8 asynchronous replica workers each employing the ADAM algorithm [21] with default parameters and the learning rate set initially to 10−3, then reduced to 10−4 and 10−5 after 400k and 500k training steps, respectively.”
MethodRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“Especially, Sequence-to-Sequence (seq2seq) neural networks have proved to be very successful on complex transduction tasks, such as machine translation [8, 9, 10], speech recognition [11, 12, 13], or lip-reading [14].”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“Especially, Sequence-to-Sequence (seq2seq) neural networks have proved to be very successful on complex transduction tasks, such as machine translation [8, 9, 10], speech recognition [11, 12, 13], or lip-reading [14].”
BackgroundHinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 1×
“Past attentions vectors were used as auxiliary inputs in the emitting RNN either directly [34], or as cumulative coverage information [35].”
MethodAsynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 1×
“Smoothing techniques increase the entropy of a model’s predictions, a technique that was used to promote exploration in reinforcement learning [27, 28, 29].”
MethodGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 2×
“Our solution that employs a coverage penalty at decode time only is most similar to the one used by the Google Translation system [10].”
BackgroundMethod
Research works printed but not linked: 2
- [1][1] Y. LeCun, Y. Bengio, and G. Hinton, 'Deep learning,' Nature , vol. 521, no. 7553, pp. 436-444, 2015. -
- [23][23] A. Graves and N. Jaitly, 'Towards End-To-End Speech Recognition with Recurrent Neural Networks,' in ICML '14 , 2014, pp. 17641772. -
Cited by
Items 101–200 of 398
Improving End-to-End Speech Recognition with Policy Learning
Zhou, Y., Xiong, C., & Socher, R. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462361 · Cited by 43
Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Model
Liu, A. H., Lee, H.-Y., & Lee, L.-S. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683602 · Cited by 38
Improving Tail Performance of a Deliberation E2E ASR Model Using a Large Text Corpus
Peyser, C., Mavandadi, S., Sainath, T. N., Apfel, J., Pang, R., & Kumar, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1465 · Cited by 40
Denisov, P., & Vu, N. T. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2456 · Cited by 39
Efficient Attention using a Fixed-Size Memory Representation
Britz, D., Guan, M. Y., & Luong, M.-T. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1040 · Cited by 41
“, 2016), speech recognition (Chan et al., 2015; Chorowski and Jaitly, 2016), image captioning (Xu et al.”
BackgroundSemi-Supervised Learning with Data Augmentation for End-to-End ASR
Weninger, F. J., Mana, F., Gemello, R., Andrés-Ferrer, J., & Zhan, P. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1337 · Cited by 38
Tree-Constrained Pointer Generator for End-to-End Contextual Speech Recognition
Sun, G., Zhang, C., & Woodland, P. C. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9687915 · Cited by 54
Dynamic Search for Inference-Time Alignment in Diffusion ModelsPreprint
Li, X., Uehara, M., Su, X., Scalia, G., Biancalani, T., Regev, A., Levine, S., & Ji, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2503.02039 · Cited by 39
“The decoding strategy, which dictates how sentences are generated from the model, is a critical component of text generation in autoregressive language models [37, 38, 39].”
FixMatch-LS: Semi-supervised skin lesion classification with label smoothing
Zhou, S., Tian, S., Yu, L., Wu, W., Zhang, D., Peng, Z., Zhou, Z., & Wang, J. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.104709 · Cited by 38
Semi-Supervised Speech Recognition via Local Prior MatchingPreprint
Hsu, W.-N., Lee, A., Synnaeve, G., & Hannun, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10336 · Cited by 35
“Neural Network Architecture The proposal model ry|x and the online model qy|x are sequence-to-sequence neural networks (Bahdanau et al., 2016; Chorowski & Jaitly, 2017) with the same time-depth separable (TDS) architecture proposed in Hannun et al. (2019).”
BackgroundMethodLearning Soft Labels via Meta LearningPreprint
Vyas, N., Saxena, S., & Voice, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.09496 · Cited by 34
“This technique has been widely adopted in different problems like image-classification [17, 31], reducing word-error rate [1], improving machine translation [26] etc.”
Bai, Y., Yi, J., Tao, J., Tian, Z., & Wen, Z. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1554 · Cited by 33
Less is More: Improved RNN-T Decoding Using Limited Label Context and Path Merging
Prabhavalkar, R., He, Y., Rybach, D., Campbell, S. M., Narayanan, A., Strohman, T., & Sainath, T. N. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414212 · Cited by 38
A Review of On-Device Fully Neural End-to-End Automatic Speech Recognition Algorithms
Kim, C., Gowda, D., Lee, D., Kim, J., Kumar, A., Kim, S.-S., Garg, A., & Han, C. (2020) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/ieeeconf51394.2020.9443456 · Cited by 44
“To get further performance improvement, researchers have proposed various techniques of incorporating external language models such as shallow-fusion [35], deep-fusion [36], and cold-fusion [37].”
BackgroundStudy of Speech Recognition Using CNN
Pittala, R. B., Tejopriya, B. R., & Pala, E. (2022) · International Conference on Artificial Intelligence and Smart Energy (ICAIS) · doi:10.1109/icais53314.2022.9743083 · Cited by 31
“And all of these nodes are linked to other weights given them as neurons in our biological brain connected to synapses [10].”
BackgroundComparable Study Of Modeling Units For End-To-End Mandarin Speech Recognition
Zou, W., Jiang, D., Zhao, S., Yang, G., & Li, X. (2018) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp.2018.8706661 · Cited by 31
Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog Evaluation
Liang, W., Zou, J., & Zhou, Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.126 · Cited by 43
Optimal Completion Distillation for Sequence LearningPreprint
Sabour, S., Chan, W., & Norouzi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.01398 · Cited by 42
A Comprehensive Survey of Fake Text Detection on Misinformation and LM-Generated Texts
Kwon, S.-C., & Jang, B. (2025) · IEEE Access · doi:10.1109/access.2025.3538805 · Cited by 37
Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement
Eskimez, Ş. E., Wang, X., Tang, M., Yang, H., Zhu, Z., Chen, Z., Wang, H., & Yoshioka, T. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-220 · Cited by 39
“Label-smoothed cross-entropy loss [33] was applied for training.”
MethodMomentum Pseudo-Labeling: Semi-Supervised ASR With Continuously Improving Pseudo-Labels
Higuchi, Y., Moritz, N., Roux, J. L., & Hori, T. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3195367 · Cited by 44
Label smoothing and task-adaptive loss function based on prototype network for few-shot learning
Gao, F., Luo, X., Yang, Z., & Zhang, Q. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.09.018 · Cited by 31
Simple Unsupervised Summarization by Contextual Matching
Zhou, J., & Rush, A. M. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1503 · Cited by 38
“Analysis. fluency model and compare the cluster smoothing (CS) approach with softmax temperature (TEMPx with x being the temperature) commonly used for generation in LM-integrated models (Chorowski and Jaitly, 2016) as well as no adjustment (NA).”
MethodUeno, S., Mimura, M., Sakai, S., & Kawahara, T. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682816 · Cited by 38
“Although a separate language model can be trained with the text data, it is not straightforward for the A2W model to combine it in decoding as in [17, 18] since the vocabulary of the A2W model cannot be updated according to the language model.”
MethodChen, C., Li, R., Hu, Y., Siniscalchi, S. M., Chen, P.-Y., Chng, E., & Yang, C.-H. H. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2402.05457 · Cited by 54
“Despite the shift to hybrid HMM-DNN models, the basic decoding/rescoring structure persisted and led to new fusion approaches for LM integration under the emergence of end-of-end (E2E) ASR models (Chorowski & Jaitly, 2016; Sriram et al., 2017).”
MethodListen, Attend, Spell and Adapt: Speaker Adapted Sequence-to-Sequence ASR
Weninger, F. J., Andrés-Ferrer, J., Li, X., & Zhan, P. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2719 · Cited by 26
Learning Word-Level Confidence for Subword End-To-End ASR
Qiu, D., Li, Q., He, Y., Zhang, Y., Li, B., Cao, L., Prabhavalkar, R., Bhatia, D. S., Li, W., Hu, K., et al. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413966 · Cited by 26
“Changes to teacher-forcing maximum likelihood training such as label smoothing [26] and scheduled sampling [27] can make the output probabilities less peaky, but the values still do not correlate with word accuracy well.”
BackgroundExtending Recurrent Neural Aligner for Streaming End-to-End Speech Recognition in Mandarin
Dong, L., Zhou, S., Chen, W. R., & Xu, B. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1086 · Cited by 25
Li, Z., Cao, S., Zhai, M., Ding, N., Zhang, Z., & Hu, B. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127637 · Cited by 26
Autoregressive Predictive Coding: A Comprehensive Study
Yang, G.-P., Yeh, S.-L., Chung, Y.-A., Glass, J. R., & Tang, H. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3203608 · Cited by 24
“A baseline with 40-dimensional log Mel features as input is also presented and achieves results similar to prior work [71].”
ResultImproving Hybrid CTC/Attention Architecture for Agglutinative Language Speech Recognition
Ren, Z., Yolwas, N., Slamu, W., Cao, R., & Wang, H. (2022) · Sensors · doi:10.3390/s22197319 · Cited by 24
“When decoding, the coverage penalty parameter is added [65], the decoder is made to pay attention to each token of the input sequence x evenly, and some tokens are prevented from being given too much attention.”
Multi-Stream End-to-End Speech Recognition
Li, R., Wang, X., Mallidi, S. H., Watanabe, S., Hori, T., & Heřmanský, H. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2959721 · Cited by 28
A Soft Label Deep Learning to Assist Breast Cancer Target Therapy and Thyroid Cancer Diagnosis
Wang, C., Lin, K., Lin, Y., Khalil, M.-A., Chu, K.-L., & Chao, T. (2022) · Cancers · doi:10.3390/cancers14215312 · Cited by 23
“tion [57], and language translation [58] to improve the performance of the deep learning models.”
BackgroundJOIST: A Joint Speech and Text Streaming Model for ASR
Sainath, T. N., Prabhavalkar, R., Bapna, A., Zhang, Y., Huo, Z., Chen, Z., Li, B., Wang, W., & Strohman, T. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022774 · Cited by 39
“A common solution to this issue is to leverage an external neural language model (LM), trained on a much larger amount of unpaired text data, which can be incorporated into an E2E model during decoding: e.g., shallow fusion [8, 9], or rescoring [7].”
BackgroundRethinking Image Mixture for Unsupervised Visual Representation LearningPreprint
Shen, Z., Liu, Z., Liu, Z., Savvides, M., & Darrell, T. (2020) · arXiv (Cornell University) · Cited by 25
Large-Scale Language Model Rescoring on Long-Form Data
Chen, T., Allauzen, C., Huang, Y., Park, D., Rybach, D., Huang, W., Cabrera, R., Audhkhasi, K., Ramabhadran, B., Moreno, P. J., et al. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096429 · Cited by 22
“Decoding algorithms [10, 11, 12] employ fusion strategies, such as shallow [13], cold [14], deep [15] and component [16] fusion.”
MethodHiguchi, Y., Yan, B., Arora, S., Ogawa, T., Kobayashi, T., & Watanabe, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.402 · Cited by 33
Cif-Based Collaborative Decoding for End-to-End Contextual Speech Recognition
Han, M., Dong, L., Zhou, S., & Xu, B. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9415054 · Cited by 38
Advancing Sequence-to-Sequence Based Speech Recognition
Tüske, Z., Audhkhasi, K., & Saon, G. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3018 · Cited by 20
“The last component corresponds to a modified version of the coverage term of [23, 24], where I is the indicator function, and τ is a tunable parameter.”
BackgroundAdapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognition
Ling, S., Hu, Y., Qian, S., Ye, G., Qian, Y., Gong, Y., Lin, E., & Zeng, M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448204 · Cited by 29
“A straightforward approach is to leverage an external language model (LM) or even a LLM in the beam-search-based inference stage through LM fusion methods such as shallow fusion [3, 4], cold fusion [5, 6], and deep fusion [7].”
MethodA Comparison of Label-Synchronous and Frame-Synchronous End-to-End Models for Speech RecognitionPreprint
Dong, L., Cheng, Y., Wang, J., Zhou, S., Xu, S., Jia, X., & Xu, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.10113 · Cited by 21
“s set to 4.0. We only apply dropout to the SAN, whose attention dropout and residual dropout are all set to 0.2 except the models on CN12000h that is set to 0.1. We use the uniform label smoothing in [30] and set it to 0.2 for both of the ASR models and the language models. Parallel scheduled sampling (PSS) [31] with…”
MethodInfluentialAn Investigation of how Label Smoothing Affects GeneralizationPreprint
Chen, B., Ziyin, L., Wang, Z., & Liang, P. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12648 · Cited by 23
Rapid Optimization for Jailbreaking LLMs via Subconscious Exploitation and EchopraxiaPreprint
Shen, G., Cheng, S., Zhang, K., Tao, G., An, S., Yan, L., Zhang, Z., Ma, S., & Zhang, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.05467 · Cited by 23
“At each time step i , a certain decoding strategy is applied to decode the token x i from the corresponding token distribution [18, 23, 35].”
Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition
Meng, Z., Wu, Y., Kanda, N., Lu, L., Chen, X., Ye, G., Sun, E., Li, J., & Gong, Y. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-2075 · Cited by 19
Data Augmentation for ASR Using TTS Via a Discrete Representation
Ueno, S., Mimura, M., Sakai, S., & Kawahara, T. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688218 · Cited by 32
“With regard to the end-to-end ASR model, the connectionist temporal classification (CTC)-based model [1], the attention-based encoder-decoder model [2], recurrent neural network transducer (RNN-T) [3], and transformer-based model [4, 5] have achieved high performance.”
MethodLearning and Evaluation Methodologies for Polyphonic Music Sequence Prediction With LSTMs
Ycart, A., & Benetos, E. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2987130 · Cited by 19
“contrast, in speech recognition, most system include a language model, and even simple ones such as trigrams can halve the word error rate of a system [34].”
BackgroundSemantic Label Smoothing for Sequence to Sequence Problems
Łukasik, M., Jain, H., Menon, A. K., Kim, S., Bhojanapalli, S., Yu, F., & Kumar, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.405 · Cited by 22
Massively Multilingual Shallow Fusion with Large Language Models
Hu, K., Sainath, T. N., Li, B., Du, N., Huang, Y., Dai, A. M., Zhang, Y., Cabrera, R., Chen, Z., & Strohman, T. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094796 · Cited by 19
“Shallow fusion is a way to incorporate external language models to ASR models during decoding [27, 28].”
MethodWindowed Attention Mechanisms for Speech Recognition
Zhang, S., Loweimi, E., Bell, P., & Renals, S. J. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682224 · Cited by 19
“Thereby, the attention mechanism has an awareness of the location or the coverage [7].”
BackgroundLabel Smoothing and Adversarial RobustnessPreprint
Fu, C., Chen, H., Ruan, N., & Jia, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.08233 · Cited by 22
Machine learning approach to force reconstruction in photoelastic materials
Sergazinov, R., & Kramár, M. (2021) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ac29d5 · Cited by 23
Learning What's Easy: Fully Differentiable Neural Easy-First Taggers
Martins, A. F. T., & Kreutzer, J. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1036 · Cited by 28
“Recently, Chorowski and Jaitly (2016) developed an heuristic with a threshold on the total attention as a “coverage criterion” (see their Eq.”
BackgroundCoarse-to-Fine Nutrition Prediction
Wang, B., Bu, T., Hu, Z., Yang, L., Zhao, Y., & Li, X. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3313638 · Cited by 21
“Extensive research has been conducted on label smoothing [44], [45] and its application across various domains, such as image classifi-cation [46] and speech recognition [47].”
BackgroundDeng, X., & Zhang, Z. M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i8.16885 · Cited by 19
Residual Energy-Based Models for End-to-End Speech Recognition
Li, Q., Zhang, Y., Li, B., Cao, L., & Woodland, P. C. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-690 · Cited by 17
Revisiting Label Smoothing and Knowledge Distillation Compatibility: What was Missing?Preprint
Chandrasegaran, K., Tran, N.-T., Zhao, Y., & Cheung, N. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.14532 · Cited by 48
“…accuracy of deep neural networks across multiple tasks including image classification (He et al., 2019; Real et al., 2019; Zoph et al., 2018; Huang et al., 2019), machine translation (Vaswani et al., 2017) and speech recognition (Chorowski & Jaitly, 2017; Chiu et al., 2018; Pereyra et al., 2017).”
BackgroundLarge Language Models for Dysfluency Detection in Stuttered Speech
Wagner, D., Bayerl, S. P., Baumann, I., Nöth, E., Riedhammer, K., & Bocklet, T. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-2120 · Cited by 24
Improving Automatic Speech Recognition and Speech Translation via Word Embedding Prediction
Chuang, S.-P., Liu, A. H., Sung, T.-W., & Lee, H.-Y. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.3037543 · Cited by 16
“Typical examples include classic language model rescoring methods [14], [15], training an additional text-to-speech model to produce pseudo-paired speech-text data [16]–[18], and adversarial learning involving training an extra criticizing language model [19].”
MethodASR Rescoring and Confidence Estimation with Electra
Futami, H., Inaguma, H., Mimura, M., Sakai, S., & Kawahara, T. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688175 · Cited by 28
Independent Language Modeling Architecture for End-To-End ASR
Pham, V. T., Xu, H., Khassanov, Y., Zeng, Z., Chng, E. S., Ni, C., Ma, B., & Li, H. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054116 · Cited by 16
“The main approaches for E2E ASR are attention-based encoder-decoder [1, 2, 3, 4, 5, 6], Connectionist Temporal Classification (CTC) [7, 8] and the hybrid CTC/attention architectures [9, 10].”
MethodPatel, S. (2023) · International Journal of Science and Research (IJSR) · doi:10.21275/sr23216193027 · Cited by 16
Unsupervised pre-training for sequence to sequence speech recognitionPreprint
Fan, Z., Zhou, S., & Xu, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12418 · Cited by 19
“A common approach to leverage text-only data is training a language model (LM), and fusing the output of decoder and the pre-trained LM. Deep fusion [17] and shallow fusion [18] are two approaches to integrate LM into E2E system.”
Ko, J., Yi, B., & Yun, S.-Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i7.26004 · Cited by 20
“Moreover, several studies (Szegedy et al. 2016; Pereyra et al. 2017; Müller, Kornblith, and Hinton 2019; Chorowski and Jaitly 2017) have validated that LS boosts model generalizability, and Li, Dasarathy, and Berisha (2020) proposed the need for datadependent smoothing.”
BackgroundRevisiting Label Smoothing Regularization with Knowledge Distillation
Wang, J., Zhang, P., He, Q., Li, Y., & Hu, Y. (2021) · Applied Sciences · doi:10.3390/app11104699 · Cited by 18
“[19], LSR has been widely used in image classification [20–22], speech recognition [23], and machine translation [24].”
MethodLanguage model fusion for streaming end to end speech recognitionPreprint
Cabrera, R., Liu, X., Ghodsi, M., Matteson, Z., Weinstein, E., & Kannan, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04487 · Cited by 15
“Most experiments used neural LMs, and some used n-gram fst LMs [1, 16, 12, 17].”
MethodA Semi-Supervised Complementary Joint Training Approach for Low-Resource Speech Recognition
Du, Y., Zhang, J., Fang, X., Wu, M., & Yang, Z. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3313434 · Cited by 15
“For text-only data, the key idea is to train an external language model (LM) for joint decoding by fusing or rescoring [26], [27], [28].”
MethodSelf-Attention Channel Combinator Frontend for End-to-End Multichannel Far-Field Speech Recognition
Gong, R., Quillen, C., Sharma, D., Goderre, A., Laínez, J. E. G., & Milanović, L. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1190 · Cited by 14
Liu, Y., Ma, R., Xu, H., He, Y., Ma, Z., & Zhang, W. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-606 · Cited by 14
Wen, H., & Kang, J. (2022) · Journal of Electrocardiology · doi:10.1016/j.jelectrocard.2022.10.002 · Cited by 14
End-to-End Acoustic Modeling Using Convolutional Neural Networks
Passricha, V., & Aggarwal, R. K. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-818130-0.00002-7 · Cited by 19
A new joint CTC-attention-based speech recognition model with multi-level multi-head attention
Qin, C.-X., Zhang, W., & Qu, D. (2019) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-019-0161-0 · Cited by 14
“Pyramidal encoder + attention + label smoothing [36] 6.”
BackgroundUnderstanding Medical Conversations: Rich Transcription, Confidence Scores & Information Extraction
Soltau, H., Wang, M., Izhak Shafran, J., & Shafey, L. E. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-691 · Cited by 14
A Generalizable and Discriminative Learning Method for Deep EEG-Based Motor Imagery Classification
Huang, X., Zhou, N., & Choi, K. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.760979 · Cited by 14
“This technology has successfully increased the performance of CNN models across multiple tasks, including image classification (Szegedy et al., 2016), speech recognition (Chorowski and Jaitly, 2016), and machine translation (Vaswani et al., 2017).”
Gehrmann, S., Strobelt, H., Rush, A. M., et al. (2019) · doi:10.18653/v1/p19-3 · Cited by 14
Pretraining by Backtranslation for End-to-End ASR in Low-Resource Settings
Wiesner, M., Renduchintala, A., Watanabe, S., Liu, C., Dehak, N., & Khudanpur, S. P. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3254 · Cited by 14
“Bolstering the decoder with a language model (LM) trained on supplemental text data is one such method that improves end-to-end ASR performance [4, 5]; however, more significant gains can be obtained by training on additional synthetically perturbed speech [6, 7, 8], or by multilingual training, which augments the tra…”
MethodRethinking Label Smoothing on Multi-Hop Question Answering
Yin, Z., Wang, Y., Hu, X., Wu, Y., Yan, H. R., Zhang, X., Cao, Z., Huang, X., & Qiu, X. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-6207-5_5 · Cited by 25
Character-Aware Attention-Based End-to-End Speech Recognition
Meng, Z., Gaur, Y., Li, J., & Gong, Y. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9004018 · Cited by 13
“4 A label-smoothed cross-entropy [30] loss is minimized during training.”
BackgroundExploring End-to-End Multi-Channel ASR with Bias Information for Meeting Transcription
Wang, X., Kanda, N., Gaur, Y., Chen, Z., Meng, Z., & Yoshioka, T. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383500 · Cited by 13
Model Unit Exploration for Sequence-to-Sequence Speech Recognition.Preprint
Irie, K., Prabhavalkar, R., Kannan, A., Bruguier, A., Rybach, D., & Nguyen, P. (2019) · arXiv (Cornell University) · Cited by 13
Improving accuracy of rare words for RNN-Transducer through unigram shallow fusionPreprint
Ravi, V., Gu, Y., Gandhe, A., Rastrow, A., Liu, L. L., Filimonov, D., Novotney, S., & Bulyko, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00133 · Cited by 14
Continuous Offline Handwriting Recognition using Deep Learning ModelsPreprint
Sueiras, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.13328 · Cited by 13
End-to-End Spelling Correction Conditioned on Acoustic Feature for Code-Switching Speech Recognition
Zhang, S., Yi, J., Tian, Z., Bai, Y., Tao, J., Liu, X., & Wen, Z. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1242 · Cited by 12
“In order to further improve the ASR performance, various post-processing techniques are used [15, 16, 17, 18, 19].”
BackgroundMethodA group behavior prediction model based on sparse representation and complex message interactions
Li, Q., Hu, B., Xu, W. L., & Xiao, Y. (2022) · Information Sciences · doi:10.1016/j.ins.2022.04.023 · Cited by 12
A Comparison Study on AI Language Detector
Singh, A. (2023) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc57344.2023.10099219 · Cited by 13
“There are different approaches such as generative Pre-trained Transfer [18][19], deep neural networks [20], Convolutional Neural Networks (CNNs) [21] and Recurrent Neural Networks (RNNs) [22].”
MethodLiu, A. H., Sung, T.-W., Chuang, S.-P., Lee, H.-Y., & Lee, L.-S. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053324 · Cited by 12
“End-to-end-ASR-Pytorch examples include the classic language model rescoring methods [6, 7], training an additional text-to-speech model to produce pseudo paired speech-text data [8, 9, 10], and achieving adversarial learning by training an extra criticizing language model [11].”
MethodA Comparative Study on Neural Architectures and Training Methods for Japanese Speech Recognition
Karita, S., Kubo, Y., Bacchiani, M., & Jones, L. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-775 · Cited by 12
Label Smoothing Improves Machine Unlearning
Di, Z. (2026) · Synthesis lectures on computer vision · doi:10.1007/978-3-032-17282-2_8 · Cited by 13
Commonsense Justification for Action Explanation
Yang, S., Gao, Q., Saba-Sadiya, S., & Chai, J. Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1283 · Cited by 16
“The temperature based softmax distribution (Chorowski and Jaitly, 2016) was used to sample the easy and the hard distracting verbs based on the pre-trained GloVe (Pennington et al., 2014) embedding cosine similarity.”
MethodUsted: Improving ASR with a Unified Speech and Text Encoder-Decoder
Yusuf, B., Gandhe, A., & Sokolov, A. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746554 · Cited by 13
ReFixMatch-LS: reusing pseudo-labels for semi-supervised skin lesion classification
Zhou, S., Tian, S., Yu, L., Wu, W., Zhang, D., Peng, Z., & Zhou, Z. (2023) · Medical & Biological Engineering & Computing · doi:10.1007/s11517-022-02743-5 · Cited by 11
“Label smoothing has been widely used successfully in a series of tasks in deep learning models, including image classification [21], speech recognition [22], and machine translation [23].”
MethodPhone-Informed Refinement of Synthesized Mel Spectrogram for Data Augmentation in Speech Recognition
Ueno, S., & Kawahara, T. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746582 · Cited by 11
“Among the end-to-end ASR models, the connectionist temporal classification (CTC)-based model [1], the attention-based encoder-decoder model [2], recurrent neural network transducer (RNN-T) [3], and transformer-based models [4, 5] have been investigated.”
2DHeadPose: A simple and effective annotation method for the head pose in RGB images and its dataset
Wang, Y., Zhou, W., & Zhou, J. (2023) · Neural Networks · doi:10.1016/j.neunet.2022.12.021 · Cited by 11
Multi-encoder multi-resolution framework for end-to-end speech recognitionPreprint
Li, R., Wang, X., Mallidi, S. H., Hori, T., Watanabe, S., & Heřmanský, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.04897 · Cited by 11
Contextual Spelling Correction with Large Language Models
Song, G. H., Wu, Z., Pundak, G., Chandorkar, A., Joshi, K., Velez, X., Caseiro, D., Haynor, B., Wang, W., Siddhartha, N., et al. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389637 · Cited by 19
“Using shallow fusion with external language models (LMs) can significantly improve E2E ASR models’ recognition accuracy on rare words [37, 38].”
BackgroundDo End-to-End Speech Recognition Models Care About Context?
Borgholt, L., Havtorn, J. D., Agić, Ż., Søgaard, A., Maaløe, L., & Igel, C. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1750 · Cited by 11
Self-and-Mixed Attention Decoder with Deep Acoustic Structure for Transformer-Based LVCSR
Zhou, X., Lee, G., Yılmaz, E., Long, Y., Liang, J., & Li, H. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2556 · Cited by 10
CACnet: Cube Attentional CNN for Automatic Speech Recognition
Zhang, N., Wang, J., Wei, W., Qu, X., Cheng, N., & Xiao, J. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533666 · Cited by 10
Transformer-Based Lip-Reading with Regularized Dropout and Relaxed Attention
Li, Z., Lohrenz, T., Dunkelberg, M., & Fingscheidt, T. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10023442 · Cited by 13
“Recently, overconfidence caused by the sparse softmax distribution has been observed in attention-based neural networks [34, 35, 36].”
BackgroundCIF: Continuous Integrate-and-Fire for End-to-End Speech RecognitionPreprint
Dong, L., & Xu, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11235 · Cited by 10
Mixture Content Selection for Diverse Sequence GenerationPreprint
Cho, J., Seo, M., & Hajishirzi, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01953 · Cited by 10
Metadata
- Authors
- Chorowski, J., & Jaitly, N.
- Affiliations
- Google (United States) (US)
- ORCID
- Chorowski, Jan
- Venue
- Interspeech 2017
- Publisher
- ISCA
- Published
- 2017-08-20
- Pages
- 523-527
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2577366047
- Cited by
- 358
- References
- 34
- Citations by year
- 2026: 4 · 2025: 14 · 2024: 13 · 2023: 39 · 2022: 26 · 2021: 73 · 2020: 88 · 2019: 49 · 2018: 40 · 2017: 12
- FWCI
- 23.58
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Keywords
- attention-based models, language model integration, trigram language model, end-to-end speech recognition, speaker-independent recognition, Wall Street Journal
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.