Towards Better Decoding and Language Model Integration in Sequence to Sequence Models
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.
Abstract
The recently proposed Sequence-to-Sequence (seq2seq) framework advocates replacing complex data processing pipelines, such as an entire automatic speech recognition system, with a single neural network trained in an end-to-end fashion.In this contribution, we analyse an attention-based seq2seq speech recognition system that directly transcribes recordings into characters.We observe two shortcomings: overconfidence in its predictions and a tendency to produce incomplete transcriptions when language models are used.We propose practical solutions to both problems achieving competitive speaker independent word error rates on the Wall Street Journal dataset: without separate language models we reach 10.6% WER, while together with a trigram language model, we reach 6.7% WER.
Overview
CoChat Summary
TL;DR
This paper analyzes two failure modes of an attention-based seq2seq speech recognizer: overconfident character predictions and incomplete transcripts when using language models. It proposes label smoothing plus decode-time coverage handling, reaching 10.6% WER without a separate language model and 6.7% WER with a trigram language model on WSJ.
Key findings
- The authors report that the baseline seq2seq model on WSJ achieved 14.2% WER on eval92, while unigram label smoothing reduced this to 10.6% and temporal/neighborhood smoothing to 10.7% on eval92.
- With an extended trigram language model and coverage, the best reported result was 6.7% WER on eval92; the corresponding dev93 score was 9.7% for temporal label smoothing + coverage.
- They found that model overconfidence makes beam search less effective and that increasing SoftMax temperature can help beam search, but without changing greedy decoding accuracy.
- For incomplete transcripts, simply constraining EOS was not sufficient at large beam widths; promoting coverage helped, while maximizing transcript length alone caused more word insertions and worse overall WER.
Methods
- Attention-based seq2seq speech recognition model built on Listen, Attend and Spell: a multilayer Bi-LSTM listener, an LSTM speller, and attention.
- Experiments were run on the Wall Street Journal dataset, training on si284, validating on dev93, and evaluating on eval92; inputs were 80-dimensional mel-scale filterbanks with first and second differences and per-speaker normalization.
- Main analysis compared decoding strategies and regularizers: SoftMax temperature changes, label smoothing variants (uniform/unigram and neighborhood/temporal), beam search settings, and a decode-time coverage penalty with an extended trigram language model.
Limitations
- The paper states that adding a language model introduces two challenges: model overconfidence makes balancing the beam-search terms difficult, and incomplete transcripts occur unless a coverage term is added.
- The authors note that simply maximizing transcript length yields more word insertion errors and worse overall WER.
- Evident from the text: results are reported on a single dataset (WSJ) with a particular ASR setup, so generalization beyond this setting is not established in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 7 Not checked · 37 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe two shortcomings: overconfidence in its predictions and a tendency to produce incomplete transcriptions when language models are used.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A width 100 beam search with a trigram language model finds the second transcript, which misses the beginning of the utterance.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we didn't observe consistent gains from this extension of the beam search criterion.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Temporal LS | 6.6M | 14.1 | 10.7 |”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, incomplete transcripts were produced unless a recording coverage term was added.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 46 claims and to verify the 9 references still open.
- It may state two separate claims as one. 16
- Probably not a substantial claim. 5
- The quote only partly states this claim. 2
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“Deep learning [1] has led to many breakthroughs including speech and image recognition [2, 3, 4, 5, 6, 7].”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“Networks were trained using 8 asynchronous replica workers each employing the ADAM algorithm [21] with default parameters and the learning rate set initially to 10−3, then reduced to 10−4 and 10−5 after 400k and 500k training steps, respectively.”
MethodRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“Especially, Sequence-to-Sequence (seq2seq) neural networks have proved to be very successful on complex transduction tasks, such as machine translation [8, 9, 10], speech recognition [11, 12, 13], or lip-reading [14].”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“Especially, Sequence-to-Sequence (seq2seq) neural networks have proved to be very successful on complex transduction tasks, such as machine translation [8, 9, 10], speech recognition [11, 12, 13], or lip-reading [14].”
BackgroundHinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 1×
“Past attentions vectors were used as auxiliary inputs in the emitting RNN either directly [34], or as cumulative coverage information [35].”
MethodAsynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 1×
“Smoothing techniques increase the entropy of a model’s predictions, a technique that was used to promote exploration in reinforcement learning [27, 28, 29].”
MethodGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 2×
“Our solution that employs a coverage penalty at decode time only is most similar to the one used by the Google Translation system [10].”
BackgroundMethod
Research works printed but not linked: 2
- [1][1] Y. LeCun, Y. Bengio, and G. Hinton, 'Deep learning,' Nature , vol. 521, no. 7553, pp. 436-444, 2015. -
- [23][23] A. Graves and N. Jaitly, 'Towards End-To-End Speech Recognition with Recurrent Neural Networks,' in ICML '14 , 2014, pp. 17641772. -
Cited by
Items 201–300 of 398
Comprehensive disentanglement with fine-grained feature mitigation for domain generalization
Shao, Y., Wang, C., Jia, Q., & Zhao, W. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107757 · Cited by 10
Mu, B., Wei, K., Guo, P., & Xie, L. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3589856 · Cited by 14
“Numerous ASR and LM fusion methods have been introduced, including shallow fusion [8], [42], [43], [44], component fusion [10], cold fusion [11], and deliberation [9], [45], [46], [47].”
Exploring Model Units and Training Strategies for End-to-End Speech Recognition
Huang, M., Lu, Y., Wang, L., Qian, Y., & Yu, K. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003834 · Cited by 9
“To further exploit the performance of attention model, we apply label smoothing [31, 32] to our best system on wordpiece-1k units.”
MethodAn Attention-Based Joint Acoustic and Text on-Device End-To-End Model
Sainath, T. N., Pang, R., Weiss, R. J., He, Y., Chiu, C., & Strohman, T. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053510 · Cited by 9
“For example, training an RNNLM on text-only data and fusing this into the E2E model, via techniques such as shallow, cold and deep fusion, has shown improvements on long-tail queries [10, 11, 12].”
MethodMultitask Training with Text Data for End-to-End Speech Recognition
Wang, P., Sainath, T. N., & Weiss, R. J. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-683 · Cited by 21
Minimising Biasing Word Errors for Contextual ASR With the Tree-Constrained Pointer Generator
Sun, G., Zhang, C., & Woodland, P. C. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3224286 · Cited by 9
“A coverage penalty [61] of 0.01 was applied to AED models.”
MethodInfluentialAerial image recognition in discriminative bi-transformer
Zhao, Y., Chen, Y., Lu, X., Zhou, L., & Xiong, S. (2023) · Signal Processing · doi:10.1016/j.sigpro.2023.108963 · Cited by 9
Chatbot Development Through the Ages : A Survey
Shah, I., Jhawar, S., Khater, A., Jacob, A., & Potdar, G. P. (2023) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit2390329 · Cited by 9
Improved Chord Recognition by Combining Duration and Harmonic Language ModelsPreprint
Korzeniowski, F., & Widmer, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.05335 · Cited by 9
Stay on topic with Classifier-Free Guidance
Sanchez, G., Fan, H., Spangher, A., Levi, E., Ammanamanchi, P. S., & Biderman, S. (2023) · arXiv · doi:10.48550/arxiv.2306.17806 · Cited by 8
“Early efforts in this vein aimed at increasing coherence include now-standard techniques like temperature-scaling [17], nucleus sampling [38] and heuristics (e.g. repetition penalties [31]).”
BackgroundOn Minimum Word Error Rate Training of the Hybrid Autoregressive Transducer
Lu, L., Meng, Z., Kanda, N., Li, J., & Gong, Y. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-161 · Cited by 8
Label Hierarchy Transition: Delving Into Class Hierarchies to Enhance Deep Classifiers
Wang, R., Cai, D., Xiao, K., Jia, X., Han, X., & Meng, D. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3683788 · Cited by 8
“Label embedding aims to encode the correlation information among classes into the traditional one-hot labels, which has been widely explored in multi-class classification [5], [6], [50], [51].”
Zhang, L., Wu, S., & Wang, Z. (2025) · Electronics · doi:10.3390/electronics14040802 · Cited by 8
Zhang, Y., Yao, S., Wang, P., Wu, H., Xu, Z., Wang, Y., & Zhang, Y. (2022) · Applied Sciences · doi:10.3390/app122211830 · Cited by 8
Speech-Text Based Multi-Modal Training with Bidirectional Attention for Improved Speech Recognition
Yang, Y., Xu, H., Huang, H., Chng, E. S., & Li, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096726 · Cited by 9
“In order to well exploit text data, one of the simplest ways is to employ text data to train an external language model (LM) [19,20] that is fused with ASR system, yielding improved results.”
MethodIntegrating Knowledge Into End-to-End Speech Recognition From External Text-Only Data
Bai, Y., Yi, J., Tao, J., Wen, Z., Tian, Z., & Zhang, S. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3066274 · Cited by 9
“Label smoothing has become a standard method to improve the performance of AED models [43].”
MethodAcoustic-to-Word Recognition with Sequence-to-Sequence Models
Palaskar, S., & Metze, F. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639664 · Cited by 8
Dynamic temporal residual network for sequence modeling
Yan, R., Peng, L., Xiao, S., Johnson, M. T., & Wang, S. (2019) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-019-00328-x · Cited by 8
“Recently, deep neural networkswith attention [3,6] or selfattention [34]mechanismhave shown improvedperformance in sequencemodeling tasks.”
BackgroundSong, M., Zhao, Y., Wang, S., & Han, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053589 · Cited by 8
“Additionally, in a sequence-to-sequence speech recognition task, a neighborhood smoothing scheme [9], which allocates a probability α to neighboring tokens in the training transcript, was used to avoid errors of skipping characters.”
MethodMasumura, R., Makishima, N., Ihori, M., Takashima, A., Tanaka, T., & Orihashi, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1930 · Cited by 8
“The unpaired text data are usually used for language model (LM) fusion [9,10].”
MethodAcoustic-to-Word Recognition with Sequence-to-Sequence ModelsPreprint
Palaskar, S., & Metze, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.09597 · Cited by 8
Attentional Speech Recognition Models Misbehave on Out-of-domain UtterancesPreprint
Keung, P., Niu, W., Lu, Y., Salazar, J., & Bhardwaj, V. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.05150 · Cited by 8
Whisper Has an Internal Word Aligner
Yeh, S.-L., Meng, Y., & Tang, H. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434635 · Cited by 9
“The coverage of input by the attention is another intuitive measure [30], [31].”
A Likelihood Ratio Based Domain Adaptation Method for E2E Models
Choudhury, C., Gandhe, A., Ding, X., & Bulyko, I. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746480 · Cited by 7
Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition
Lohrenz, T., Schwarz, P., Li, Z., & Fingscheidt, T. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688298 · Cited by 7
Bai, N., Luo, R., Nourian, P., & Roders, A. P. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.34 · Cited by 9
“, 2016), speech (Chorowski and Jaitly, 2017) and natural language processing”
BackgroundComparing the performance of classic voice-driven assistive systems for dysarthric speech
Zheng, W.-Z., Han, J.-Y., Cheng, H.-L., Chu, W.-C., Chen, K.-C., & Lai, Y.-H. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.104447 · Cited by 7
Acoustic-to-Phrase Models for Speech Recognition
Gaur, Y., Li, J., Meng, Z., & Gong, Y. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-3056 · Cited by 7
“To optimize them, we minimize the label-smoothed cross-entropy [27] loss between the posterior probabilities from the decoder and the ground truth tokens.”
MethodTowards Understanding Attention-Based Speech Recognition Models
Qin, C.-X., & Qu, D. (2020) · IEEE Access · doi:10.1109/access.2020.2970758 · Cited by 7
“It is more interpretable than other methods, and it achieves rather promising results [5], [15], [16].”
Advancing Momentum Pseudo-Labeling with Conformer and Initialization Strategy
Higuchi, Y., Moritz, N., Roux, J. L., & Hori, T. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746275 · Cited by 14
Principled Comparisons for End-to-End Speech Recognition: Attention vs Hybrid at the 1000-Hour Scale
Rouhe, A., Grósz, T., & Kurimo, M. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3336517 · Cited by 9
“To deal with the length bias of AED-models [49], we use an end-of-sentence probability threshold and an attention coverage penalty [50].”
MethodImproving Multilingual and Code-Switching ASR Using Large Language Model Generated Text
Hu, K., Sainath, T. N., Li, B., Zhang, Y., Cheng, Y., Wang, T., Zhang, Y., & Liu, F. (2023) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru57964.2023.10389644 · Cited by 7
“A common solution has been the integration of external neural language models trained on an extensive corpus of unpaired text data [1–4].”
BackgroundRobust Beam Search for Encoder-Decoder Attention Based Speech Recognition Without Length Bias
Zhou, W. G., Schlüter, R., & Ney, H. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1958 · Cited by 7
“Such issues are observed in many applications such as speech recognition [8, 9] and neural machine translation (NMT) [10, 11, 12].”
BackgroundMultitask Training with Text Data for End-to-End Speech RecognitionPreprint
Wang, P., Sainath, T. N., & Weiss, R. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14318 · Cited by 9
“The most common method for leveraging text-only data is to train a language model (LM) and integrate it into the recognition process using shallow [17], cold [18], or deep fusion [19].”
MethodSemi-Supervised end-to-end Speech Recognition via Local Prior Matching
Hsu, W.-N., Lee, A., Synnaeve, G., & Hannun, A. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383552 · Cited by 7
“Neural Network Architecture: The proposal model r y | x and the online model q y | x are sequence-to-sequence neural networks [60, 18] with the same time-depth separable (TDS) architecture proposed in [61].”
BackgroundMethodSpotting words in silent speech videos: a retrieval-based approach
Jha, A., Namboodiri, V. P., & Jawahar, C. V. (2019) · Machine Vision and Applications · doi:10.1007/s00138-019-01006-y · Cited by 7
“Subsequent attention-based models outperformed the CTC-based models[12].”
Cross-Modal Transformer-Based Neural Correction Models for Automatic Speech Recognition
Tanaka, T., Masumura, R., Ihori, M., Takashima, A., Moriya, T., Ashihara, T., Orihashi, S., & Makishima, N. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1992 · Cited by 6
Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data
Gheini, M., Likhomanenko, T., Sperber, M., & Setiawan, H. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.483 · Cited by 8
Improving OOV Detection and Resolution with External Language Models in Acoustic-to-Word ASR
Inaguma, H., Mimura, M., Sakai, S., & Kawahara, T. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639563 · Cited by 6
Fast Distributional Smoothing for Regularization in CTC Applied to Text Recognition
Tanaka, R., Ono, S., & Furuhata, A. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00056 · Cited by 6
“Proposed methods for regularizing sequence learning include label smoothing [5], increasing temperature in the softmax function [6] and simply adding a confidence penalty [7].”
MethodHCADecoder: A Hybrid CTC-Attention Decoder for Chinese Text Recognition
Cai, S., Xue, W., Li, Q., & Zhao, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86334-0_12 · Cited by 6
Simplified End-to-End MMI Training and Voting for ASRPreprint
Fritz, L., & Burshtein, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.10356 · Cited by 6
Understanding (Generalized) Label Smoothing when Learning with Noisy Labels.Preprint
Wei, J., Liu, H., Liu, T., Niu, G., & Liu, Y. (2021) · arXiv (Cornell University) · Cited by 9
Improving End-To-End Modeling for Mispronunciation Detection with Effective Augmentation MechanismsPreprint
Lo, T.-H., Sung, Y., & Chen, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08731 · Cited by 6
Structure-Aware Label Smoothing for Graph Neural NetworksPreprint
Wang, Y., Cai, Y., Liang, Y., Wang, W. W., Ding, H., Chen, M., Tang, J., & Hooi, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.00499 · Cited by 6
Partial Greedy Algorithm to Extract a Minimum Phonetically-and-Prosodically Rich Sentence Set
Alfiansyah, F., & Suyanto, S. (2018) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2018.091274 · Cited by 5
“In [8], the researchers consider that the LAS system is a more successful model than the CTCbased systems.”
BackgroundReducing Flipping Errors in Deep Neural Networks
Deng, X., Xiao, Y., Long, B., & Zhang, Z. M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i6.20603 · Cited by 8
Predicting Customer Satisfaction with Soft Labels for Ordinal Classification
Manderscheid, E., & Lee, M. A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.62 · Cited by 8
“In another domain such as speech recognition, Chorowski and Jaitly, 2016 used label smoothing to reduce word error rate on the WSJ dataset.”
MethodIn and Out-of-Domain Text Adversarial Robustness via Label Smoothing
Yang, Y., Dan, S., Roth, D., & Lee, I. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.58 · Cited by 5
Speaker and Language Aware Training for End-to-End ASR
Bansal, S., Malhotra, K., & Ganapathy, S. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9004000 · Cited by 5
“A deep fusion approach [11] attempts to fuse hidden states of pre-trained external RNN-LM and pre-trained ASR with the help of gating mechanism and trains the fused model further on the ASR training data.”
BackgroundMethodA Deliberation-Based Joint Acoustic and Text Decoder
Mavandadi, S., Sainath, T. N., Hu, K., & Wu, Z. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-165 · Cited by 5
“Broadly speaking, these approaches use some combination of an LM trained on text data (shallow, cold, deep fusion [10, 11, 12, 13]) and a multi-stage training procedure that incorporates unpaired data (“weak distillation” [14], “backtranslation” [15], “cycle-consistency” [16, 17, 18]).”
MethodProvilkov, I., & Malinin, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.677 · Cited by 5
Speech-and-Text Transformer: Exploiting Unpaired Text for End-to-End Speech Recognition
Wang, Q., Zhou, X., & Li, H. (2023) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000001 · Cited by 5
Perception and Semantic Aware Regularization for Sequential Confidence Calibration
Peng, Z., Luo, Y., Chen, T., Xu, K., & Huang, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01027 · Cited by 6
Label-Smooth Learning for Fine-Grained Visual Categorization
Mo, X., Wei, T., Zhang, H., Huang, Q., & Luo, W. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-41404-7_2 · Cited by 6
Dynamic Temporal Residual Learning for Speech Recognition
Xie, J., Yan, R., Xiao, S., Peng, L., Johnson, M. T., & Zhang, W.-Q. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054653 · Cited by 5
“The encoder is usually an LSTM network, while the decoder adopts connectionist temporal classification (CTC) [11, 12] or attention mechanism [13].”
Huang, Z., Li, P., Ji, X., Zhang, P., & Yan, Y. (2021) · International Symposium on Chinese Spoken Language Processing (ISCSLP) · doi:10.1109/iscslp49672.2021.9362080 · Cited by 5
“Although the CI uniform LSR and the CI unigram LSR have shown success on several tasks[17][2], they are insufficient for CSSR training.”
BackgroundKoehn, P. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108608480.003 · Cited by 5
LMC-SMCA: A New Active Learning Method in ASR
Sun, X., Wang, B., Liu, S., Lu, T., Shan, X. J., & Yang, Q. (2021) · IEEE Access · doi:10.1109/access.2021.3062157 · Cited by 5
“Label smoothing [41], [42] is a regularization mechanism to prevent the model from making over-confident predictions.”
BackgroundContextual Density Ratio for Language Model Biasing of Sequence to Sequence ASR Systems
Andrés-Ferrer, J., Albesano, D., Zhan, P., & Vozila, P. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-443 · Cited by 5
“In addition, because of the attention models, we used a coverage score [27] of 0.”
MethodDeep Neural Network Calibration for E2E Speech Recognition System
Lee, M.-H., & Chang, J. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-176 · Cited by 8
“While DNNs exhibit remarkable classification accuracy, some recent studies revealed that very complex and sophisticated deep neural networks are frequently overfitted on the training dataset, and they generate overconfidence probability distribution than the actual distribution [3, 4].”
BackgroundInfluentialConcept-Based Label Distribution Learning for Text Classification
Li, H., Huang, G., Li, Y., Zhang, X., & Wang, Y. (2022) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-022-00144-y · Cited by 5
“Chorowski and Jaitly [15] used label smoothing to reduce the word error rate on speech recognition tasks.”
MethodTowards Fast and Accurate Streaming End-to-End ASRPreprint
Li, B. H., Chang, S.-Y., Sainath, T. N., Pang, R., He, Y., Strohman, T., & Wu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.11544 · Cited by 5
“We then run the LAS model on each sequence in the teacher-forcing mode to compute a score, which combines log probability of the sequence and the attention coverage penalty [20].”
MethodAutomatic Chord Recognition with Higher-Order Harmonic Language Modelling
Korzeniowski, F., & Widmer, G. (2018) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553600 · Cited by 4
Word Similarity Based Label Smoothing in Rnnlm Training for ASR
Song, M., Zhao, Y., Wang, S., & Han, M. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383598 · Cited by 4
“This technique has since been investigated and proven useful on image classification [6], speech recognition [7], and machine translation [8] tasks.”
MethodMasumura, R., Sato, H. P., Tanaka, T., Moriya, T., Ijima, Y., & Oba, T. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2111 · Cited by 4
“In addition, a heuristic criterion that uses log-linear interpolation of a generation probability computed from an end-to-end ASR model and that computed from an external language model (called shallow fusion) was also examined [26, 27].”
MethodInaguma, H., & Kawahara, T. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1110 · Cited by 4
Koehn, P. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108608480.009 · Cited by 7
Lee, M.-H., & Chang, J. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747082 · Cited by 4
Li, M., Do, C.-T., & Doddipatla, R. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094800 · Cited by 4
“So far, there have been studies primarily focusing on the integration of the ELMs, which include shallow fusion [10, 11], deep fusion [12], cold fusion [13] and component fusion [14].”
BackgroundMethodInfluentialConfidence-aware calibration and scoring functions for curriculum learning
Ao, S., Rueger, S., & Siddharthan, A. (2023) · International Conference on Machine Vision · doi:10.1117/12.2679353 · Cited by 4
Learning from a Lightweight Teacher for Efficient Knowledge DistillationPreprint
Liu, Y., Zhang, W., & Wang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09163 · Cited by 4
“Label smoothing has been successfully utilized to improve the accuracy of deep learning models across a range of tasks, including image classification [38], speech recognition [4], and machine translation [43].”
MethodFusion Recurrent Neural NetworkPreprint
Sun, Y., Wang, Y., Fu, K., Wang, Z., Zhang, C., & Ye, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04069 · Cited by 4
“It is widely used in recommendation systems [45], speech recognition [4, 5, 11, 41], natural language processing [7, 35, 36, 43], intelligent transportation system [13, 40] and other fields.”
BackgroundData Troubles in Sentence Level Confidence Estimation for Machine TranslationPreprint
Chelba, C. I., Zhou, J., Yuezhang, Li, Kazawa, H., Klingner, J., & Niu, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13856 · Cited by 4
“Since our training data is annotated by a single annotator (for both cost effectiveness and anno-tator availability reasons), we model the uncertainty on training labels by using label smoothing (Chorowski and Jaitly, 2016) during training.”
Lee, C.-W., Lee, J.-H., & Chang, J. (2024) · IEEE Signal Processing Letters · doi:10.1109/lsp.2024.3434467 · Cited by 3
“The incorporation of LM into the acoustic model (AM) through LM shallow fusion combines the output probabilities of both models to enhance the transcription accuracy [22], [23].”
MethodEnhancing Language Model Alignment: A Confidence-Based Approach to Label Smoothing
Huang, B., Sharma, H., & Mao, Y. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.1189 · Cited by 7
SQUIREDL: Sparse Sequence-to-Sequence Uncertainty Estimation in Evidential Deep Learning
Vavaroutas, S., Dang, T., Rocheteau, E., & Mascolo, C. (2025) · ACM Transactions on Computing for Healthcare · doi:10.1145/3723049 · Cited by 4
“While the modelling of temporal dependencies is well-established in domains such as language translation and voice-enabled commands [12], the consideration of temporal dependencies for uncertainty estimation remains largely untapped.”
Sayed, S. A., Seoud, R. A. A. A. A., & Youssry, H. (2024) · Journal of Electrical and Computer Engineering · doi:10.1155/2024/4976944 · Cited by 4
“In addition, temporal smoothing [37] was combined with a distribution probability mass to generate the neighbouring token unit in the transcript for label smoothing.”
MethodCOLAM: Co-Learning of Deep Neural Networks and Soft Labels via Alternating Minimization
Li, X., Xiong, H., An, H., Xu, C., & Dou, D. (2022) · Neural Processing Letters · doi:10.1007/s11063-022-10830-9 · Cited by 3
“Researchers have widely adopted label smoothing to optimize a broad range of tasks, including but not limited to image classification [15, 16, 17], speech recognition [18], and machine translation [19].”
BackgroundKoehn, P. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108608480.018 · Cited by 3
Multi-Granularity Sequence Alignment Mapping for Encoder-Decoder Based End-to-End ASR
Tang, J., Zhang, J., Song, Y., McLoughlin, I. V., & Dai, L.-R. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3101921 · Cited by 3
On-the-Fly Text Retrieval for end-to-end ASR Adaptation
Yusuf, B., Gourav, A., Gandhe, A., & Bulyko, I. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095857 · Cited by 6
Hentschel, M., Nishikawa, Y., Komatsu, T., & Fujita, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447305 · Cited by 3
“In order to cope with these errors, it is necessary to combine an ex-ternal LM, learned from large-scale data, with the E2E-ASR sys-tem in shallow fusion [3, 4] during inference.”
MethodPark, J.-H., Kim, C., & Sung, W. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383583 · Cited by 3
Crowd-Powered Hybrid Classification Services: Calibration is all you need
Sayin, B., Krivosheev, E., Ramírez, J., Casati, F., Александровна, Т. Е., Маланина, В. А., & Yang, J. (2021) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws53863.2021.00019 · Cited by 3
Multilevel Text Normalization with Sequence-to-Sequence Networks and Multisource LearningPreprint
Ruzsics, T., & Samardżić, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.11340 · Cited by 3
“In order to include additional target-side sequential context we follow the line of the work, where pretrained target-side language model (LM) is integrated into a seq2seq framework for speech recognition (Bahdanau et al., 2016; Chorowski and Jaitly, 2016; Kannan et al., 2018) and machine translation (Gülçehre et al.”
Domain Adaptation via Teacher-Student Learning for End-to-End Speech RecognitionPreprint
Meng, Z., Li, J., Gaur, Y., & Gong, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.01798 · Cited by 3
“A label-smoothed cross-entropy [42] loss is minimized during training.”
BackgroundIterative Teaching by Label SynthesisPreprint
Liu, W., Liu, Z., Wang, H., Paull, L., Schölkopf, B., & Weller, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14432 · Cited by 6
“, knowledge distillation [26, 53], label smoothing [11, 59, 73], and self-training [104].”
BackgroundNon-Intrusive Automatic Speech Recognition Refinement: A SurveyPreprint
Peyghan, M. R., Roudi, S. S., Zouashkiani, S., Amini, S., Rajabi, F., & Ghaemmaghami, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.07285 · Cited by 3
“Beyond uniform proposed in [193] and unigram in [194], smoothing in [121] introduces neighborhood smoothing , wherein the residual probability is concentrated on tokens within ± 1 and ± 2 positions in the reference transcript, thereby reflecting the temporal structure of speech.”
Self-Relaxed Joint Training: Sample Selection for Severity Estimation with Ordinal Noisy Labels
Takezaki, S., Tanaka, K., & Uchida, S. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00046 · Cited by 2
“Label smoothing (, or soft labeling) has been introduced to improve accuracy across many tasks [4, 27, 32, 33, 43].”
Link Prediction Under Non-targeted Attacks: Do Soft Labels Always Help?
Memariani, A., Röder, M., Sharma, A., Demir, C., & Ngomo, A.-C. N. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-09527-5_6 · Cited by 2
Robust & Compact End-to-End Hindi Language ASR System
Nikam, H., Bhargava, M., Dhote, P., Patil, S., Sasmal, S., Singh, L., Mehta, S., & Kumar, A. (2022) · International Conference on Industry 4.0 Technology (I4Tech) · doi:10.1109/i4tech55392.2022.9952846 · Cited by 2
Horii, K., Ohta, K., Nishimura, R., Ogawa, A., & Kitaoka, N. (2023) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc58517.2023.10317137 · Cited by 2
Fucci, D., Gaido, M., Papi, S., Cettolo, M., Negri, M., & Bentivogli, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.705 · Cited by 6
Katsumata, K., Vo, D. M., & Nakayama, H. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00487 · Cited by 3
“Label smoothing [54] replaces hard targets with soft targets by taking the weighted average of the original targets, avoiding overconfidence on several tasks [9, 57].”
BackgroundMeshed Context-Aware Beam Search for Image Captioning
Zhao, F., Yu, Z., Wang, T., & Zhao, H. (2024) · Entropy · doi:10.3390/e26100866 · Cited by 4
“Additionally, length normalization beam search [36] compensates for the bias towards shorter sequences by normalizing scores based on sequence length, ensuring fairer comparisons among outputs.”
BackgroundProper Error Estimation and Calibration for Attention-Based Encoder-Decoder Models
Lee, M.-H., & Chang, J. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3492799 · Cited by 2
“On the other hand, [13] showed that the AED-based ASR model tends to overestimate which could be alleviated by using thelabelsmoothingmethod.”
InfluentialLarge Language Models Meet Speech Recognition
Chen, P., & Liu, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-76770-8_9 · Cited by 2
Improving Contextual Asr Via Multi-Grained Fusion With Large Language Models
Zhou, S., & Li, Z. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460896 · Cited by 2
Metadata
- Authors
- Chorowski, J., & Jaitly, N.
- Affiliations
- Google (United States) (US)
- ORCID
- Chorowski, Jan
- Venue
- Interspeech 2017
- Publisher
- ISCA
- Published
- 2017-08-20
- Pages
- 523-527
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2577366047
- Cited by
- 358
- References
- 34
- Citations by year
- 2026: 4 · 2025: 14 · 2024: 13 · 2023: 39 · 2022: 26 · 2021: 73 · 2020: 88 · 2019: 49 · 2018: 40 · 2017: 12
- FWCI
- 23.58
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Keywords
- attention-based models, language model integration, trigram language model, end-to-end speech recognition, speaker-independent recognition, Wall Street Journal
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.