Towards Better Decoding and Language Model Integration in Sequence to Sequence Models
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.
Abstract
The recently proposed Sequence-to-Sequence (seq2seq) framework advocates replacing complex data processing pipelines, such as an entire automatic speech recognition system, with a single neural network trained in an end-to-end fashion.In this contribution, we analyse an attention-based seq2seq speech recognition system that directly transcribes recordings into characters.We observe two shortcomings: overconfidence in its predictions and a tendency to produce incomplete transcriptions when language models are used.We propose practical solutions to both problems achieving competitive speaker independent word error rates on the Wall Street Journal dataset: without separate language models we reach 10.6% WER, while together with a trigram language model, we reach 6.7% WER.
Overview
CoChat Summary
TL;DR
This paper analyzes two failure modes of an attention-based seq2seq speech recognizer: overconfident character predictions and incomplete transcripts when using language models. It proposes label smoothing plus decode-time coverage handling, reaching 10.6% WER without a separate language model and 6.7% WER with a trigram language model on WSJ.
Key findings
- The authors report that the baseline seq2seq model on WSJ achieved 14.2% WER on eval92, while unigram label smoothing reduced this to 10.6% and temporal/neighborhood smoothing to 10.7% on eval92.
- With an extended trigram language model and coverage, the best reported result was 6.7% WER on eval92; the corresponding dev93 score was 9.7% for temporal label smoothing + coverage.
- They found that model overconfidence makes beam search less effective and that increasing SoftMax temperature can help beam search, but without changing greedy decoding accuracy.
- For incomplete transcripts, simply constraining EOS was not sufficient at large beam widths; promoting coverage helped, while maximizing transcript length alone caused more word insertions and worse overall WER.
Methods
- Attention-based seq2seq speech recognition model built on Listen, Attend and Spell: a multilayer Bi-LSTM listener, an LSTM speller, and attention.
- Experiments were run on the Wall Street Journal dataset, training on si284, validating on dev93, and evaluating on eval92; inputs were 80-dimensional mel-scale filterbanks with first and second differences and per-speaker normalization.
- Main analysis compared decoding strategies and regularizers: SoftMax temperature changes, label smoothing variants (uniform/unigram and neighborhood/temporal), beam search settings, and a decode-time coverage penalty with an extended trigram language model.
Limitations
- The paper states that adding a language model introduces two challenges: model overconfidence makes balancing the beam-search terms difficult, and incomplete transcripts occur unless a coverage term is added.
- The authors note that simply maximizing transcript length yields more word insertion errors and worse overall WER.
- Evident from the text: results are reported on a single dataset (WSJ) with a particular ASR setup, so generalization beyond this setting is not established in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 7 Not checked · 37 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe two shortcomings: overconfidence in its predictions and a tendency to produce incomplete transcriptions when language models are used.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A width 100 beam search with a trigram language model finds the second transcript, which misses the beginning of the utterance.”
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we didn't observe consistent gains from this extension of the beam search criterion.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Temporal LS | 6.6M | 14.1 | 10.7 |”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Second, incomplete transcripts were produced unless a recording coverage term was added.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
20 more claims
Create a free account to see all 46 claims and to verify the 9 references still open.
- It may state two separate claims as one. 16
- Probably not a substantial claim. 5
- The quote only partly states this claim. 2
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“Deep learning [1] has led to many breakthroughs including speech and image recognition [2, 3, 4, 5, 6, 7].”
BackgroundAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“Networks were trained using 8 asynchronous replica workers each employing the ADAM algorithm [21] with default parameters and the learning rate set initially to 10−3, then reduced to 10−4 and 10−5 after 400k and 500k training steps, respectively.”
MethodRethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“Especially, Sequence-to-Sequence (seq2seq) neural networks have proved to be very successful on complex transduction tasks, such as machine translation [8, 9, 10], speech recognition [11, 12, 13], or lip-reading [14].”
BackgroundNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“Especially, Sequence-to-Sequence (seq2seq) neural networks have proved to be very successful on complex transduction tasks, such as machine translation [8, 9, 10], speech recognition [11, 12, 13], or lip-reading [14].”
BackgroundHinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationTensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016) · arXiv · doi:10.48550/arxiv.1603.04467 · Cited by 9,408
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 1×
“Past attentions vectors were used as auxiliary inputs in the emitting RNN either directly [34], or as cumulative coverage information [35].”
MethodAsynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132 · cited 1×
“Smoothing techniques increase the entropy of a model’s predictions, a technique that was used to promote exploration in reinforcement learning [27, 28, 29].”
MethodGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 2×
“Our solution that employs a coverage penalty at decode time only is most similar to the one used by the Google Translation system [10].”
BackgroundMethod
Research works printed but not linked: 2
- [1][1] Y. LeCun, Y. Bengio, and G. Hinton, 'Deep learning,' Nature , vol. 521, no. 7553, pp. 436-444, 2015. -
- [23][23] A. Graves and N. Jaitly, 'Towards End-To-End Speech Recognition with Recurrent Neural Networks,' in ICML '14 , 2014, pp. 17641772. -
Cited by
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558
“We also use a coverage penalty c [29].”
MethodHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266
“…together without architectural modifications: Ramachandran et al. (2016) use language models to initialize the encoder and decoder side of the seq2seq model independently, and Chorowski and Jaitly (2016) combine the predictions of the language model and seq2seq model solely at inference time.”
MethodHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242
“…together without architectural modifications: Ramachandran et al. (2016) use language models to initialize the encoder and decoder side of the seq2seq model independently, and Chorowski and Jaitly (2016) combine the predictions of the language model and seq2seq model solely at inference time.”
MethodSpeech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition
Dong, L., Xu, S., & Xu, B. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462506 · Cited by 1,100
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models
Chiu, C.-C., Sainath, T. N., Wu, Y., Prabhavalkar, R., Nguyen, P., Chen, Z., Kannan, A., Weiss, R. J., Rao, K., Gonina, E., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462105 · Cited by 1,043
Self-citationHybrid CTC/Attention Architecture for End-to-End Speech Recognition
Watanabe, S., Hori, T., Kim, S., Hershey, J. R., & Hayashi, T. (2017) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2017.2763455 · Cited by 800
“Another approach is the coverage term recently proposed in [14], which is incorporated in the decoding objective in Eq.”
BackgroundMethodNeural Machine Translation: A Review
Stahlberg, F. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12007 · Cited by 397
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation
Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G., Jones, L., Schuster, M., Shazeer, N., Parmar, N., et al. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1008 · Cited by 418
“The RNN-based NMT approach, or RNMT, was quickly established as the de-facto standard for NMT, and gained rapid adoption into large-scale systems in industry, e.g. Baidu (Zhou et al., 2016), Google (Wu et al., 2016), and Systran (Crego et al., 2016).”
MethodZhang, Q., Lu, H., Sak, H., Tripathi, A., McDermott, E., Koo, S., & Kumar, S. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053896 · Cited by 433
GLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3019 · Cited by 470
“, 2018), beam search (Chorowski and Jaitly, 2016; Shao et al., 2017), temperature-modulated sampling (Dagan and Engelson, 1995), or even implicitly with rule-based templated approaches.”
BackgroundGLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. M. (2019) · arXiv · doi:10.48550/arxiv.1906.04043 · Cited by 460
“, 2018), beam search (Chorowski and Jaitly, 2016; Shao et al., 2017), temperature-modulated sampling (Dagan and Engelson, 1995), or even implicitly with rule-based templated approaches.”
BackgroundKoehn, P. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108608480 · Cited by 236
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372
Sequence-to-Sequence Models Can Directly Translate Foreign Speech
Weiss, R. J., Chorowski, J., Jaitly, N., Wu, Y., & Chen, Z. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-503 · Cited by 293
“The model has also recently achieved promising results on automatic speech recognition (ASR) even without the use of language models [8, 9, 10, 11].”
BackgroundSelf-citationCold Fusion: Training Seq2Seq Models Together with Language Models
Sriram, A., Jun, H., Satheesh, S., & Coates, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1392 · Cited by 270
Quartznet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions
Kriman, S., Beliaev, S., Ginsburg, B., Huang, J., Kuchaiev, O., Lavrukhin, V., Leary, R., Li, J., & Zhang, Y. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053889 · Cited by 278
“We used following end-to-end models trained on standard speech features3 for comparison: 1) RNN-CTC [30] - CTC model with 5 bidirectional LSTM layers, 500 cells in each layer; 2) ResCNN-LAS [31]: Listen-Attend-Spell model with deep residual convLSTM encoder and LSTM decoder + label smoothing; 3) Wav2Letter++ [29] - CT…”
MethodEnd-to-End Speech Recognition: A Survey
Prabhavalkar, R., Hori, T., Sainath, T. N., Schlüter, R., & Watanabe, S. (2024) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2023.3328283 · Cited by 223
“Another prominent feature of E2E systems is their direct character-level modeling avoiding phoneme-based modeling and pronunication lexica [18], [216], [194], [15], [259], [187], [260], [261], [262], [112], [263], with some even heading for whole-word modeling [259], [25].”
BackgroundMethodInfluentialAn Overview of End-to-End Automatic Speech Recognition
Wang, D., Wang, X., & Lv, S. (2019) · Symmetry · doi:10.3390/sym11081018 · Cited by 252
“The early encoder was basically within three layers [48,52,53,61], and gradually developed to four layers [49,51,54,57], five layers [59,62], six layers [55,60].”
BackgroundOn Generative Spoken Language Modeling from Raw Audio
Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A., et al. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00430 · Cited by 75
“We compute either a PER (without fusing additional word/character-based LM (Chorowski and Jaitly, 2016) or imposing lexical constraints), or a CER after the LM (we use a CER instead of a WER to give partial credit).”
An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model
Kannan, A., Wu, Y., Nguyen, P., Sainath, T. N., Chen, Z., & Prabhavalkar, R. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462682 · Cited by 253
Improved Noisy Student Training for Automatic Speech Recognition
Park, D. S., Zhang, Y., Jia, Y., Han, W., Chiu, C.-C., Li, B., Wu, Y., & Le, Q. V. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-1470 · Cited by 218
Exploring neural transducers for end-to-end speech recognition
Battenberg, E., Chen, J., Child, R., Coates, A., Li, Y. G. Y., Liu, H., Satheesh, S., Sriram, A., & Zhu, Z. (2017) · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru.2017.8268937 · Cited by 227
“The attention model in Table 5 has a WER of 17.4 after beam search on the WSJ dev’93 set, which matches the previously published results (17.9) in [9].”
ResultInfluentialOnline and Linear-Time Attention by Enforcing Monotonic Alignments
Raffel, C., Luong, M.-T., Liu, P. J., Weiss, R. J., & Eck, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.00784 · Cited by 199
“…thank Jan Chorowski, Mark Daoust, Pietro Kreitlon Carolino, Dieterich Lawson, Navdeep Jaitly, George Tucker, Quoc V. Le, Kelvin Xu, Cinjon Resnick, Melody Guan, Matthew D. Hoffman, Jeffrey Dean, Kevin Swersky, Ashish Vaswani, and members of the Google Brain team for helpful discussions and insight.”
BackgroundMethodInfluentialMultilingual Speech Recognition with a Single End-to-End Model
Toshniwal, S., Sainath, T. N., Weiss, R. J., Li, B., Moreno, P., Weinstein, E., & Rao, K. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461972 · Cited by 210
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
Synnaeve, G., Xu, Q., Kahn, J., Likhomanenko, T., Grave, É., Pratap, V., Sriram, A., Liptchinsky, V., & Collobert, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08460 · Cited by 150
Self-Training for End-to-End Speech Recognition
Kahn, J., Lee, A., & Hannun, A. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9054295 · Cited by 196
“Sequence-to-sequence models are known to easily fail at inference in two ways: looping and early stopping [14].”
BackgroundMonotonic Chunkwise AttentionPreprint
Chiu, C., & Raffel, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.05382 · Cited by 248
“Localized label smoothing (Chorowski & Jaitly, 2017) was applied to the target outputs with weights [0 .”
MethodUnsupervised Speech Recognition
Baevski, A., Hsu, W.-N., Conneau, A., & Auli, M. (2021) · arXiv · doi:10.48550/arxiv.2105.11084
“we can observe that this is equivalent to the formulation of language model fusion for end-toend systems that combines posterior and prior probabilities of text for decoding (Chorowski and Jaitly, 2017).”
MethodDoes label smoothing mitigate label noise?Preprint
Łukasik, M., Bhojanapalli, S., Menon, A. K., & Kumar, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02819 · Cited by 266
“Label smoothing is commonly used to improve the performance of deep learning models [Szegedy et al., 2016, Chorowski and Jaitly, 2017, Vaswani et al., 2017, Zoph et al., 2018, Real et al., 2018, Huang et al., 2019, Li et al., 2020].”
MethodMaking Better Mistakes: Leveraging Class Hierarchies With Deep Networks
Bertinetto, L., Mueller, R., Τερτίκας, Κ., Samangooei, S., & Lord, N. A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01252 · Cited by 259
Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing
Tian, Y., Li, D., & Xu, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_26 · Cited by 255
Streamflow Prediction Using Deep Learning Neural Network: Case Study of Yangtze River
Liu, D., Jiang, W., Mu, L., & Wang, S. (2020) · IEEE Access · doi:10.1109/access.2020.2993874 · Cited by 154
“The En-De architecture is usually used in seq2seq (sequence to sequence) problems [46].”
MethodEnd-to-end Speech Recognition Using Lattice-free MMI
Hadian, H., Sameti, H., Povey, D., & Khudanpur, S. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1423 · Cited by 137
A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition
Toshniwal, S., Kannan, A., Chiu, C.-C., Wu, Y., Sainath, T. N., & Livescu, K. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639038 · Cited by 149
Specaugment on Large Scale Datasets
Park, D. S., Zhang, Y., Chiu, C.-C., Chen, Y., Li, B., Chan, W., Le, Q. V., & Wu, Y. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053205 · Cited by 140
“We use shallow fusion [22] with an LSTM language model (LM) with two fusion parameters— the LM weight and coverage penalty [23].”
MethodTwo-Pass End-to-End Speech Recognition
Sainath, T. N., Pang, R., Rybach, D., He, Y., Prabhavalkar, R., Li, W., Visontai, M., Liang, Q., Strohman, T., Wu, Y., et al. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1341 · Cited by 134
A Spelling Correction Model for End-to-end Speech Recognition
Guo, J., Sainath, T. N., & Weiss, R. J. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683745 · Cited by 132
Joint CTC/attention decoding for end-to-end speech recognition
Hori, T., Watanabe, S., & Hershey, J. R. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1048 · Cited by 177
“In decoding, we also added a result of the coverage-term based decoding (Chorowski and Jaitly, 2016), as discussed in Section 3.”
BackgroundMethodHybrid Autoregressive Transducer (HAT)
Variani, E., Rybach, D., Allauzen, C., & Riley, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053600 · Cited by 130
“There has been extensive efforts to develop decoding algorithms that can use external LMs, so-called fusion methods [27, 28, 29, 30, 31].”
BackgroundWatanabe, S., Hori, T., & Hershey, J. R. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268945 · Cited by 129
“End-to-end automatic speech recognition (ASR) has shown the effectiveness recently by greatly reducing ASR building procedure, and by reaching the state-of-the-art performance obtained by conventional hybrid systems [1, 2, 3, 4].”
BackgroundBoundary Smoothing for Named Entity Recognition
Zhu, E., & Li, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.490 · Cited by 174
End-to-end Speech Recognition With Word-Based Rnn Language Models
Hori, T., Cho, J., & Watanabe, S. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639693 · Cited by 145
Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-based Decoding
Li, X., Zhao, Y., Wang, C., Scalia, G., Eraslan, G., Nair, S., Biancalani, T., Ji, S., Regev, A., Levine, S., et al. (2025) · neural information processing systems · doi:10.52202/085713-3194 · Cited by 139
CIF: Continuous Integrate-And-Fire for End-To-End Speech Recognition
Dong, L., & Xu, B. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054250 · Cited by 174
Imputer: Sequence Modelling via Imputation and Dynamic ProgrammingPreprint
Chan, W., Saharia, C., Hinton, G. E., Norouzi, M., & Jaitly, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08926 · Cited by 132
Self-citationTowards Fast and Accurate Streaming End-To-End ASR
Li, B., Chang, S.-Y., Sainath, T. N., Pang, R., He, Y., Strohman, T., & Wu, Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054715 · Cited by 105
“We then run the LAS model on each sequence in the teacher-forcing mode to compute a score, which combines log probability of the sequence and the attention coverage penalty [20].”
MethodSequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions
Hannun, A., Lee, A., Xu, Q., & Collobert, R. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2460 · Cited by 119
Back-Translation-Style Data Augmentation for end-to-end ASR
Hayashi, T., Watanabe, S., Zhang, Y., Toda, T., Hori, T., Astudillo, R. F., & Takeda, K. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639619 · Cited by 123
End-to-End Speech Recognition from the Raw Waveform
Zeghidour, N., Usunier, N., Synnaeve, G., Collobert, R., & Emmanuel Dupoux, A. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2414 · Cited by 99
Structured Fusion Networks for Dialog
Mehri, S., Srinivasan, T., & Eskénazi, M. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5921 · Cited by 99
“The simplest such method, commonly referred to as Shallow Fusion, is to add a language modeling term, pLM (y), to the cost function during inference (Chorowski and Jaitly, 2016).”
MethodA Density Ratio Approach to Language Model Fusion in End-to-End Automatic Speech Recognition
McDermott, E., Sak, H., & Variani, E. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003790 · Cited by 96
SoftSeg: Advantages of soft versus binary training for image segmentation
Gros, C., Lemay, A., & Cohen‐Adad, J. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102038 · Cited by 110
Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognition
Meng, Z., Parthasarathy, S., Sun, E., Gaur, Y., Kanda, N., Lu, L., Chen, X., Zhao, R., Li, J., & Gong, Y. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383515 · Cited by 125
“However, the benefit of subtracting internal LM is larger for RNN-T than for AED because ILME gets greater relative WER reductions with RNN-T than with AED from Shallow Fusion.”
BackgroundMethodInfluentialShan, C., Weng, C., Wang, G., Su, D., Luo, M., Yu, D., & Xie, L. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682490 · Cited by 86
“In addition, softmax smoothing [8, 22] is used to smooth the network label prediction distribution during decoding as given in equation (7).”
MethodAttention-Based End-to-End Speech Recognition on Voice Search
Shan, C., Zhang, J., Wang, Y., & Xie, L. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462492 · Cited by 81
“Recently, some works on end-to-end neural network ASR systems such as Connectionist Temporal Classification (CTC) [4, 5, 6] and attention-based encoderdecoder [7, 8, 9, 10, 11, 12, 13, 14] have emerged.”
BackgroundA General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks
Tang, Y., Pino, J., Wang, C., Ma, X., & Genzel, D. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9415058 · Cited by 103
Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep LearningPreprint
Gordon-Rodríguez, E., Loaiza-Ganem, G., Pleiss, G., & Cunningham, J. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.05231 · Cited by 134
“Label smoothing [27] has enjoyed rapid growth and widespread use as a means to reduce overfitting and improve the out-of-sample accuracy of neural network classifiers across a range of tasks including computer vision [31, 24], speech recognition [8], and machine translation [30].”
BackgroundEnd-to-End Multi-Speaker Speech Recognition
Settle, S., Roux, J. L., Hori, T., Watanabe, S., & Hershey, J. R. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8461893 · Cited by 78
Un-mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning
Shen, Z., Liu, Z., Liu, Z., Savvides, M., Darrell, T., & Xing, E. P. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20119 · Cited by 105
“…adopted successfully to improve the performance of deep neural models across a wide range of tasks, including image classification [Szegedy et al. 2016], object detection [Krothapalli and Abbott 2020], machine translation [Vaswani et al. 2017], and speech recognition [Chorowski and Jaitly 2016].”
BackgroundInvestigating End-to-end Speech Recognition for Mandarin-english Code-switching
Shan, C., Weng, C., Wang, G., Su, D., Luo, M., Yu, D., & Xie, L. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8682850 · Cited by 74
“Softmax smoothing is applied to smooth the distribution of label prediction during decoding [15, 26].”
MethodAn Overview of Language Models: Recent Developments and Outlook
Wei, C., Wang, Y.-C., Wang, B., & Kuo, C. ‐. J. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000010 · Cited by 84
“Shallow fusion [157, 158, 159] considers log-linear interpolation between the scores from an E2E ASR model and an external LM at the decoding stage.”
MethodLabel Confusion Learning to Enhance Text Classification Models
Guo, B., Han, S., Xiao, H., Huang, H., & Lu, T. L. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17529 · Cited by 93
Arabic speech recognition using end‐to‐end deep learning
Alsayadi, H. A., Abdelhamid, A. A., Hegazy, I., & Fayed, Z. T. (2021) · IET Signal Processing · doi:10.1049/sil2.12057 · Cited by 79
“The scalar vales, namely coverage term, are used for each hypothesis in the beam to encourage the long transcripts [56].”
BackgroundMethodEspresso: A Fast End-to-End Neural Speech Recognition Toolkit
Wang, Y., Chen, T., Xu, H., Ding, S., Lv, H., Shao, Y., Peng, N., Xie, L., Watanabe, S., & Khudanpur, S. P. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003968 · Cited by 85
Difficulty-Controllable Multi-hop Question Generation from Knowledge Graphs
Kumar, V., Hua, Y., Ramakrishnan, G., Qi, G., Gao, L., & Li, Y.-F. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30793-6_22 · Cited by 103
“Motivated by previous work [7], we use label smoothing for regularisation with an uncertainty of 0.”
MethodOnline Hybrid CTC/Attention End-to-End Automatic Speech Recognition Architecture
Miao, H., Cheng, G., Zhang, P., & Yan, Y. (2020) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2020.2987752 · Cited by 73
“Therefore, our online joint CTC/attention decoding method can get rid of the requirements for length penalty factor, coverage term [38] and other predefined terms that are often adopted in label-synchronous decoding.”
MethodInformation extraction from historical handwritten document images with a context-aware neural model
Toledo, J. I., Carbonell, M., Fornés, A., & Lladós, J. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2018.08.020 · Cited by 64
Letter-Based Speech Recognition with Gated ConvNetsPreprint
Liptchinsky, V., Synnaeve, G., & Collobert, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.09444 · Cited by 63
“3 (Chorowski and Jaitly, 2016) 6.”
Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical StudyPreprint
Shen, Z., Liu, Z., Xu, D., Chen, Z., Cheng, K.-T. T., & Savvides, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.00676 · Cited by 81
“…and have been applied in many state-of-the-art models, such as language translation (Vaswani et al., 2017; Tan et al., 2019; Zhou et al., 2020), image classification (Xie et al., 2019; He et al., 2019) and speech recognition (Chiu et al., 2018; Pereyra et al., 2017; Chorowski & Jaitly, 2017).”
MethodDual-mode ASR: Unify and Improve Streaming ASR with Full-context ModelingPreprint
Yu, J., Han, W., Gulati, A., Chiu, C., Li, B., Sainath, T. N., Wu, Y., & Pang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06030 · Cited by 68
Recent advances in conversational NLP : Towards the standardization of Chatbot buildingPreprint
Mnasri, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.09025 · Cited by 72
“…using Recurrent Neural Networks (RNN) to tackle complex sequence-to-sequence prediction problems such as machine translation, image captioning (Park et al., 2017), speech recognition (Chorowski and Jaitly, 2016), text summarization (Rush et al., 2015; Nallapati et al., 2016) and question-answering.”
Generalized Entropy Regularization or: There’s Nothing Special about Label Smoothing
Meister, C., Salesky, E., & Cotterell, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.615 · Cited by 77
Mixture Content Selection for Diverse Sequence Generation
Cho, J., Seo, M., & Hajishirzi, H. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1308 · Cited by 75
Adaptive Spatiotemporal InceptionNet for Traffic Flow Forecasting
Wang, Y., Jing, C., Huang, W., Jin, S., & Lv, X. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3237205 · Cited by 55
“Sequence-to-sequence (Seq2seq) [33] structures are widely used in a variety of regression tasks, such as machine translation [61], multi-step ahead time series prediction [62], and other various sequence learning tasks.”
BackgroundGram-CTC: Automatic Unit Selection and Target Decomposition for Sequence LabellingPreprint
Liu, H., Zhu, Z., Li, X., & Satheesh, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00096 · Cited by 55
Towards End-to-End Code-Switching Speech RecognitionPreprint
Luo, N., Jiang, D., Zhao, S., Gong, C., Zou, W., & Li, X. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.13091 · Cited by 55
“During training stage, scheduled sampling and unigram label smoothing are applied as described in [26, 27, 28].”
MethodLeveraging Sequence-to-Sequence Speech Synthesis for Enhancing Acoustic-to-Word Speech Recognition
Mimura, M., Ueno, S., Inaguma, H., Sakai, S., & Kawahara, T. (2018) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639589 · Cited by 53
“It was shown to be very effective to incorporate external language models in speech recognition using conventional seq2seq models based on characters [27, 28, 29].”
BackgroundDistilling the Knowledge of BERT for Sequence-to-Sequence ASR
Futami, H., Inaguma, H., Ueno, S., Mimura, M., Sakai, S., & Kawahara, T. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1179 · Cited by 63
Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations
Islam, M., & Glocker, B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-78191-0_52 · Cited by 71
Improving the Performance of Online Neural Transducer Models
Sainath, T. N., Chiu, C., Prabhavalkar, R., Kannan, A., Wu, Y., Nguyen, P., & Chen, Z. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462366 · Cited by 51
An End-to-End Language-Tracking Speech Recognizer for Mixed-Language Speech
Seki, H., Watanabe, S., Hori, T., Roux, J. L., & Hershey, J. R. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462180 · Cited by 50
“End-to-end automatic speech recognition (ASR) has recently proven its effectiveness by reaching levels of accuracy equivalent to the state-of-the-art conventional hybrid systems [1, 2, 3, 4], while surpassing them in terms of ease of development.”
BackgroundMulti-level language modeling and decoding for open vocabulary end-to-end speech recognition
Hori, T., Watanabe, S., & Hershey, J. R. (2017) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2017.8268948 · Cited by 55
“We also applied a unigram label smoothing technique [29] to avoid over-confidence predictions.”
MethodDomain Adaptation via Teacher-Student Learning for End-to-End Speech Recognition
Meng, Z., Li, J., Gaur, Y., & Gong, Y. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003776 · Cited by 49
“A label-smoothed cross-entropy [42] loss is minimized during training.”
BackgroundNo Need for a Lexicon? Evaluating the Value of the Pronunciation Lexica in End-to-End Models
Sainath, T. N., Prabhavalkar, R., Kumar, S., Lee, S., Kannan, A., Rybach, D., Schogol, V., Nguyen, P., Li, B., Wu, Y., et al. (2018) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2018.8462380 · Cited by 48
Semi-supervised Training for End-to-end Models via Weak Distillation
Li, B., Sainath, T. N., Pang, R., & Wu, Z. (2019) · ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2019.8682172 · Cited by 46
“In addition, interpolating independently trained end-to-end and LM models via shallow fusion has been explored, both for neural machine translation [6] and ASR [4, 7].”
MethodHyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models
Chen, C., Hu, Y., Yang, C.-H. H., Siniscalchi, S. M., Chen, P., & Chng, E. S. (2023) · neural information processing systems · doi:10.52202/075280-1375 · Cited by 112
Flat-Start Single-Stage Discriminatively Trained HMM-Based Models for ASR
Hadian, H., Sameti, H., Povey, D., & Khudanpur, S. P. (2018) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2018.2848701 · Cited by 45
An Online Attention-Based Model for Speech Recognition
Fan, R., Zhou, P., Chen, W., Jia, J., & Liu, G. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2218 · Cited by 44
Ding, H., Chen, K., & Huo, Q. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86331-9_39 · Cited by 48
A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generation
Higuchi, Y., Chen, N., Fujita, Y., Inaguma, H., Komatsu, T., Lee, J., Nozaki, J., Wang, T., & Watanabe, S. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688157 · Cited by 56
On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
Irie, K., Prabhavalkar, R., Kannan, A., Bruguier, A., Rybach, D., & Nguyen, P. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2277 · Cited by 53
“6We find it crucial to constrain the emission of end-of-sentence (EOS) tokens [35] to penalize short sentences (rather than applying length normalization) in shallow fusion: we only allow the model to emit EOS when its score is within 1.”
BackgroundMethodInternal Language Model Training for Domain-Adaptive End-To-End Speech Recognition
Meng, Z., Kanda, N., Gaur, Y., Parthasarathy, S., Sun, E., Lu, L., Chen, X., Li, J., & Gong, Y. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9415039 · Cited by 61
“ILMT with ILME inference performs the best, achieving 17.6%, 10.5% and 6.1% relative WER reductions from the standard AED training with AED inference, Shallow Fusion and ILME inference, respectively.”
MethodInfluentialLearning to segment fetal brain tissue from noisy annotations
Karimi, D., Rollins, C. K., Velasco‐Annis, C., Ouaalam, A., & Gholipour, A. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102731 · Cited by 47
Towards Understanding Label SmoothingPreprint
Xu, Y., Xu, Y., Qian, Q., Li, H., & Jin, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11653 · Cited by 43
“…results have shown that LSR has significant successes in many deep learning applications including image classification [Zoph et al., 2018, He et al., 2019], speech recognition [Chorowski and Jaitly, 2017, Zeyer et al., 2018], and language translation [Vaswani et al., 2017, Nguyen and Salazar, 2019].”
BackgroundUn-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation LearningPreprint
Shen, Z., Liu, Z., Liu, Z., Savvides, M., Darrell, T., & Xing, E. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.05438 · Cited by 45
“…adopted successfully to improve the performance of deep neural models across a wide range of tasks, including image classification [Szegedy et al. 2016], object detection [Krothapalli and Abbott 2020], machine translation [Vaswani et al. 2017], and speech recognition [Chorowski and Jaitly 2016].”
BackgroundWhy are Sequence-to-Sequence Models So Dull? Understanding the Low-Diversity Problem of Chatbots
Jiang, S., & de Rijke, M. (2018) · doi:10.18653/v1/w18-5712 · Cited by 53
“, 2016), therefore methods like neighborhood smoothing (Chorowski and Jaitly, 2016) may be used to solve the low-diversity”
On the Limit of English Conversational Speech Recognition
Tüske, Z., Saon, G., & Kingsbury, B. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-211 · Cited by 41
The Health ChatBots in Telemedicine: Intelligent Dialog System for Remote Support
Vasileiou, M. V., & Maglogiannis, I. G. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/4876512 · Cited by 40
Convolutional Neural Networks for Raw Speech Recognition
Passricha, V., & Aggarwal, R. K. (2018) · IntechOpen eBooks · doi:10.5772/intechopen.80026 · Cited by 40
“7% WER with a trigram language model for Wall Street Journal dataset [4].”
MethodEnd-to-end Multimodal Speech Recognition
Palaskar, S., Sanabria, R., & Metze, F. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462439 · Cited by 37
Metadata
- Authors
- Chorowski, J., & Jaitly, N.
- Affiliations
- Google (United States) (US)
- ORCID
- Chorowski, Jan
- Venue
- Interspeech 2017
- Publisher
- ISCA
- Published
- 2017-08-20
- Pages
- 523-527
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2577366047
- Cited by
- 358
- References
- 34
- Citations by year
- 2026: 4 · 2025: 14 · 2024: 13 · 2023: 39 · 2022: 26 · 2021: 73 · 2020: 88 · 2019: 49 · 2018: 40 · 2017: 12
- FWCI
- 23.58
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Keywords
- attention-based models, language model integration, trigram language model, end-to-end speech recognition, speaker-independent recognition, Wall Street Journal
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.