Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks
The DOI matches its Crossref record; 30 of the 37 references it cites were found in a registry or an index.
Abstract
Both Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) have shown improvements over Deep Neural Networks (DNNs) across a wide variety of speech recognition tasks. CNNs, LSTMs and DNNs are complementary in their modeling capabilities, as CNNs are good at reducing frequency variations, LSTMs are good at temporal modeling, and DNNs are appropriate for mapping features to a more separable space. In this paper, we take advantage of the complementarity of CNNs, LSTMs and DNNs by combining them into one unified architecture. We explore the proposed architecture, which we call CLDNN, on a variety of large vocabulary tasks, varying from 200 to 2,000 hours. We find that the CLDNN provides a 4-6% relative improvement in WER over an LSTM, the strongest of the three individual models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The combination of CNNs and LSTMs in a unified framework has already offered state-of-the-art results in the speech recognition domain, where modelling temporal information is required [16].”
From Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition (2016) · cites this paper as [16]
In the citing paper
“This kind of architecture is able to capture time dependencies on features extracted by convolutional operations.”
From Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition (2016) · cites this paper as [16]
In the citing paper
“in the time series domain, we can see some examples of CNNs where not every convolutional layer is followed by a pooling operation [16].”
From Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition (2016) · cites this paper as [16]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
(2010) · Cited by 14,134
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationLarge Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · neural information processing systems · Cited by 4,489
Self-citationOn the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310 · cited 1×
“One may argue that if the LSTMs are better initialized, such that better temporal modeling can be performed, are CNNs really necessary? Our initial experiments with LSTMs use Gaussian random weight initialization, which produces eigenvalues of the initial recurrent network which are close to zero, thus increasing the…”
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973 · cited 5×
“In fact, the original LSTM work in [3] looked at modeling a sequence of 20 consecutive short-term xt C .”
BackgroundMethodInfluentialSelf-citationDeep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · IEEE Signal Processing Magazine · Cited by 2,136
Self-citationDeng, L., Hinton, G., & Kingsbury, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639344 · Cited by 1,178
Deep convolutional neural networks for LVCSR
Sainath, T. N., Mohamed, A.-R., Kingsbury, B., & Ramabhadran, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639347 · Cited by 1,053 · cited 14×
“To show the benefit of CNNs over DNNs, we also report results for adding the DNN before the LSTM layer.”
BackgroundMethodInfluentialSelf-citationConversational speech transcription using context-dependent deep neural networks
Seide, F., Li, G., & Yu, D. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-169 · Cited by 852
Sak, H., Senior, A., & Beaufays, F. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.1128 · Cited by 838
Self-citationSequential Deep Learning for Human Action Recognition
Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., & Baskurt, A. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25446-8_4 · Cited by 782
Traffic sign recognition with multi-scale Convolutional Networks
Sermanet, P., & LeCun, Y. (2011) · The 2011 International Joint Conference on Neural Networks · doi:10.1109/ijcnn.2011.6033589 · Cited by 759 · cited 2×
“Each CNN, LSTM and DNN block captures information about the input representation at different scales [10].”
Method(2014) · International Conference on Learning Representations · Cited by 634
Understanding how Deep Belief Networks perform acoustic modelling
Mohamed, A.-R., Hinton, G., & Penn, G. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288863 · Cited by 265 · cited 2×
“For example, it has been shown that CNNs learn speakeradapted/discriminatively trained features, and thus remove variation in the input [5].”
BackgroundEnsemble deep learning for speech recognition
Deng, L., & Platt, J. C. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-433 · Cited by 248 · cited 1×
“Combining CNN, LSTMs and DNNs has been explored in [9].”
MethodImprovements to Deep Convolutional Neural Networks for LVCSR
Sainath, T. N., Kingsbury, B., Mohamed, A.-R., Dahl, G. E., Saon, G., Soltau, H., Beran, T., Aravkin, A. Y., & Ramabhadran, B. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707749 · Cited by 243
Self-citationLattice-based optimization of sequence classification criteria for neural-network acoustic modeling
Kingsbury, B. (2009) · 2009 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2009.4960445 · Cited by 232 · cited 1×
“In addition, in this section we report numbers after both cross-entropy (CE) and sequence training [17], a strategy which has shown to give consistent gains over CE training [20].”
BackgroundHierarchical Structures of Neural Networks for Phoneme Recognition
Schwarz, P., Matejka, P., & Cernocky, J. (2006) · 2006 IEEE International Conference on Acoustics Speed and Signal Processing Proceedings · doi:10.1109/icassp.2006.1660023 · Cited by 202 · cited 1×
“, [13, 14]).”
The IBM Attila speech recognition toolkit
Soltau, H., Saon, G., & Kingsbury, B. (2010) · 2010 IEEE Spoken Language Technology Workshop · doi:10.1109/slt.2010.5700829 · Cited by 116 · cited 1×
“In fact, state-of-the art GMM/HMM systems perform speaker adaptation, using techniques such as vocal tract length normalization (VTLN) and feature-space maximum likelihood linear regression (fMLLR), before performing temporal modeling via HMMs [6].”
MethodSpatio-Temporal Convolutional Sparse Auto-Encoder for Sequence Classification
Baccouche, M., Mamalet, F., Wolf, C., García, C., & Baskurt, A. M. (2012) · British Machine Vision Conference (BMVC) · doi:10.5244/c.26.124 · Cited by 95
Foundations and Trends in Signal Processing: DEEP LEARNING – Methods and Applications
Deng, L., & Yu, D. (2014) · Cited by 81
Exemplar-Based Sparse Representation Features: From TIMIT to LVCSR
Sainath, T. N., Ramabhadran, B., Picheny, M., Nahamoo, D., & Kanevsky, D. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2155060 · Cited by 75 · cited 1×
“This recipe order has been shown to be appropriate for LVCSR tasks [7].”
MethodSelf-citationJoint training of convolutional and non-convolutional neural networks
Soltau, H., Saon, G., & Sainath, T. N. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854669 · Cited by 72
Self-citationAsynchronous stochastic optimization for sequence training of deep neural networks
Heigold, G., McDermott, E., Vanhoucke, V., Senior, A., & Bacchiani, M. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854672 · Cited by 64
Self-citationSequence classification using the high-level features extracted from deep neural networks
Deng, L., & Chen, J. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854926 · Cited by 49
Unfolded recurrent neural networks for speech recognition
Saon, G., Soltau, H., Emami, A., & Picheny, M. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-81 · Cited by 47 · cited 2×
“This is similar in spirit to the hidden to output model proposed in [4], and also tested for speech, though with RNNs [8].”
ResultSemi-supervised bootstrapping approach for neural network feature extractor training
Grezl, F., & Karafiát, M. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707775 · Cited by 45 · cited 1×
“, [13, 14]).”
Deep scattering spectra with deep neural networks for LVCSR tasks
Sainath, T. N., Peddinti, V., Kingsbury, B. E. D., Fousek, P., Ramabhadran, B., & Nahamoo, D. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-225 · Cited by 23 · cited 1×
“In [12] we found that adding this linear layer after the CNN layers allows for a reduction in parameters with no loss in accuracy.”
BackgroundSelf-citation
Cited by
A Review of Recurrent Neural Networks: LSTM Cells and Network Architectures
Yu, Y., Si, X., Hu, C., & Zhang, J. (2019) · Neural Computation · doi:10.1162/neco_a_01199 · Cited by 5,591
The Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
Deep learning with convolutional neural networks for EEG decoding and visualization
Schirrmeister, R. T., Springenberg, J. T., Fiederer, L. D. J., Glasstetter, M., Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W., & Ball, T. (2017) · Human Brain Mapping · doi:10.1002/hbm.23730 · Cited by 3,751
“, 2016); however, in this field, hybrid models combining ConvNets with other machine-learning components, notably recurrent networks, and deep neural networks without convolutions are also competitive (Li and Wu, 2015; Sainath et al., 2015b; Sak et al., 2015).”
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761
“Nevertheless, such an approach is not strictly part of the architecture, and in the time series domain, we can see some examples of CNNs where not every convolutional layer is followed by a pooling operation [16].”
BackgroundCNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952132 · Cited by 2,023
Metadata
- Authors
- Sainath, T. N., Vinyals, O., Senior, A., & Sak, H.
- Affiliations
- Google (United States) (US)
- Venue
- 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2015-04
- Pages
- 4580-4584
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W1600744878
- Cited by
- 1,678
- References
- 37
- Citations by year
- 2026: 57 · 2025: 136 · 2024: 140 · 2023: 148 · 2022: 179 · 2021: 241 · 2020: 215 · 2019: 189 · 2018: 172 · 2017: 112 · 2016: 68 · 2015: 19
- FWCI
- 69.69
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- convolutional neural networks, long short-term memory, deep neural network, speech recognition, large vocabulary speech recognition, word error rate, temporal modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.