The DOI matches its Crossref record; 207 of the 207 references it cites were found in a registry or an index.
A tutorial survey of architectures, algorithms, and applications for deep learning
Open accessAbstract
In this invited paper, my overview material on the same topic as presented in the plenary overview session of APSIPA-2011 and the tutorial material presented in the same conference [1] are expanded and updated to include more recent developments in deep learning. The previous and the updated materials cover both theory and applications, and analyze its future directions. The goal of this tutorial survey is to introduce the emerging area of deep learning or hierarchical learning to the APSIPA community. Deep learning refers to a class of machine learning techniques, developed largely since 2006, where many stages of non-linear information processing in hierarchical architectures are exploited for pattern classification and for feature learning. In the more recent literature, it is also connected to representation learning, which involves a hierarchy of features or concepts where higher-level concepts are defined from lower-level ones and where the same lower-level concepts help to define higher-level ones. In this tutorial survey, a brief history of deep learning research is discussed first. Then, a classificatory scheme is developed to analyze and summarize major work reported in the recent deep learning literature. Using this scheme, I provide a taxonomy-oriented survey on the existing deep architectures and algorithms in the literature, and categorize them into three classes: generative, discriminative, and hybrid. Three representative deep architectures – deep autoencoders, deep stacking networks with their generalization to the temporal domain (recurrent networks), and deep neural networks (pretrained with deep belief networks) – one in each of the three classes, are presented in more detail. Next, selected applications of deep learning are reviewed in broad areas of signal and information processing including audio/speech, image/vision, multimodality, language modeling, natural language processing, and information retrieval. Finally, future directions of deep learning are discussed and analyzed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
(2010) · Cited by 14,134
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationLearning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
(2012) · Cited by 8,009
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951
(2011) · Cited by 6,993
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 2×
“More recent work on neural network LM with deep architectures can be found in [51, 200–203].”
BackgroundResultGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Convolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.5555/303568.303704 · Cited by 4,231
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051
The Handbook of brain theory and neural networks
(1996) · Choice Reviews Online · doi:10.5860/choice.33-4484 · Cited by 3,870
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,479
Large Scale Distributed Deep Networks
Dean, J. B., Corrado, G. S., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P. A., Yang, K., et al. (2012) · Cited by 3,199
Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011
Self-citationA Practical Guide to Training Restricted Boltzmann Machines
Hinton, G. E. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_32 · Cited by 2,852
(2011) · Cited by 2,752
Phoneme recognition using time-delay neural networks
Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., & Lang, K. J. (1989) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/29.21701 · Cited by 2,643
Why Does Unsupervised Pre-training Help Deep Learning?
Erhan, D., Bengio, Y., Courville, A. C., Manzagol, P.-A., Vincent, P., & Samy Bengio, A. (2010) · Journal of Machine Learning Research · Cited by 2,631
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412
What is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153
Deep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · IEEE Signal Processing Magazine · Cited by 2,115
Self-citationLearning deep structured semantic models for web search using clickthrough data
Huang, P.-S., He, X., Gao, J., Deng, L., Acero, A., & Heck, L. (2013) · Proceedings of the 22nd ACM international conference on Information & Knowledge Management · doi:10.1145/2505515.2505665 · Cited by 2,009
Self-citationSalakhutdinov, R., & Hinton, G. E. (2009) · International Conference on Artificial Intelligence and Statistics · Cited by 1,890
Generating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · International Conference on Machine Learning · Cited by 1,783
Acoustic Modeling Using Deep Belief Networks
Mohamed, A.-R., Dahl, G. E., & Hinton, G. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2109382 · Cited by 1,727
Building high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695 · cited 2×
“Nevertheless, the CNN has been found highly effective and been commonly used in computer vision and image recognition (LeCun et al., 1998; Ciresan et al., 2012; Le et al., 2012; Dean et al., 2012; Krizhevsky et al., 2012).”
BackgroundMethodContractive Auto-Encoders: Explicit Invariance During Feature Extraction
Rifai, S., Vincent, P., Muller, X., Glorot, X., & Bengio, Y. (2011) · International Conference on Machine Learning · Cited by 1,600
Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions
Socher, R., Pennington, J., Huang, E., Ng, A. Y., & Manning, C. D. (2011) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · Cited by 1,431
Hinton, G. E., Krizhevsky, A., & Wang, S. D. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-21735-7_6 · Cited by 1,360
(2012) · neural information processing systems · Cited by 1,291
Salakhutdinov, R., & Hinton, G. (2009) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2008.11.006 · Cited by 1,246
(2011) · Cited by 1,192
Deng, L., Hinton, G., & Kingsbury, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639344 · Cited by 1,178
Self-citationSequence Transduction with Recurrent Neural Networks
Graves, A. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1211.3711 · Cited by 1,170
Deep Machine Learning - A New Frontier in Artificial Intelligence Research [Research Frontier]
Arel, I., Rose, D. C., & Karnowski, T. P. (2010) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2010.938364 · Cited by 1,121
Connectionist Speech Recognition: A Hybrid Approach
Bourlard, H. A., & Morgan, N. H. (1993) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 1,115
Efficient Learning of Sparse Representations with an Energy-Based Model
Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0147 · Cited by 1,106
On optimization methods for deep learning
Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., Le, Q. V., & Ng, A. Y. (2011) · International Conference on Machine Learning · Cited by 1,076
Improving Word Representations via Global Context and Multiple Word Prototypes
Huang, E., Socher, R., Manning, C. D., & Ng, A. Y. (2012) · Cited by 1,073
Deep convolutional neural networks for LVCSR
Sainath, T. N., Mohamed, A.-R., Kingsbury, B., & Ramabhadran, B. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639347 · Cited by 1,053
(2009) · neural information processing systems · Cited by 962
The hierarchical hidden Markov model: Analysis and applications
Fine, S., Singer, Y., & Tishby, N. (1998) · Machine Learning · doi:10.1023/a:1007469218079 · Cited by 906
Applying Convolutional Neural Networks concepts to hybrid NN-HMM model for speech recognition
Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., & Penn, G. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288864 · Cited by 876
Conversational speech transcription using context-dependent deep neural networks
Seide, F., Li, G., & Yu, D. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-169 · Cited by 852
A Scalable Hierarchical Distributed Language Model
Mnih, A., & Hinton, G. E. (2008) · Cited by 838
Building high-level features using large scale unsupervised learning
Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G. S., Dean, J., Le, Q. V., & Ng, A. Y. (2012) · International Conference on Machine Learning · Cited by 822
Statistical parametric speech synthesis using deep neural networks
Zen, H., Senior, A., & Schuster, M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639215 · Cited by 809
Recent advances in deep learning for speech research at Microsoft
Deng, L., Li, J., Huang, J.-T., Yao, K., Yu, D., Seide, F., Seltzer, M., Zweig, G., He, X., Williams, J., et al. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639345 · Cited by 801
Self-citationDynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection
Socher, R., Huang, E., Pennin, J., Manning, C. D., & Ng, A. Y. (2011) · neural information processing systems · Cited by 782
Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
Zeiler, M. D., & Fergus, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3557 · Cited by 755
Sparse Feature Learning for Deep Belief Networks
Ranzato, M., Boureau, Y.-L., & LeCun, Y. (2007) · neural information processing systems · Cited by 744
Kernel Methods for Deep Learning
Cho, Y., & Saul, L. K. (2009) · eScholarship (California Digital Library) · Cited by 722
Modeling Human Motion Using Binary Latent Variables
Taylor, G. W., Hinton, G. E., & Roweis, S. T. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0173 · Cited by 689
Classification using discriminative restricted Boltzmann machines
Larochelle, H., & Bengio, Y. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390224 · Cited by 655
(2013) · Cited by 651
Minimum Phone Error and I-smoothing for improved discriminative training
Povey, D., & Woodland, P. C. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5743665 · Cited by 622
Minimum classification error rate methods for speech recognition
Juang, B.-H., Hou, W., & Lee, C.-H. (1997) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.568732 · Cited by 618
Learning Convolutional Feature Hierarchies for Visual Recognition
Kavukcuoglu, K., Sermanet, P., Boureau, Y.-L., Gregor, K., Mathieu, M., & LeCun, Y. (2010) · neural information processing systems · Cited by 604
A time-delay neural network architecture for isolated word recognition
Lang, K. J., Waibel, A. H., & Hinton, G. E. (1990) · Neural Networks · doi:10.1016/0893-6080(90)90044-l · Cited by 596
From HMM's to segment models: a unified view of stochastic modeling for speech recognition
Ostendorf, M., Digalakis, V. V., & Kimball, O. A. (1996) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.536930 · Cited by 587
Three new graphical models for statistical language modelling
Mnih, A., & Hinton, G. (2007) · Proceedings of the 24th international conference on Machine learning · doi:10.1145/1273496.1273577 · Cited by 563
Learning Recurrent Neural Networks with Hessian-Free Optimization
Martens, J., & Sutskever, I. (2011) · International Conference on Machine Learning · Cited by 525
Bilingual Word Embeddings for Phrase-Based Machine Translation
Zou, W. Y., Socher, R., Cer, D., & Manning, C. D. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1141 · Cited by 518
Deep Learning of Representations: Looking Forward
Bengio, Y. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-39593-2_1 · Cited by 513
Strategies for training large scale neural network language models
Mikolov, T., Deoras, A., Povey, D., Burget, L., & Cernocky, J. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163930 · Cited by 509
The handbook of brain theory and neural networks
Rattay, F. (1996) · Simulation Practice and Theory · doi:10.1016/0928-4869(96)83761-x · Cited by 500
Advances in optimizing recurrent networks
Bengio, Y., Boulanger-Lewandowski, N., & Pascanu, R. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639349 · Cited by 469
Ma, J., & Plonka, G. (2010) · IEEE Signal Processing Magazine · doi:10.1109/msp.2009.935453 · Cited by 464
Machine Learning Paradigms for Speech Recognition: An Overview
Deng, L., & Li, X. (2013) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2013.2244083 · Cited by 456
Self-citationDeep Learning and Its Applications to Signal and Information Processing [Exploratory DSP
Yu, D., & Deng, L. (2011) · IEEE Signal Processing Magazine · doi:10.1109/msp.2010.939038 · Cited by 455
Self-citationAn application of recurrent nets to phone probability estimation
Robinson, A. J. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279192 · Cited by 441
Exponential Family Harmoniums with an Application to Information Retrieval
Welling, M., Rosen‐Zvi, M., & Hinton, G. E. (2004) · Cited by 423
Mesnil, G., He, X., Deng, L., & Bengio, Y. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-596 · Cited by 413
Self-citationUnsupervised learning of hierarchical representations with convolutional deep belief networks
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2011) · Communications of the ACM · doi:10.1145/2001269.2001295 · Cited by 388
Exploring convolutional neural network structures and optimization techniques for speech recognition
Abdel-Hamid, O., Deng, L., & Yu, D. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-744 · Cited by 384
Self-citationTraining recurrent neural networks
Hinton, G. E., & Sutskever, I. (2013) · Cited by 383
Binary coding of speech spectrograms using a deep auto-encoder
Deng, L., Seltzer, M. L., Yu, D., Acero, A., Mohamed, A.-R., & Hinton, G. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-487 · Cited by 375
Self-citationBottou, L., & Cun, Y. L. (2003) · Cited by 359
3D Object Recognition with Deep Belief Nets
Nair, V. B., & Hinton, G. E. (2009) · neural information processing systems · Cited by 356
Sum-Product Networks: A New Deep Architecture
Poon, H., & Domingos, P. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1202.3732 · Cited by 352
Recurrent neural networks for noise reduction in robust ASR
Maas, A. L., Le, Q. V., O'Neil, T. M., Vinyals, O., Nguyen, P., & Ng, A. Y. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-6 · Cited by 334
Layered representations for learning and inferring office activity from multiple sensory channels
Oliver, N., Garg, A., & Horvitz, E. (2004) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2004.02.004 · Cited by 316
Sum-product networks: A new deep architecture
Poon, H., & Domingos, P. (2011) · 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) · doi:10.1109/iccvw.2011.6130310 · Cited by 315
Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine
Dahl, G. E., Ranzato, M., Mohamed, A., & Hinton, G. E. (2010) · Cited by 266
Understanding how Deep Belief Networks perform acoustic modelling
Mohamed, A.-R., Hinton, G., & Penn, G. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288863 · Cited by 265
Application of pretrained deep neural networks to large vocabulary speech recognition
Jaitly, N., Nguyen, P., Senior, A., & Vanhoucke, V. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-10 · Cited by 243
On deep generative models with applications to recognition
Ranzato, M., Susskind, J., Mnih, V., & Hinton, G. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995710 · Cited by 237
Deng, L., & O'Shaughnessy, D. (2018) · CRC Press eBooks · doi:10.1201/9781482276237 · Cited by 236
Self-citationMaximum likelihood estimation for multivariate mixture observations of markov chains (Corresp.)
Juang, B.-H., Levinson, S., & Sondhi, M. (1986) · IEEE Transactions on Information Theory · doi:10.1109/tit.1986.1057145 · Cited by 235
Lattice-based optimization of sequence classification criteria for neural-network acoustic modeling
Kingsbury, B. (2009) · 2009 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2009.4960445 · Cited by 232
Learning a better representation of speech soundwaves using restricted boltzmann machines
Jaitly, N., & Hinton, G. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947700 · Cited by 231 · cited 1×
“The work also showed the importance of raw speech features of spectrogram --- back from the longpopular MFCC features, but not yet reaching the raw speechwaveform level (Sheikhzadeh and Deng, 1994; Jaitly and Hinton, 2011).”
BackgroundDevelopments and directions in speech recognition and understanding, Part 1 [DSP Education]
Baker, J., Deng, L., Glass, J., Khudanpur, S., Lee, C.-H., Morgan, N., & O'Shaughnessy, D. (2009) · IEEE Signal Processing Magazine · doi:10.1109/msp.2009.932166 · Cited by 228
Self-citationKingsbury, B., Sainath, T. N., & Soltau, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-3 · Cited by 220
Investigation of full-sequence training of deep belief networks for speech recognition
Mohamed, A.-R., Yu, D., & Deng, L. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-304 · Cited by 210
Self-citationScalable stacking and learning for building deep architectures
Deng, L., Yu, D., & Platt, J. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6288333 · Cited by 210
Self-citationGlobal optimization of a neural network-hidden Markov model hybrid
Bengio, Y., De Mori, R., Flammia, G., & Kompe, R. (1992) · IEEE Transactions on Neural Networks · doi:10.1109/72.125866 · Cited by 203
Yu, D., Deng, L., & Dahl, G. E. (2010) · neural information processing systems · Cited by 189
Self-citationMaking Deep Belief Networks effective for large vocabulary continuous speech recognition
Sainath, T. N., Kingsbury, B., Ramabhadran, B., Fousek, P., Novak, P., & Mohamed, A.-R. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163900 · Cited by 189
Speech translation: coupling of recognition and translation
Ney, H. (1999) · 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258) · doi:10.1109/icassp.1999.758176 · Cited by 189
Deng, L., Abdel-Hamid, O., & Yu, D. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638952 · Cited by 186
Self-citationHumphrey, E. J., Bello, J. P., & LeCun, Y. (2012) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1415725 · Cited by 166
Hutchinson, B., Deng, L., & Yu, D. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.268 · Cited by 165
Self-citationChallenges in adopting speech recognition
Deng, L., & Huang, X. (2004) · Communications of the ACM · doi:10.1145/962081.962108 · Cited by 165
Self-citationLearning Invariant Feature Hierarchies
LeCun, Y. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33863-2_51 · Cited by 164
Deep Learning and Its Applications to Signal and Information Processing
Yu, D., & Deng, L. (2011) · IEEE Signal Processing Magazine · Cited by 159
Self-citationDeep Learning for Efficient Discriminative Parsing
Collobert, R. (2011) · Cited by 158
How the brain might work: a hierarchical and temporal model for learning and recognition
George, D. (2008) · Cited by 156
Deng, L., Aksmanovic, M., Sun, X., & Wu, C. (1994) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.326610 · Cited by 152
Self-citationDeep convex net: a scalable architecture for speech pattern classification
Deng, L., & Yu, D. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-607 · Cited by 152
Self-citationLing, Z.-H., Deng, L., & Yu, D. (2013) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2013.2269291 · Cited by 150
Self-citationDeep Learning for NLP (without Magic)
Socher, R., Bengio, Y., & Manning, C. D. (2012) · Cited by 150
Deng, L., & Sun, D. X. (1994) · The Journal of the Acoustical Society of America · doi:10.1121/1.409839 · Cited by 144
Self-citationDiscriminative learning in sequential pattern recognition
He, X., Li, D., & Chou, W. (2008) · IEEE Signal Processing Magazine · doi:10.1109/msp.2008.926652 · Cited by 143
Towards deeper understanding: Deep convex networks for semantic utterance classification
Tür, G., Deng, L., Hakkani-Tür, D. Z., & He, X. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6289054 · Cited by 143
Self-citationExploiting sparseness in deep neural networks for large vocabulary speech recognition
Yu, D., Seide, F. T. B., Li, G., & Deng, L. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288897 · Cited by 139
Self-citationNgiam, J., Chen, Z., Koh, P. W., & Ng, A. Y. (2011) · International Conference on Machine Learning · Cited by 137
Structured Output Layer neural network language model
Le, H.-S., Oparin, I., Allauzen, A., Gauvain, J., & Yvon, F. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5947610 · Cited by 136
Deep and Wide: Multiple Layers in Automatic Speech Recognition
Morgan, N. H. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2116010 · Cited by 130
Stoyanov, V., Ropson, A., & Eisner, J. M. (2011) · International Conference on Artificial Intelligence and Statistics · Cited by 124
Peng, J., Bo, L., & Xu, J. (2009) · Cited by 122
Exploiting deep neural networks for detection-based speech recognition
Siniscalchi, S. M., Yu, D., Deng, L., & Lee, C. (2012) · Neurocomputing · doi:10.1016/j.neucom.2012.11.008 · Cited by 120
Self-citationUse of kernel deep convex networks and end-to-end learning for spoken language understanding
Deng, L., Tür, G., He, X., & Hakkani-Tür, D. Z. (2012) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2012.6424224 · Cited by 115
Self-citationGraphical model architectures for speech recognition
Bilmes, J. A., & Bartels, C. D. (2005) · IEEE Signal Processing Magazine · doi:10.1109/msp.2005.1511827 · Cited by 115
The Deep Tensor Neural Network With Applications to Large Vocabulary Speech Recognition
Yu, D., Deng, L., & Seide, F. T. B. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2012.2227738 · Cited by 114
Self-citationA Better Way to Pretrain Deep Boltzmann Machines
Hinton, G. E., & Salakhutdinov, R. (2012) · neural information processing systems · Cited by 112
Discovering Binary Codes for Documents by Learning Deep Generative Models
Hinton, G. E., & Salakhutdinov, R. (2010) · Topics in Cognitive Science · doi:10.1111/j.1756-8765.2010.01109.x · Cited by 111
Deng, L. (1992) · Signal Processing · doi:10.1016/0165-1684(92)90112-a · Cited by 109
Self-citationPapandreou, G., Katsamanis, A., Pitsikalis, V., & Maragos, P. A. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2008.2011515 · Cited by 109
A segmental CRF approach to large vocabulary continuous speech recognition
Zweig, G., & Nguyen, P. (2009) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2009.5372916 · Cited by 106
Multi-distribution deep belief network for speech synthesis
Kang, S., Qian, X., & Meng, H. M. L. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639225 · Cited by 105
Pushing the envelope - aside [speech recognition
Morgan, N., Zhu, Q., Stolcke, A., Sönmez, K., Sivadas, S., Shinozaki, T., Mari Ostendorf, E., Jain, P., Heřmanský, H., Ellis, D. P. W., et al. (2005) · IEEE Signal Processing Magazine · doi:10.1109/msp.2005.1511826 · Cited by 96
Cohen, W. W., & Carvalho, V. R. (2005) · Cited by 94
Learning emotion-based acoustic features with deep belief networks
Schmidt, E. M., & Kim, Y. E. (2011) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/aspaa.2011.6082328 · Cited by 89
Deng, L., & Ma, J. (2000) · The Journal of the Acoustical Society of America · doi:10.1121/1.1315288 · Cited by 89
Self-citationDeep stacking networks for information retrieval
Deng, L., He, X., & Gao, J. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638239 · Cited by 87
Self-citationProduction models as a structural basis for automatic speech recognition
Deng, L., Ramsay, G., & Sun, D. (1997) · Speech Communication · doi:10.1016/s0167-6393(97)00018-6 · Cited by 84
Self-citationLi, D., Yu, D., & Acero, A. (2006) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2006.878265 · Cited by 84
Analysis of MLP-Based Hierarchical Phoneme Posterior Probability Estimator
Pinto, J., Garimella, S., Magimai.-Doss, M., Hynek Hermansky, B., & Bourlard, H. A. (2010) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2010.2045943 · Cited by 80
Speech Recognition Using Augmented Conditional Random Fields
Hifny, Y., & Renals, S. J. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2008.2010286 · Cited by 80
Sun, J., & Deng, L. (2002) · The Journal of the Acoustical Society of America · doi:10.1121/1.1420380 · Cited by 71
Self-citationLee, C.-H. (2004) · Interspeech (USB) · Cited by 70
Efficient and effective algorithms for training single-hidden-layer neural networks
Yu, D., & Deng, L. (2011) · Pattern Recognition Letters · doi:10.1016/j.patrec.2011.12.002 · Cited by 67
Self-citationCalibration of Confidence Measures in Speech Recognition
Yu, D., Li, J., & Deng, L. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2141988 · Cited by 67
Self-citationYu, D., Siniscalchi, S. M., Deng, L., & Lee, C. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288837 · Cited by 66
Self-citationSparse Multilayer Perceptron for Phoneme Recognition
Sivaram, G. S. V. S., & Heřmanský, H. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2129510 · Cited by 65
A Probabilistic Interaction Model for Multipitch Tracking With Factorial Hidden Markov Models
Wohlmayr, M., Stark, M. M., & Pernkopf, F. (2010) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2010.2064309 · Cited by 63
Autoregressive Models for Statistical Parametric Speech Synthesis
Shannon, M., Zen, H., & Byrne, B. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2012.2227740 · Cited by 61
An Overview of Deep-Structured Learning for Information Processing
Deng, L. (2011) · Cited by 60
Self-citationContinuous Stochastic Feature Mapping Based on Trajectory HMMs
Zen, H., Nankaku, Y., & Tokuda, K. (2010) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2010.2049685 · Cited by 60
Learning with Recursive Perceptual Representations
Vinyals, O., Jia, Y., Deng, L., & Darrell, T. (2012) · Cited by 60
Self-citationUpdated MINDS report on speech recognition and understanding, Part 2 [DSP Education]
Baker, J. M., Deng, L., Khudanpur, S. P., Lee, C., Glass, J. R., Morgan, N., & O’Shaughnessy, D. D. (2009) · IEEE Signal Processing Magazine · doi:10.1109/msp.2009.932707 · Cited by 59
Self-citationSheikhzadeh, H., & Deng, L. (1994) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.260337 · Cited by 56
Self-citationLarge-Margin Minimum Classification Error Training for Large-Scale Speech Recognition Tasks
Yu, D., Deng, L., He, X., & Acero, A. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.367275 · Cited by 55
Self-citationBridle, J. S. (1998) · Medical Entomology and Zoology · Cited by 53
Energy-Based Models in Document Recognition and Computer Vision
LeCun, Y., Chopra, S., Ranzato, M., & Huang, F.-S. (2007) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2007.4378728 · Cited by 52
Computational Models for Speech Production
Deng, L. (1999) · Springer eBooks · doi:10.1007/978-3-642-60087-6_20 · Cited by 51
Self-citationDeng, L., Kenny, P. J., Lennig, M., Gupta, V., Seitz, F., & Mermelstein, P. G. (1991) · IEEE Transactions on Signal Processing · doi:10.1109/78.134406 · Cited by 51
Self-citationProduct of Experts for Statistical Parametric Speech Synthesis
Zen, H., Gales, M., Nankaku, Y., & Tokuda, K. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2165280 · Cited by 50
Mathematical Foundations of Speech and Language Processing
Johnson, M., Khudanpur, S., Mari Ostendorf, E., & Rosenfeld, R. (2004) · The IMA volumes in mathematics and its applications · doi:10.1007/978-1-4419-9017-4 · Cited by 47
Joint state and parameter estimation for a target-directed nonlinear dynamic system model
Togneri, R. B., & Deng, L. (2003) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2003.819013 · Cited by 47
Self-citationHutchinson, B., Deng, L., & Yu, D. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6288994 · Cited by 46
Self-citationSequential Labeling Using Deep-Structured Conditional Random Fields
Yu, D., Wang, S., & Deng, L. (2010) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2010.2075990 · Cited by 46
Self-citationLing, Z.-H., Deng, L., & Yu, D. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639187 · Cited by 46
Self-citationA scalable approach to using DNN-derived features in GMM-HMM based acoustic modeling for LVCSR
Yan, Z., Huo, Q., & Xu, J. F. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-47 · Cited by 46
Deng, L., & Yu, D. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366945 · Cited by 45
Self-citationSpeech-Centric Information Processing: An Optimization-Oriented Approach
He, X., & Deng, L. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2236631 · Cited by 44
Self-citationDeng, L. F., & Aksmanovic, M. (1997) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.593305 · Cited by 44
Self-citationHierarchical Bayesian Language Models for Conversational Speech Recognition
Huang, S., & Renals, S. J. (2010) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2010.2040782 · Cited by 44
SWITCHING DYNAMIC SYSTEM MODELS FOR SPEECH ARTICULATION AND ACOUSTICS
Denc, L. (2004) · Cited by 43
A Novel Framework and Training Algorithm for Variable-Parameter Hidden Markov Models
Yu, D., Deng, L., Gong, Y., & Acero, A. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2009.2020890 · Cited by 41
Self-citationEnhanced Phone Posteriors for Improving Speech Recognition Systems
Ketabdar, H., & Bourlard, H. A. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2009.2023162 · Cited by 41
Large vocabulary word recognition using context-dependent allophonic hidden Markov models
Deng, L., Lennig, M., Seitz, F., & Mermelstein, P. (1990) · Computer Speech & Language · doi:10.1016/0885-2308(90)90015-x · Cited by 40
Self-citationBilmes, J. (2010) · IEEE Signal Processing Magazine · doi:10.1109/msp.2010.938078 · Cited by 40
Deep segmental neural networks for speech recognition
Abdel‐Hamid, O., Deng, L., Yu, D., & Jiang, H. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-455 · Cited by 39
Self-citationLanguage recognition using deep-structured conditional random fields
Yu, D., Wang, S., Karam, Z. N., & Deng, L. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5495072 · Cited by 38
Self-citationDeng, L., Hassanein, K., & Elmasry, M. I. (1994) · Neural Networks · doi:10.1016/0893-6080(94)90027-2 · Cited by 36
Self-citationTarget-Directed Mixture Dynamic Models for Spontaneous Speech Recognition
Ma, J., & Deng, L. (2004) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2003.818074 · Cited by 36
Self-citationDeep-structured hidden conditional random fields for phonetic recognition
Yu, D., & Deng, L. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-35 · Cited by 36
Self-citationEquivalence of Generative and Log-Linear Models
Heigold, G., Ney, H., Lehnen, P., Gass, T., & Schlüter, R. (2010) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2010.2082532 · Cited by 34
Dynamic Speech Models: Theory, Algorithms, and Applications
Deng, L. (2006) · Synthesis lectures on speech and audio processing · doi:10.1007/978-3-031-02555-6 · Cited by 34
Self-citationMulti-style adaptive training for robust cross-lingual spoken language understanding
He, X., Deng, L., Hakkani-Tür, D. Z., & Tür, G. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639292 · Cited by 33
Self-citationA stochastic model of speech incorporating hierarchical nonstationarity
Li, D. (1993) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.242494 · Cited by 32
Training Continuous Space Language Models: Some Practical Issues
Le, H.-S., Allauzen, A., Wisniewski, G., & Yvon, F. (2010) · Empirical Methods in Natural Language Processing · Cited by 31
Learning Semantic Representations for the Phrase Translation ModelPreprint
Gao, J., He, X., Yih, W.-T., & Deng, L. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.0482 · Cited by 28
Self-citationDeng, L., Yu, D., & Acero, A. (2005) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tsa.2005.854107 · Cited by 28
Self-citationHierarchical learning : theory with applications in speech and vision
Bouvrie, J. (2009) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 26
Ma, J., & Deng, L. (2003) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2003.818075 · Cited by 26
Self-citation
Cited by
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review
Rawat, W., & Wang, Z. (2017) · Neural Computation · doi:10.1162/neco_a_00990 · Cited by 3,503
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239
“its ability to handle large amounts of unlabeled data, deep l arning techniques have provided powerful tools to deal with big data analysis [31], [122].”
BackgroundSarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00815-1 · Cited by 2,678
Deep learning for visual understanding: A review
Guo, Y., Liu, Y., Oerlemans, A., Lao, S., Wu, S., & Lew, M. S. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.116 · Cited by 2,504
“Deep networks have been shown to be successful for computer vision tasks because they can extract appropriate features while jointly performing discrimination [9,13].”
BackgroundPrivacy-preserving deep learning
Shokri, R., & Shmatikov, V. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813687 · Cited by 2,356
Metadata
- Authors
- Deng, L.
- Venue
- APSIPA Transactions on Signal and Information Processing
- Publisher
- Emerald
- Published
- 2014
- Volume
- 3
- Issue
- 1
- ISSN
- 2048-7703 (ISSN-L)
- OpenAlex
- W2123585936
- Cited by
- 725
- References
- 207
- FWCI
- 39.94
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.