Learning a better representation of speech soundwaves using restricted boltzmann machines
Abstract
State of the art speech recognition systems rely on preprocessed speech features such as Mel cepstrum or linear predictive coding coefficients that collapse high dimensional speech sound waves into low dimensional encodings. While these have been successfully applied in speech recognition systems, such low dimensional encodings may lose some relevant information and express other information in a way that makes it difficult to use for discrimination. Higher dimensional encodings could both improve performance in recognition tasks, and also be applied to speech synthesis by better modeling the statistical structure of the sound waves. In this paper we present a novel approach for modeling speech sound waves using a Restricted Boltzmann machine (RBM) with a novel type of hidden variable and we report initial results demonstrating phoneme recognition performance better than the current state-of-the-art for methods based on Mel cepstrum coefficients.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
Self-citationAn Information-Maximization Approach to Blind Separation and Blind Deconvolution
Bell, A. J., & Sejnowski, T. J. (1995) · Neural Computation · doi:10.1162/neco.1995.7.6.1129 · Cited by 8,986 · see citing works
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 3× · see citing works
“a weight parameter, wij ∈ W is as follows [6]:”
BackgroundMethodSelf-citationA Practical Guide to Training Restricted Boltzmann Machines
Hinton, G. E. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_32 · Cited by 2,852 · see citing works
Self-citationInformation processing in dynamical systems: foundations of harmony theory
Smolensky, P. (1986) · The MIT Press eBooks · Cited by 1,673 · see citing works
Advances in Neural Information Processing Systems 19 (NIPS 2006)
Simonotto, E., Whalley, H. C., Lawrie, S. M., Murray, L. M., McGonigle, D. J., & Storkey, A. (2006) · Cited by 1,651 · see citing works
(2009) · neural information processing systems · Cited by 962 · see citing works
Efficient coding of natural sounds
Lewicki, M. S. (2002) · Nature Neuroscience · doi:10.1038/nn831 · Cited by 667 · see citing works
An application of recurrent nets to phone probability estimation
Robinson, A. J. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279192 · Cited by 441 · see citing works
Phone Recognition with the Mean-Covariance Restricted Boltzmann Machine
Dahl, G. E., Ranzato, M., Mohamed, A., & Hinton, G. E. (2010) · Cited by 268 · see citing works
Self-citationLearning to Represent Spatial Transformations with Factored Higher-Order Boltzmann Machines
Memisevic, R., & Hinton, G. E. (2010) · Neural Computation · doi:10.1162/neco.2010.01-09-953 · Cited by 227 · see citing works
Self-citationLarge Margin Gaussian Mixture Modeling for Phonetic Classification and Recognition
Sha, F., & Saul, L. K. (2006) · doi:10.1109/icassp.2006.1660008 · Cited by 124 · see citing works
Heterogeneous measurements and multiple classifiers for speech recognition
Halberstadt, A. K., & Glass, J. (1998) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1998-522 · Cited by 122 · see citing works
Speech feature extraction using independent component analysis
Lee, J., Jung, H., Lee, T.-W., & Lee, S. Y. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2000.862023 · Cited by 98 · see citing works
Speech Recognition Using Augmented Conditional Random Fields
Hifny, Y., & Renals, S. J. (2009) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2008.2010286 · Cited by 81 · see citing works
CUDAMat: a CUDA-based matrix class for Python
Mnih, V. (2009) · Cited by 68 · see citing works
Improved phone recognition using Bayesian triphone models
Ming, J., & Smith, F. J. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1998.674454 · Cited by 46 · see citing works
Deng, L., & Yu, D. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366945 · Cited by 45 · see citing works
Lewicki, M. S. (2010) · Nature · doi:10.1038/466821a · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239 · see citing works
“During the past few decades, machine learning algorithms have been widely used in areas such as automatic speech recognition (ASR) and acoustic modeling [76,116,118,126].”
MethodZheng, W.-L., & Lu, B.-L. (2015) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2015.2431497 · Cited by 2,477 · see citing works
“MLP, SVMs, CRFs) in many challenge tasks, especially in speech and image domains [29]–[31].”
BackgroundTowards End-To-End Speech Recognition with Recurrent Neural Networks
Graves, A., & Jaitly, N. (2014) · Cited by 1,866 · see citing works
Self-citationDeep Learning: Methods and Applications
Deng, L. (2014) · now publishers, Inc. eBooks · doi:10.1561/9781601988157 · Cited by 1,785 · see citing works
Improving deep neural networks for LVCSR using rectified linear units and dropout
Dahl, G. E., Sainath, T. N., & Hinton, G. E. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639346 · Cited by 1,310 · see citing works
“the only use of these units in speech to date was in [10] where Gaussian-ReLU RBMs were used to learn features from raw, unlabeled acoustic data to later use in phone recognition experiments on TIMIT.”
MethodSelf-citationA review of unsupervised feature learning and deep learning for time-series modeling
Längkvist, M., Karlsson, L., & Loutfi, A. (2014) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.01.008 · Cited by 1,234 · see citing works
“Only a few works have attempted this with speech recognition (Jaitly and Hinton, 2011; Smith and Lewicki, 2005) and physiological data (Wulsin et al.”
BackgroundLei, Y., Jia, F., Lin, J., Xing, S., & Ding, S. X. (2016) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2016.2519325 · Cited by 1,191 · see citing works
“Traditionally, the learned features f i of x is obtained by combining these local features f i j in an aggregate way [38].”
BackgroundSpeech Recognition Using Deep Neural Networks: A Systematic Review
Nassif, A. B., Shahin, I. M. A., Attili, I. B., Azzeh, M., & Shaalan, K. F. (2019) · IEEE Access · doi:10.1109/access.2019.2896880 · Cited by 1,159 · see citing works
AST: Audio Spectrogram Transformer
Gong, Y., Chung, Y.-A., & Glass, J. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-698 · Cited by 1,159 · see citing works
“E[1] E[2] E[3] E[4] E[5] E[6] E[7] E[8] P[1] P[2] P[3] P[4] P[5] P[6] P[7] P[8] Linear Output”
BackgroundAdieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network
Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472669 · Cited by 904 · see citing works
“In one of the first studies that suggested learning better features for automatic speech recognition (ASR) that used directly the speech waveform was Jaitly and Hinton [6].”
Deep Learning for Audio Signal Processing
Purwins, H., Li, B., Virtanen, T. I., Schlüter, J., Chang, S.-Y., & Sainath, T. N. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2908700 · Cited by 881 · see citing works
“[18] and [19] use a full-resolution magnitude spectrum, [20]–[23] directly use a raw waveform representation of the audio signals as inputs and learn datadriven filters jointly with the rest of the network for the target tasks.”
MethodA tutorial survey of architectures, algorithms, and applications for deep learning
Deng, L. (2014) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2013.9 · Cited by 737 · see citing works
“The work also showed the importance of raw speech features of spectrogram --- back from the longpopular MFCC features, but not yet reaching the raw speechwaveform level (Sheikhzadeh and Deng, 1994; Jaitly and Hinton, 2011).”
BackgroundLearning the speech front-end with raw waveform CLDNNs
Sainath, T. N., Weiss, R. J., Senior, A., Wilson, K., & Vinyals, O. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-1 · Cited by 447 · see citing works
“Past work has achieved phase invariance using convolutional layers which pool in time [4, 5, 7] or DNN layers with large, potentially overcomplete, hidden units [6], which can capture the same filter shape at a variety of phases.”
BackgroundMethodInfluentialA comprehensive survey and analysis of generative models in machine learning
Harshvardhan, G. M., Gourisaria, M. K., Pandey, M., & Rautaray, S. S. (2020) · Computer Science Review · doi:10.1016/j.cosrev.2020.100285 · Cited by 437 · see citing works
End-to-end learning for music audio
Dieleman, S., & Schrauwen, B. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854950 · Cited by 377 · see citing works
“Although there has been some work about learning features directly from speech signal fragments [9, 10, 11], to our knowledge feature learning has not been applied directly to raw music audio signals in literature.”
BackgroundDeep learning-based clustering approaches for bioinformatics
Karim, M. R., Beyan, O., Zappa, A., Costa, I. G., Rebholz‐Schuhmann, D., Cochez, M., & Decker, S. (2019) · Briefings in Bioinformatics · doi:10.1093/bib/bbz170 · Cited by 304 · see citing works
“NMMC and UMMC) DBN is employed as the feature extractor in which Restricted Boltzmann Machines (RBM) [60] formed the basic building block.”
MethodAnalysis of CNN-based speech recognition system using raw speech as input
Palaz, D., Magimai-Doss, M., & Collobert, R. (2015) · Interspeech 2015 · doi:10.21437/interspeech.2015-3 · Cited by 288 · see citing works
“Features learning from the raw speech signal using neural networks-based systems has also been investigated in [12].”
BackgroundSpeech Emotion Classification Using Attention-Based LSTM
Xie, Y., Liang, R., Liang, Z., Huang, C.-W., Zou, C., & Schuller, B. (2019) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2019.2925934 · Cited by 268 · see citing works
“(2) Hinton [45] believed that deep learning has the ability to automatically learn feature changes, and can learn deep features related to tasks from the underlying speech features.”
BackgroundFundamentals of speech recognition
Li, J., Deng, L., Haeb‐Umbach, R., & Gong, Y. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-802398-3.00002-7 · Cited by 269 · see citing works
Intelligent Fault Diagnosis Method Based on Full 1-D Convolutional Generative Adversarial Network
Guo, Q., Li, Y., Song, Y., Wang, D., & Chen, W. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2934901 · Cited by 262 · see citing works
“Deep learning, which implies a breakthrough in artificial intelligence, exhibits excellent performance in speech recognition [3], [4], image restoration [5], [6], natural language processing [7], multimodal image-text [8], and games [9].”
BackgroundSleep Stage Classification Using Unsupervised Feature Learning
Längkvist, M., Karlsson, L., & Loutfi, A. (2012) · Advances in Artificial Neural Systems · doi:10.1155/2012/107046 · Cited by 248 · see citing works
“This has proven to give good results in other areas such as vision tasks [10], object recognition [16], motion capture data [17], speech recognition [18], and bacteria identification [19].”
BackgroundLiu, H., Zhou, J., Xu, Y., Zheng, Y., Peng, X., & Jiang, W. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.07.034 · Cited by 244 · see citing works
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
“Feature learning directly from the raw waveform was proposed in 2011 [133].”
BackgroundSpeech acoustic modeling from raw multichannel waveforms
Hoshen, Y., Weiss, R. J., & Wilson, K. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178847 · Cited by 232 · see citing works
“This is in line with previously reported results [2, 3, 5].”
MethodResultMultichannel Signal Processing With Deep Neural Networks for Automatic Speech Recognition
Sainath, T. N., Weiss, R. J., Wilson, K., Li, B., Narayanan, A., Variani, E., Bacchiani, M., Izhak Shafran, J., Senior, A., Chin, K. K., et al. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2672401 · Cited by 231 · see citing works
“The mic array is placed at the center of the room and closer to the wall, respectively, to capture reasonably different reverberation characteristics.”
BackgroundUnsupervised speech representation learning using WaveNet autoencoders
Chorowski, J., Weiss, R. J., Bengio, S., & van den Oord, A. (2019) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2938863 · Cited by 325 · see citing works
“to learn auditory-like frequency decompositions on music [23] and speech [24]–[27] in their lower layers.”
BackgroundConvolutional Neural Networks-based continuous speech recognition using raw speech signal
Palaz, D., Magimai.-Doss, M., & Collobert, R. (2015) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2015.7178781 · Cited by 191 · see citing works
“Features learning from raw speech using neural networks-based systems has been investigated in [17].”
BackgroundRecent progresses in deep learning based acoustic models
Yu, D., & Li, J. (2017) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2017.7510508 · Cited by 170 · see citing works
“Inspired by the end-to-end processing in the machine learning community, there are always efforts [51]–[54] trying to replace the Mel-filter-bank extraction by directly learning filters using a network to process the raw speech waveforms and training it jointly with the recognizer network.”
MethodXing, S., Lei, Y., Wang, S., & Jia, F. (2020) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2020.2972461 · Cited by 172 · see citing works
“RBM is a kind of stochastic neural network with two layers [25].”
BackgroundPalaz, D., Magimai-Doss, M., & Collobert, R. (2019) · Speech Communication · doi:10.1016/j.specom.2019.01.004 · Cited by 168 · see citing works
“Feature learning from raw speech using neural networks-based systems has been investigated in [10].”
Background3D CNN-Based Speech Emotion Recognition Using K-Means Clustering and Spectrograms
Hajarolasvadi, N., & Demirel, H. (2019) · Entropy · doi:10.3390/e21050479 · Cited by 157 · see citing works
“[15] proposed a limited-weight-sharing scheme that models the speech features for speech recognition systems while [11] proposed a new method for modeling speech signals using Restricted Boltzmann Machine.”
MethodA Survey of Automatic Text Summarization: Progress, Process and Challenges
Mridha, M. F., Lima, A. A., Nur, K. M., Das, S. C., Hasan, M., & Kabir, M. M. (2021) · IEEE Access · doi:10.1109/access.2021.3129786 · Cited by 162 · see citing works
“[293], [294], [295], [296], [297], [298] are the research that focused on RBM method for text summarization.”
BackgroundDeep convex net: a scalable architecture for speech pattern classification
Deng, L., & Yu, D. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-607 · Cited by 149 · see citing works
“And many years later the use of more powerful Restricted Boltzmann Machine (RBM) overcomes some difficulty encountered earlier [10].”
MethodDeep learning methods for underwater target feature extraction and recognition
Hu, G., Wang, K., Peng, Y., Qiu, M., Shi, J., & Liu, L. (2018) · Computational Intelligence and Neuroscience · doi:10.1155/2018/1214301 · Cited by 151 · see citing works
“In 2012, four major international scientific research institutions summed up the progress made by deep learning in sound recognition task [15] and pointed out in the paper that the experiments show the effect of deep neural network is better than the one of traditional Gaussian mixed model.”
BackgroundTowards deeper understanding: Deep convex networks for semantic utterance classification
Tür, G., Deng, L., Hakkani-Tür, D. Z., & He, X. (2012) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2012.6289054 · Cited by 123 · see citing works
“Following the introduction of this RBM learning and layer-by-layer construction of deep architectures, DBNs have been successfully used for numerous tasks in speech and language processing, including acoustic modeling [8, 9, 10] and intent determination [6].”
MethodJamalipour, A., & Murali, S. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3126811 · Cited by 132 · see citing works
Aldwairi, T., Perera, D., & Novotny, M. A. (2018) · Computer Networks · doi:10.1016/j.comnet.2018.07.025 · Cited by 124 · see citing works
An Ensemble Stacked Convolutional Neural Network Model for Environmental Event Sound Recognition
Li, S., Yao, Y. F., Hu, J., Liu, G., Yao, X., & Hu, J. (2018) · Applied Sciences · doi:10.3390/app8071152 · Cited by 118 · see citing works
EOG-based drowsiness detection using convolutional neural networks
Zhu, X., Zheng, W., Lu, B., Chen, X., Chen, S., & Wang, C. (2014) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2014.6889642 · Cited by 117 · see citing works
“Such unsupervised training method has been proved to reach awesome results in areas such as object recognition [15] and speech recognition [16].”
MethodKoçak, B., Durmaz, E. Ş., Kuş, E. A., & Ulusan, M. B. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20443 · Cited by 114 · see citing works
A first look into a Convolutional Neural Network for speech emotion detection
Bertero, D., & Fung, P. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953131 · Cited by 117 · see citing works
“Deep learning from raw audio is now replacing traditional feature-based learning in all speech-related tasks, with Automatic Speech Recognition the most prominent field [6, 7, 8].”
BackgroundThree Classes of Deep Learning Architectures and Their Applications: A Tutorial Survey
Deng, L. (2012) · APSIPA Transactions on Signal and Information Processing · Cited by 110 · see citing works
LEAF: A Learnable Frontend for Audio ClassificationPreprint
Zeghidour, N., Teboul, O., Quitry, F. D. C., & Tagliasacchi, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.08596 · Cited by 116 · see citing works
“The first notable contributions in this field emerged for ASR, with Jaitly & Hinton (2011) pretraining Restricted Boltzmann Machines from the waveform, and Palaz et al. (2013) training a hybrid DNN-HMM model, replacing mel-filterbanks by several layers of convolution.”
MethodEmotion Recognition from Variable-Length Speech Segments Using Deep Learning on Spectrograms
Ma, X., Wu, Z., Jia, J., Xu, M., Meng, H. M. L., & Cai, L. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-2228 · Cited by 102 · see citing works
“In speech recognition, one of the first studies that suggested learning better features for automatic speech recognition (ASR) that used directly the speech waveform was Jaitly and Hinton [10].”
Barushka, A., & Hájek, P. (2018) · Applied Intelligence · doi:10.1007/s10489-018-1161-y · Cited by 100 · see citing works
“The combination of dropout regularization and ReL units has shown promising synergistic effects [38].”
BackgroundDeep learning: from speech recognition to language and multimodal processing
Deng, L. (2016) · APSIPA Transactions on Signal and Information Processing · doi:10.1017/atsip.2015.22 · Cited by 95 · see citing works
“As an initial attempt toward this goal, the study carried out by Jaitly and Hinton [95] made use of speech sound waves as the raw input feature to a deep learning system.”
Learning Invariant Representations of Molecules for Atomization Energy Prediction
Montavon, G., Hansen, K., Fazli, S., Biegler, F., Ziehe, A., Tkatchenko, A., von Lilienfeld, O. A., & Müller, K.-R. (2014) · Cited by 99 · see citing works
Comparison and Analysis of SampleCNN Architectures for Audio Classification
Kim, T., Lee, J., & Nam, J. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2909479 · Cited by 89 · see citing works
“waveforms to uncover hearing mechanism or improve audio classification date back to more than a decade [18]–[21].”
BackgroundGas Recognition under Sensor Drift by Using Deep Learning
Liu, Q., Hu, X., Ye, M., Cheng, X., & Li, F. (2015) · International Journal of Intelligent Systems · doi:10.1002/int.21731 · Cited by 86 · see citing works
Unsupervised Fault Diagnosis of a Gear Transmission Chain Using a Deep Belief Network
He, J., Yang, S., & Gan, C. (2017) · Sensors · doi:10.3390/s17071564 · Cited by 73 · see citing works
“Deep learning [26–29] has recently proven its capability for unsupervised feature learning in various fields such as speech recognition [30,31], motion capture [32], visual recognition [33,34], and physiology [35,36].”
BackgroundUsmani, U. A., Happonen, A., & Watada, J. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-10464-0_11 · Cited by 69 · see citing works
Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future TrendsPreprint
Latif, S., Rana, R. K., Khalifa, S., Jurdak, R., Qadir, J., & Schuller, B. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.00378 · Cited by 87 · see citing works
“For instance, Jaitly and Hinton used RBMs for phoneme recognition in [52], and showed that RBMs can learn more discriminative features that achieved better performance compared to MFCCs. Interestingly, RBMs can also learn filterbanks from raw speech.”
BackgroundCombining bag-of-words and sentiment features of annual reports to predict abnormal stock returns
Hájek, P. (2017) · Neural Computing and Applications · doi:10.1007/s00521-017-3194-2 · Cited by 65 · see citing works
“The synergistic effects of combining rectified linear units with dropout regularization have been demonstrated by [69].”
BackgroundMachine Learning for Speaker Recognition
Mak, M., & Chien, J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781108552332 · Cited by 55 · see citing works
Deep Learning in Paralinguistic Recognition Tasks: Are Hand-crafted Features Still Relevant?
Wagner, J., Schiller, D., Seiderer, A., & André, E. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-1238 · Cited by 54 · see citing works
“The finding that DNNs have the potential to learn robust feature representations [13, 14] has led to breakthroughs in various audio classification tasks such as speech recognition [15], sound event classification [16] and automatic music transcriptions [17].”
BackgroundWaldmann, I. P. (2016) · The Astrophysical Journal · doi:10.3847/0004-637x/820/2/107 · Cited by 54 · see citing works
“…(e.g., Krizhevsky et al. 2012; Liu et al. 2014; Wang et al. 2016; Shen et al. 2015), speech and music recognition (e.g., Hung et al. 2005; Jaitly & Hinton 2011; Zhang & Wu 2013; Pradeep & Kumaraswamy 2014), biology (e.g., Head-Gordon & Stillinger 1993; Plebe 2007; Wu & McLarty 2012;…”
BackgroundBrain–computer interfacing: more than the sum of its parts
Scherer, R., Faller, J., Balderas, D., Friedrich, E. V. C., Pröll, M., Allison, B. Z., & Muller-Putz, G. R. (2012) · Soft Computing · doi:10.1007/s00500-012-0895-4 · Cited by 52 · see citing works
“, speech representation (Navdeep and Hinton 2011), or features extracted from EEG signals (Balderas et al.”
BackgroundVásquez-Correa, J. C., Klumpp, P., Orozco‐Arroyave, J. R., & Nöth, E. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1405 · Cited by 50 · see citing works
“Recently, deep learning methods have also been considered to extract features from speech for specific tasks [3, 4].”
BackgroundKoçak, B., Durmaz, E. Ş., Kaya, O. K., & Kılıçkesmez, Ö. (2019) · Acta Radiologica · doi:10.1177/0284185119881742 · Cited by 50 · see citing works
Towards Deep Developmental Learning
Sigaud, O., & Droniou, A. (2015) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tamd.2015.2496248 · Cited by 47 · see citing works
Attention-Based Dense LSTM for Speech Emotion Recognition
Xie, Y., Liang, R., LIANG, Z., & Zhao, L. (2019) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2019edl8019 · Cited by 48 · see citing works
“[14] N. Jaitly and G. Hinton, “Learning a better representation of speech soundwaves using restricted boltzmann machines,” in IEEE International Conference on Acoustics, Speech and Signal Processing, pp.5884–5887, 2011.”
BackgroundKim, J., Urbano, J., Liem, C. C. S., & Hanjalić, A. (2019) · Neural Computing and Applications · doi:10.1007/s00521-019-04076-1 · Cited by 48 · see citing works
“Although variants of CNN using 1-dimensional filters also were suggested by [12, 57–59] to learn features directly from a raw audio signal in an end-to-end manner, not many works managed to use them on music classification tasks successfully [60].”
MethodAn Overview of Deep-Structured Learning for Information Processing
Deng, L. (2011) · Cited by 47 · see citing works
Novel Unsupervised Auditory Filterbank Learning Using Convolutional RBM for Speech Recognition
Sailor, H. B., & Patil, H. A. (2016) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2016.2607341 · Cited by 47 · see citing works
“This may be due to the fact that RBM was trained on randomly selected smaller windows of the speech signal and hence, they were in any random temporal phase [19].”
BackgroundMethodLooking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation
Lee, J., Chung, S.-W., Kim, S., Kang, H.-G., & Sohn, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00139 · Cited by 47 · see citing works
“However, reliably separating a target speech signal for humancomputer interaction (HCI) systems such as speech recognition [3, 4, 5], speaker recognition [6, 7, 8], and emotion recognition [9, 10] is still a challenging task because it is an ill-posed problem.”
BackgroundA Deep Neural Network for Acoustic-Articulatory Speech Inversion
Uría, B., Renals, S. J., & Richmond, K. (2011) · Cited by 45 · see citing works
Gao, S., Wang, X., Miao, X., Su, C., & Li, Y. (2019) · Journal of Signal Processing Systems · doi:10.1007/s11265-019-01463-8 · Cited by 44 · see citing works
“As a major breakthrough in the domain of data science, deep learning has shown excellent performance in speech recognition [6, 7], NLP [8], multi-mode data processing [9, 10], image restoration [11, 12] and other applications.”
BackgroundUnsupervised feature learning for electronic nose data applied to Bacteria Identification in Blood.
Längkvist, M., & Loutfi, A. (2011) · Cited by 43 · see citing works
Deep-Sparse-Representation-Based Features for Speech Recognition
Sharma, P., Abrol, V., & Sao, A. K. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2748240 · Cited by 41 · see citing works
“[41] H. B. Sailor and H. A. Patil, “Novel unsupervised auditory filterbank learning using convolutional RBM for speech recognition,” IEEE/ACM Trans.”
MethodConvolutional Neural Networks for Raw Speech Recognition
Passricha, V., & Aggarwal, R. K. (2018) · IntechOpen eBooks · doi:10.5772/intechopen.80026 · Cited by 39 · see citing works
“Some efforts were also made to model the raw speech signal with little or no preprocessing [34].”
BackgroundSpeech Quality Assessment Over Lossy Transmission Channels Using Deep Belief Networks
Affonso, E. T., Rosa, R. L., & Rodríguez, D. Z. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2773536 · Cited by 38 · see citing works
“In [42], the RBM is used for a better representation of speech sound waves, their results reported that the phoneme recognition performance with their proposed RBM model overcomes MFCC-based solutions.”
MethodCLUSTERING-BASED FEATURE LEARNING ON VARIABLE STARS
Mackenzie, C., Pichara, K., & Protopapas, P. (2016) · The Astrophysical Journal · doi:10.3847/0004-637x/820/2/138 · Cited by 38 · see citing works
“Raw time series data has been used with success in a limited number of problems, most notably by Jaitly & Hinton (2011).”
MethodEnd-to-end Phoneme Sequence Recognition using Convolutional Neural NetworksPreprint
Palaz, D., Collobert, R., & Magimai.-Doss, M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.2137 · Cited by 41 · see citing works
“The work presented in [19] successfully investigates features learning from raw speech for phoneme recognition.”
BackgroundImproving robustness of deep neural networks via spectral masking for automatic speech recognition
Li, B., & Sim, K. C. (2013) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2013.6707743 · Cited by 37 · see citing works
“The log Mel filterbank (FBank) features [15] or the waveform signals [16] have been shown to outperform the human engineered Mel frequency cepstral coefficient (MFCC) features as inputs for DNNs.”
MethodAST: Audio Spectrogram TransformerPreprint
Gong, Y., Chung, Y.-A., & Glass, J. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01778 · Cited by 145 · see citing works
“E[1] E[2] E[3] E[4] E[5] E[6] E[7] E[8] P[1] P[2] P[3] P[4] P[5] P[6] P[7] P[8] Linear Output”
BackgroundMultiscale Audio Spectrogram Transformer for Efficient Audio Classification
Zhu, W., & Omar, M. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096513 · Cited by 34 · see citing works
Li, B., & Sim, K. C. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639102 · Cited by 33 · see citing works
“With a powerful model, less pre-processed input representations, such as FBank features [1] and waveforms [16], could yield better”
MethodMRI-based radiomics analysis for differentiating phyllodes tumors of the breast from fibroadenomas
Tsuchiya, M., Masui, T., Terauchi, K., Yamada, T., Katyayama, M., Ichikawa, S., Noda, Y., & Goshima, S. (2022) · European Radiology · doi:10.1007/s00330-021-08510-8 · Cited by 33 · see citing works
“Fourth, MR image data were collected from three different scanners with different magnetic field strengths, which can affect the extracted features [26, 27].”
MethodDeep Neural Network Acoustic Models for ASR
Mohamed, A. (2014) · TSpace (University of Toronto) · Cited by 32 · see citing works
Filterbank learning using Convolutional Restricted Boltzmann Machine for speech recognition
Sailor, H. B., & Patil, H. A. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472808 · Cited by 31 · see citing works
“Recently, Restricted Boltzmann Machine (RBM) with rectified linear units (ReLUs) was used for representation learning in ASR task [9].”
BackgroundMethodInfluentialVariani, E., Sainath, T. N., Izhak Shafran, J., & Bacchiani, M. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-1459 · Cited by 31 · see citing works
“The emergence of deep neural networks for acoustic modeling [7] has recently sparked a resurgence in data-driven techniques which jointly estimate filter bank and AM parameters [8, 9, 10, 11, 12, 13, 14, 15].”
MethodQian, Y., Ubale, R., Ramanaryanan, V., Lange, P., Suendermann-Oeft, D., Evanini, K., & Tsuprun, E. (2017) · 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) · doi:10.1109/asru.2017.8268987 · Cited by 62 · see citing works
“, using filter-bank features instead of MFCC [17] or directly using speech waveform [18].”
MethodAcoustic Analysis for Automatic Speech Recognition
O’Shaughnessy, D. D. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2013.2251592 · Cited by 28 · see citing works
“den units and one layer of stochastic observable units [46].”
BackgroundDeep Neural Network Frontend for Continuous EMG-Based Speech Recognition
Wand, M., & Schmidhuber, J. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-340 · Cited by 28 · see citing works
“Current research targets two main directions: First, the feature preprocessing frontend is gradually replaced by networks trained on raw waveforms [24], and second, the HMM backend is replaced by a recurrent neural network architecture [25] usually implemented as LSTM ( Long Short-Term Memory…”
MethodEvaluating raw waveforms with deep learning frameworks for speech emotion recognition
Kilimci, Z. H., Bayraktar, Ü., & Küçükmanіşa, A. (2025) · Multimedia Tools and Applications · doi:10.1007/s11042-025-20930-y · Cited by 24 · see citing works
“…deep neural network-based acoustic method which fed with raw multichannel waveforms as input by inspiring studies Dai et al. [55], Hoshen et al. [56 , 57], Tüske et al. [58], Jaitly and Hinton [59] with - out performing feature extraction stage by constructing high-level representative features.”
Interval Type-2 Fuzzy Restricted Boltzmann Machine
Hosseini-Pozveh, M. S., Safayani, M., & Mirzaei, A. (2020) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2020.2969120 · Cited by 24 · see citing works
“RestrictedBoltzmannmachine(RBM)[1]is aspecialtypeofneuralnetworkwithmanyapplicationslike imageprocessing[2],[3],topicmodeling[4],classification,[5] andspeechrecognition[6],[7].”
End-to-End Speech Recognition in Agglutinative Languages
Mamyrbayev, O. J., Alimhan, K., Жумажанов, Б., Turdalykyzy, T., & Gusmanova, F. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-42058-1_33 · Cited by 24 · see citing works
Kim, S. K., & Kang, H.-B. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.09.081 · Cited by 25 · see citing works
Seven neurons memorizing sequences of alphabetical images via spike-timing dependent plasticity
Osogami, T., & Otsuka, M. (2015) · Scientific Reports · doi:10.1038/srep14149 · Cited by 22 · see citing works
“Similar to standard recurrent neural networks 40,49,50 , the DyBM can be unfolded though time 51,52 .”
Speech classification using SIFT features on spectrogram images
Nguyen, Q. T., & Bui, T. D. (2016) · Vietnam Journal of Computer Science · doi:10.1007/s40595-016-0071-3 · Cited by 22 · see citing works
Spam Filtering Using Regularized Neural Networks with Rectified Linear Units
Barushka, A., & Hájek, P. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49130-1_6 · Cited by 22 · see citing works
Sertolli, B., Ren, Z. X., Schuller, B. W., & Cummins, N. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101204 · Cited by 22 · see citing works
Cognitive Speech Coding: Examining the Impact of Cognitive Speech Processing on Speech Compression
Cerňak, M., Asaei, A., & Hyafil, A. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2761895 · Cited by 21 · see citing works
“At present, speech recognition and synthesis are highly dependent on machine learning tools and big data, and speech coding can benefit from it, for example by learning of better speech representation [76].”
Neural Networks and Deep Learning
Theodoridis, S. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-818803-3.00030-1 · Cited by 22 · see citing works
A survey on preprocessing and classification techniques for acoustic scene
Singh, V. K., Sharma, K., & Sur, S. N. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120520 · Cited by 21 · see citing works
Multi-level region-of-interest CNNs for end to end speech recognition
Singhal, S., Passricha, V., Sharma, P., & Aggarwal, R. K. (2018) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-018-1146-z · Cited by 21 · see citing works
“Earlier, Jaitly and Hinton (2011) used restricted Boltzmann machines, Tüske et al. (2014) used DNN and Sainath et al. (2015b) used the combination of convolutional, long short-term memory, and fully connected DNN to process the raw waveforms.”
MethodQuantum Inspired Training for Boltzmann MachinesPreprint
Wiebe, N., Kapoor, A., Granade, C., & Svore, K. M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.02642 · Cited by 21 · see citing works
“Due to the emergence of CD, Boltzmann machines, specifically restricted Boltzmann machines (RBMs) and their deep layered counterparts (DBMs), have become standard tools for solving problems in vision and speech recognition [3, 4].”
Mamyrbayev, O. J., Alimhan, K., Оралбекова, Д., Bekarystankyzy, A., & Жумажанов, Б. (2022) · Eastern-European Journal of Enterprise Technologies · doi:10.15587/1729-4061.2022.252801 · Cited by 20 · see citing works
“Some studies used deep neural networks [2] to build acoustic models, and others used RNNs [3] to build language models with excellent results.”
BackgroundUsing an autoencoder with deformable templates to discover features for automated speech recognition
Jaitly, N., & Hinton, G. E. (2013) · Interspeech (USB) · doi:10.21437/interspeech.2013-432 · Cited by 19 · see citing works
“The at values were outputs of rectified linear units (ReLU) [3, 4].”
BackgroundSelf-citationHindi speech recognition using time delay neural network acoustic modeling with i-vector adaptation
Kumar, A., & Aggarwal, R. K. (2020) · International Journal of Speech Technology · doi:10.1007/s10772-020-09757-0 · Cited by 19 · see citing works
“DNN/HMM hybrid models were also used to train the acoustic model in various work to improve the ASR system’s performance (Jaitly and Hinton 2011; Dahl et al. 2011a; Seide et al. 2011).”
Method
- Authors
- Jaitly, N., & Hinton, G.
- Affiliations
- University of Toronto (CA)
- Venue
- 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
- Publisher
- IEEE
- Published
- 2011-05
- Pages
- 5884-5887
- ISSN
- 1520-6149 (ISSN-L)
- OpenAlex
- W2132037657
- Cited by
- 231
- References
- 19
- Citations by year
- 2026: 3 · 2025: 8 · 2024: 3 · 2023: 11 · 2022: 8 · 2021: 24 · 2020: 34 · 2019: 29 · 2018: 21 · 2017: 16 · 2016: 22 · 2015: 16 · 2014: 17 · 2013: 9 · 2012: 5
- FWCI
- 9.01
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Acoustics Speech and Signal Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Keywords
- restricted Boltzmann machine, phoneme recognition, hidden variables, speech feature extraction, speech synthesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
