Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
Open accessThe DOI matches its OpenAlex record; 56 of the 56 references it cites were found in a registry or an index.
Abstract
Research on speech processing has traditionally considered the task of designing hand-engineered acoustic features (feature engineering) as a separate distinct problem from the task of designing efficient machine learning (ML) models to make prediction and classification decisions. There are two main drawbacks to this approach: firstly, the feature engineering being manual is cumbersome and requires human knowledge; and secondly, the designed features might not be best for the objective at hand. This has motivated the adoption of a recent trend in speech community towards utilisation of representation learning techniques, which can learn an intermediate representation of the input signal automatically that better suits the task at hand and hence lead to improved performance. The significance of representation learning has increased with advances in deep learning (DL), where the representations are more useful and less dependent on human knowledge, making it very conducive for tasks like classification, prediction, etc. The main contribution of this paper is to present an up-to-date and comprehensive survey on different techniques of speech representation learning by bringing together the scattered research across three distinct research areas including Automatic Speech Recognition (ASR), Speaker Recognition (SR), and Speaker Emotion Recognition (SER). Recent reviews in speech have been conducted for ASR, SR, and SER, however, none of these has focused on the representation learning from speech -- a gap that our survey aims to bridge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
A Global Geometric Framework for Nonlinear Dimensionality Reduction
Tenenbaum, J. B., Silva, V. D., & Langford, J. C. (2000) · Science · doi:10.1126/science.290.5500.2319 · Cited by 13,585
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · cited 1×
“Bidirectional RNNs [113] were proposed to model future context by passing the input sequence through two separate recurrent hidden layers.”
MethodQuantum supremacy using a programmable superconducting processor
Arute, F., Arya, K., Babbush, R., Bacon, D., Bardin, J. C., Barends, R., Biswas, R., Boixo, S., Brandao, F. G. S. L., Buell, D. A., et al. (2019) · Nature · doi:10.1038/s41586-019-1666-5 · Cited by 7,720
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072
Davis, S., & Mermelstein, P. (1980) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1980.1163420 · Cited by 5,465
DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · cited 1×
“Some popular adversarial attacks include the fast gradient sign method (FGSM) [352], Jacobian-based saliency map attack (JSMA) 1 https://www.ethnologue.com/statistics [353], and DeepFool [354]; they compute the perturbation noise based on the gradient of targeted output.”
MethodA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733 · cited 1×
“Methods such as deep Q-learning have significantly advanced the field [224].”
A Brief Survey of Deep Reinforcement Learning
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · arXiv · doi:10.48550/arxiv.1708.05866 · Cited by 4,733
In-Datacenter Performance Analysis of a Tensor Processing Unit
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., Bates, S., Bhatia, S., Boden, N., Borchers, A., et al. (2017) · Proceedings of the 44th Annual International Symposium on Computer Architecture · doi:10.1145/3079856.3080246 · Cited by 4,537 · cited 1×
“In contrast to CPUs and GPUs, advanced Tensor Processing Units (TPUs) developed by Google offer 15-30 × higher processing speeds and 30-80 × higher performance-per-watt [373].”
MethodThe Limitations of Deep Learning in Adversarial Settings
Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z. B., & Swami, A. (2016) · 2016 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2016.36 · Cited by 3,993
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
Reinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · cited 1×
“RL has been repeatedly successful in solving various problems [223].”
BackgroundNeural Discrete Representation Learning
Oord, A. V. D., Vinyals, O., & Kavukcuoglu, K. (2017) · arXiv · doi:10.48550/arxiv.1711.00937 · Cited by 2,729
Phoneme recognition using time-delay neural networks
Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., & Lang, K. J. (1989) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/29.21701 · Cited by 2,643
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642 · cited 1×
“dioAnalysis [364] A Python library facilities a wide range of feature extraction and also classification of audio signals, supervised and unsupervised segmentation and content visualisation. openSMILE [365] It enables to extract a large number of audio feature in real time. It is written in C++. Speech Recognition Tool…”
BackgroundSelf-citationDeep learning applications and challenges in big data analytics
Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015) · Journal of Big Data · doi:10.1186/s40537-014-0007-7 · Cited by 2,594
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484
A database of German emotional speech
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-446 · Cited by 2,365 · cited 1×
“Similarly, EMO-DB [58], FAU-AIBO [59], RECOLA [60], and GEMEP [61] are some popular classical datasets.”
BackgroundPrivacy-preserving deep learning
Shokri, R., & Shmatikov, V. (2015) · Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/2810103.2813687 · Cited by 2,356 · cited 1×
“To preserve users’ privacy, federated learning [385] is another alternative setting where the training of a shared global model is performed using multiple participating computing devices.”
MethodAdversarial machine learning at scale
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01236 · Cited by 2,325
VoxCeleb2: Deep Speaker Recognition
Chung, J. S., Nagrani, A., & Zisserman, A. (2018) · Interspeech 2018 · doi:10.21437/interspeech.2018-1929 · Cited by 2,256
Arjovsky, M., Chintala, S., & Bottou, L. (2017) · arXiv · doi:10.48550/arxiv.1701.07875 · Cited by 2,225
SWITCHBOARD: telephone speech corpus for research and development
Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992) · [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1992.225858 · Cited by 2,134 · cited 1×
“Some popular conventional corpora used for ASR and SR includes TIMIT [54], Switchboard [55], WSJ [56], AMI [57].”
MethodSelf-Supervised Visual Feature Learning With Deep Neural Networks: A Survey
Jing, L., & Tian, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2992393 · Cited by 2,036 · cited 1×
“Most importantly, automatically learnt feature representation is in most cases more flexible and powerful and can be applied to any data science problem in the fields of vision processing [75], text processing [76], and speech processing [77].”
BackgroundThreat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey
Akhtar, N., & Mian, A. (2018) · IEEE Access · doi:10.1109/access.2018.2807385 · Cited by 2,021
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · cited 1×
“In SER, feature engineering is more dominant and a minimalistic sets of features like GeMAPs and eGeMAPs [50] are also proposed based on affective physiological changes in voice production and their theoretical significance [50].”
BackgroundSelf-citationConvolutional, Long Short-Term Memory, fully connected Deep Neural Networks
Sainath, T. N., Vinyals, O., Senior, A., & Sak, H. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178838 · Cited by 1,678
(2018) · doi:10.21437/interspeech.2018-1456 · Cited by 1,366
Efficient Learning of Sparse Representations with an Energy-Based Model
Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0147 · Cited by 1,106
The INTERSPEECH 2009 emotion challenge
Schuller, B., Steidl, S., & Batliner, A. (2009) · Interspeech 2009 · doi:10.21437/interspeech.2009-103 · Cited by 938 · cited 1×
“Similarly, EMO-DB [58], FAU-AIBO [59], RECOLA [60], and GEMEP [61] are some popular classical datasets.”
BackgroundSelf-citationAdieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network
Trigeorgis, G., Ringeval, F., Brueckner, R., Marchi, E., Nicolaou, M. A., Schuller, B., & Zafeiriou, S. (2016) · 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2016.7472669 · Cited by 904
Self-citationSpeaker Recognition from Raw Waveform with SincNet
Ravanelli, M., & Bengio, Y. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639585 · Cited by 896
Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security
Chesney, R., & Citron, D. K. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3213954 · Cited by 880 · cited 1×
“Other recent works [375], [376] also explored audio synthesis using GANs; however, such work is at the initial stage and will likely open new prospects of future research as it transpired with the use of GANs in the domain of vision (e. g., with DeepFakes [377]).”
BackgroundIntroducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771
MelGAN: generative adversarial networks for conditional waveform synthesis
Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W. Z., Sotelo, J., de Brebisson, A., Bengio, Y., & Courville, A. (2019) · arXiv · doi:10.48550/arxiv.1910.06711 · Cited by 722 · cited 1×
“Other recent works [375], [376] also explored audio synthesis using GANs; however, such work is at the initial stage and will likely open new prospects of future research as it transpired with the use of GANs in the domain of vision (e. g., with DeepFakes [377]).”
BackgroundMcKeown, G., Valstar, M., Cowie, R., Pantic, M., & Schroder, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.20 · Cited by 665
Mao, Q., Dong, M., Huang, Z., & Zhan, Y. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2360798 · Cited by 568
OpenEAR — Introducing the munich open-source emotion and affect recognition toolkit
Eyben, F., Wollmer, M., & Schuller, B. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349350 · Cited by 418 · cited 1×
“…Python English Julius [368] C, Python Japanese ESPnet [369] Python English, Japanese, Mandarin HTK [370] C, Python None Speaker Identification ALIZE [371] C++ English Speech Emotion Recognition OpenEAR [372] C++ English, German of cores that can perform exceptionally fast matrix multiplications.”
BackgroundSelf-citationMSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception
Busso, C., Parthasarathy, S., Burmania, A., AbdelWahab, M., Sadoughi, N., & Provost, E. M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2515617 · Cited by 391 · cited 1×
“MSP-IMPROV [70] English Audio-Visual 9 hours of audiovisual data of 12 actors This corpus is recorded from dyadic interactions of actors to”
MethodText feature extraction based on deep learning: a review
Liang, H., Sun, X., Sun, Y., & Gao, Y. (2017) · EURASIP Journal on Wireless Communications and Networking · doi:10.1186/s13638-017-0993-1 · Cited by 373 · cited 1×
“Most importantly, automatically learnt feature representation is in most cases more flexible and powerful and can be applied to any data science problem in the fields of vision processing [75], text processing [76], and speech processing [77].”
BackgroundSparse Autoencoder-Based Feature Transfer Learning for Speech Emotion Recognition
Deng, J., Zhang, Z., Marchi, E., & Schuller, B. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.90 · Cited by 366
Self-citationUnsupervised speech representation learning using WaveNet autoencoders
Chorowski, J., Weiss, R. J., Bengio, S., & van den Oord, A. (2019) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2938863 · Cited by 325 · cited 3×
“AEs can learn high-level semantic content (e. g., phoneme identities) that are invariant to confounding low-level details (pitch contour or background noise) in speech [135].”
BackgroundMethodLearning a better representation of speech soundwaves using restricted boltzmann machines
Jaitly, N., & Hinton, G. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5947700 · Cited by 231 · cited 3×
“For instance, Jaitly and Hinton used RBMs for phoneme recognition in [52], and showed that RBMs can learn more discriminative features that achieved better performance compared to MFCCs. Interestingly, RBMs can also learn filterbanks from raw speech.”
BackgroundAn experimental study of speech emotion recognition based on deep convolutional neural networks
Zheng, W. Q., Yu, J. S., & Zou, Y. X. (2015) · 2015 International Conference on Affective Computing and Intelligent Interaction (ACII) · doi:10.1109/acii.2015.7344669 · Cited by 187
Domain Adversarial for Acoustic Emotion Recognition
Abdelwahab, M., & Busso, C. (2018) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2018.2867099 · Cited by 178 · cited 1×
“…[156], VAEs [79], [311] DNNs [315], [316], AEs ( [182]), VAEs ( [46]) SR AL ( [197]), DAE ( [196]), GANs ( [198]) SER DBNs [317], CNNs ( [318]), AL ( [201], [319], [320]), AEs [118], [199], [200], [321], [322] Multi-TaskLearning ASR To learn common representations using multi-objective training.”
Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation
Swietojanski, P., Li, J., & Renals, S. (2016) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2016.2560534 · Cited by 133 · cited 1×
“Another study [180] explored the unsupervised domain adaptation of the acoustic model by learning hidden unit contributions.”
BackgroundUsing regional saliency for speech emotion recognition
Aldeneh, Z., & Provost, E. M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952655 · Cited by 126
Universum Autoencoder-Based Domain Adaptation for Speech Emotion Recognition
Deng, J., Xu, X., Zhang, Z., Fruhholz, S., & Schuller, B. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2672753 · Cited by 125
Self-citationZhang, Y., Liu, Y., Weninger, F., & Schuller, B. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7953106 · Cited by 55 · cited 1×
“DNNs [323], [324], RNNs ( [325]–[327]), AL ( [190]) SR DNNs ( [328], [329]), CNNs ( [330]), RNNs ( [331]) SER DBNs ( [332]), DNNs ( [214], [216], [333], [334]), CNN-LSTM [335], LSTM ( [213], [217], [336], [337]), AL ( [338]), GANs ( [157]) not work well to another corpus having different recording…”
Self-citationUsing denoising autoencoder for emotion recognition
Xia, R., & Liu, Y. (2013) · Interspeech 2013 · doi:10.21437/interspeech.2013-256 · Cited by 28 · cited 2×
“DAEs are exploited in [155], [156] to show the suitability of DAEs for SER.”
Background
Cited by
Self-supervised speech representation learning: A review
Mohamed, A., Lee, H.-Y., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaløe, L., et al. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3207050 · Cited by 372
“One review addresses speech representation learning based on deep learning models [25],”
BackgroundKhalil, M., McGough, A. S., Pourmirza, Z., Pazhoohesh, M., & Walker, S. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105287 · Cited by 299
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295
“The SSL methods can be roughly divided into the following categories: generative-based, contrastive-based, and adversarial-based methods.”
BackgroundReinforcement learning and bandits for speech and language processing: Tutorial, review and outlook
Lin, B. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122254 · Cited by 28
Metadata
- Authors
- Latif, S., Rana, R. K., Khalifa, S., Jurdak, R., Qadir, J., & Schuller, B. W.
- Venue
- arXiv (Cornell University)
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2001.00378
- OpenAlex
- W2998249245
- Cited by
- 66
- References
- 56
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.