Data Augmentation for Deep Neural Network Acoustic Modeling
The DOI matches its Crossref record; 41 of the 44 references it cites were found in a registry or an index.
Abstract
This paper investigates data augmentation for deep neural network acoustic modeling based on label-preserving transformations to deal with data sparsity. Two data augmentation approaches, vocal tract length perturbation (VTLP) and stochastic feature mapping (SFM), are investigated for both deep neural networks (DNNs) and convolutional neural networks (CNNs). The approaches are focused on increasing speaker and speech variations of the limited training data such that the acoustic models trained with the augmented data are more robust to such variations. In addition, a two-stage data augmentation scheme based on a stacked architecture is proposed to combine VTLP and SFM as complementary approaches. Experiments are conducted on Assamese and Haitian Creole, two development languages of the IARPA Babel program, and improved performance on automatic speech recognition (ASR) and keyword search (KWS) is reported.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We also perform leave-one-subject-out (LOSO) cross-validation for reference by leveraging data augmentation [43,44,45,46], which is widely adopted in the deep learning research that requires large dataset for model training.”
From Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning (2019) · cites this paper as [45]
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Vincent Vanhoucke The authors are with the IBM T. J. Watson Research Center, Yorktown Heights, NY 10598 USA (e-mail: cuix@us.ibm.com; vgoel@us.ibm.com; bedk@us.ibm.com).”
MethodGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Vincent Vanhoucke The authors are with the IBM T. J. Watson Research Center, Yorktown Heights, NY 10598 USA (e-mail: cuix@us.ibm.com; vgoel@us.ibm.com; bedk@us.ibm.com).”
MethodHinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Self-citationContext-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011
Best practices for convolutional neural networks applied to visual document analysis
Simard, P. Y., Steinkraus, D., & Platt, J. C. (2005) · Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings. · doi:10.1109/icdar.2003.1227801 · Cited by 2,741 · cited 2×
“Data augmentation has been widely used in neural network based pattern recognition tasks [1][2][3][4], especially in image recognition where transformations such as translation, deformation and reflection [1][4] have led to significant improvements in recognition accuracy.”
BackgroundConvolutional Neural Networks for Speech Recognition
Abdel-Hamid, O., Mohamed, A.-R., Jiang, H., Deng, L., Penn, G., & Yu, D. (2014) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2014.2339736 · Cited by 2,262
Leggetter, C. J., & Woodland, P. C. (1995) · Computer Speech & Language · doi:10.1006/csla.1995.0010 · Cited by 2,192 · cited 1×
“This is done by model space maximum likelihood linear regression (MLLR) [16] based on a regression tree which dynamically determines the transformation granularity.”
MethodDeep Neural Networks for Acoustic Modeling in Speech Recognition
Hinton, G. E., Deng, L., Yu, D., Dahl, G. E., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., et al. (2012) · IEEE Signal Processing Magazine · Cited by 2,136
Self-citationMaximum likelihood linear transformations for HMM-based speech recognition
Gales, M. J. F. (1998) · Computer Speech & Language · doi:10.1006/csla.1998.0043 · Cited by 1,482 · cited 13×
“(9): is indicated (9) where is the global STC transformation and is its in verse. is the (augmented) CMLLR transformation in the diagonalized LogMel feature space.”
BackgroundMethodInfluentialVoice Conversion Based on Maximum-Likelihood Estimation of Spectral Parameter Trajectory
Toda, T., Black, A. W., & Tokuda, K. (2007) · IEEE Transactions on Audio, Speech and Language Processing · doi:10.1109/tasl.2007.907344 · Cited by 937
Continuous probabilistic transform for voice conversion
Stylianou, Y., Cappe, O., & Moulines, E. (1998) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.661472 · Cited by 931 · cited 1×
“In [18], for each utterance in the training set, a warping factor is randomly chosen from [0.9, 1.1] to warp the frequency axis.”
MethodInfluentialConversational speech transcription using context-dependent deep neural networks
Seide, F., Li, G., & Yu, D. (2011) · Interspeech 2011 · doi:10.21437/interspeech.2011-169 · Cited by 852 · cited 1×
“In the past few years, deep neural networks (DNNs) have made dramatic impact in acoustic modeling and have delivered the state-of-the-art performance in automatic speech recognition (ASR) [5][6][7][8].”
BackgroundSeide, F., Li, G., Chen, X., & Yu, D. (2011) · 2011 IEEE Workshop on Automatic Speech Recognition & Understanding · doi:10.1109/asru.2011.6163899 · Cited by 632 · cited 1×
“In the past few years, deep neural networks (DNNs) have made dramatic impact in acoustic modeling and have delivered the state-of-the-art performance in automatic speech recognition (ASR) [5][6][7][8].”
BackgroundSemi-tied covariance matrices for hidden Markov models
Gales, M. J. F. (1999) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.759034 · Cited by 537 · cited 1×
“The components of LDA features are further decorrelated by a global semi-tied covariance (STC) matrix [28].”
BackgroundVoice conversion through vector quantization
Abe, M., Nakamura, S., Shikano, K., & Kuwabara, H. (2003) · ICASSP-88., International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1988.196671 · Cited by 415 · cited 1×
“In [18], for each utterance in the training set, a warping factor is randomly chosen from [0.9, 1.1] to warp the frequency axis.”
MethodInfluentialVocal Tract Length Perturbation (VTLP) improves speech recognition
Jaitly, N., & Hinton, E. (2013) · Cited by 348
Multi-style training for robust isolated-word speech recognition
Lippmann, R., Martin, E., & Paul, D. (2005) · ICASSP '87. IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1987.1169544 · Cited by 309 · cited 1×
“Digital Object Identi.er 10.1109/TASLP.2015.2438544 abundant variations presented in the training samples under the same labels and make classi.cations that are invariant to such variations.”
BackgroundInfluentialA frequency warping approach to speaker normalization
Lee, L., & Rose, R. (1998) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.650310 · Cited by 294 · cited 4×
“In this feature extraction pipeline 13-dimensional PLP features are used as the preliminary acoustic features after cepstral mean normalization (CMN) and vocal tract length normalization (VTLN)[27].”
MethodInfluentialImprovements to Deep Convolutional Neural Networks for LVCSR
Sainath, T. N., Kingsbury, B., Mohamed, A.-R., Dahl, G. E., Saon, G., Soltau, H., Beran, T., Aravkin, A. Y., & Ramabhadran, B. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707749 · Cited by 243 · cited 2×
“2 shows a mean and speaker normalized LogMel feature extraction pipeline used in IBM CNN systems [31][32].”
MethodSelf-citationLattice-Based Search for Spoken Utterance Retrieval
Saraçlar, M., & Sproat, R. (2004) · Cited by 223
Large-vocabulary speech recognition under adverse acoustic environments
Deng, L., Acero, A., Plumpe, M., & Huang, X. (2000) · 6th International Conference on Spoken Language Processing (ICSLP 2000) · doi:10.21437/icslp.2000-657 · Cited by 222
Kingsbury, B., Sainath, T. N., & Soltau, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-3 · Cited by 220 · cited 1×
“After the pre-training, the networks are first fine-tuned by 15 iterations of cross-entropy (CE) training and then followed by 30 iterations of Hessian-free (HF) sequence training based on the state-level minimum Bayes risk (sMBR) criterion [4].”
MethodSelf-citationSpeech recognition and keyword spotting for low-resource languages : Babel project research at CUED
Gales, M., Knill, K., Ragni, A., & Rath, S. P. (2014) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 207
Data augmentation for low resource languages
Ragni, A., Knill, K., Rath, S. P., & Gales, M. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-207 · Cited by 175 · cited 1×
“DATA AUGMENTATION APPROACHES A. Vocal Tract Length Perturbation (VTLP) VTLP, which was .rst proposed in [18], is representative of a group of data augmentation schemes that generate new samples through perturbing or distorting the speech spectra of the existing training samples.”
BackgroundRelevance score normalization for metasearch
Montague, M., & Aslam, J. A. (2001) · Proceedings of the tenth international conference on Information and knowledge management · doi:10.1145/502585.502657 · Cited by 156 · cited 1×
“Sum-to-one normalization [40][41] is applied to each term in the postings list.”
MethodVoice conversion through vector quantization
Abe, M., Nakamura, S., Shikano, K., & Kuwabara, H. (1990) · Journal of the Acoustical Society of Japan (E) · doi:10.1250/ast.11.71 · Cited by 150
The IBM Attila speech recognition toolkit
Soltau, H., Saon, G., & Kingsbury, B. (2010) · 2010 IEEE Spoken Language Technology Workshop · doi:10.1109/slt.2010.5700829 · Cited by 116 · cited 1×
“The VTLP is implemented using the IBM Attila toolkit [36] where VTL warping factors are quantized between .”
MethodSelf-citationElastic spectral distortion for low resource speech recognition with deep neural networks
Kanda, N., Takeda, R., & Obuchi, Y. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707748 · Cited by 113 · cited 1×
“…using label-preserving transformations has been widely used in neural network based pattern recognition in computer vision and image recognition tasks [11] [13] where transformations such as translation, deformation and re.ection have led to signi.cant improvements in recognition accuracy.”
MethodData Augmentation for deep neural network acoustic modeling
Cui, X., Goel, V., & Kingsbury, B. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854671 · Cited by 81 · cited 1×
“The work was later followed up by [19]–[21] on large vocabulary continuous speech recognition (LVCSR).”
BackgroundSelf-citationData augmentation for deep convolutional neural network acoustic modeling
Cui, X., Goel, V., & Kingsbury, B. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178831 · Cited by 77 · cited 2×
“For SFM, we still use the same strategy in Section III-A [33].”
BackgroundMethodSelf-citationJoint training of convolutional and non-convolutional neural networks
Soltau, H., Saon, G., & Sainath, T. N. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6854669 · Cited by 72 · cited 1×
“2 shows a mean and speaker normalized LogMel feature extraction pipeline used in IBM CNN systems [31][32].”
MethodSystem combination and score normalization for spoken term detection
Mamou, J., Cui, J., Cui, X., Gales, M., Kingsbury, B. E. D., Knill, K., Mangu, L., Nolden, D., Picheny, M., Ramabhadran, B., et al. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639278 · Cited by 56
Self-citationTüske, Z., Golik, P., Nolden, D., Schlüter, R., & Ney, H. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-348 · Cited by 52 · cited 2×
“DATA AUGMENTATION APPROACHES A. Vocal Tract Length Perturbation (VTLP) VTLP, which was .rst proposed in [18], is representative of a group of data augmentation schemes that generate new samples through perturbing or distorting the speech spectra of the existing training samples.”
BackgroundMethodA high-performance Cantonese keyword search system
Kingsbury, B. E. D., Cui, J., Cui, X., Gales, M., Knill, K., Mamou, J., Mangu, L., Nolden, D., Picheny, M., Ramabhadran, B., et al. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639279 · Cited by 49 · cited 3×
“The IBM KWS systems are based on a two-pass implementation of weighted finite state transducer (WFST) indexing and search [26], [38], [8].”
MethodSelf-citationInvestigation of unsupervised adaptation of DNN acoustic models with filter bank input
Yoshioka, T., Ragni, A., & Gales, M. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854825 · Cited by 48 · cited 1×
“It has been previously used in speaker adaptation for DNN and CNN acoustic models with filter bank input features in LVCSR tasks [31][34].”
MethodDeveloping speech recognition systems for corpus indexing under the IARPA Babel program
Cui, J., Cui, X., Ramabhadran, B., Kim, J., Kingsbury, B. E. D., Mamou, J., Mangu, L., Picheny, M., Sainath, T. N., & Sethy, A. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6638969 · Cited by 42 · cited 1×
“The IBM KWS systems are based on a two-pass implementation of weighted finite state transducer (WFST) indexing and search [26], [38], [8].”
MethodSelf-citationDistance measures for speech recognition
Hunt, M. J., & Lefebvre, C. (1989) · NPARC · Cited by 34
Noise robustness in speech to speech translation
Liu, F., Gao, Y., Gu, L., & Picheny, M. (2003) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.2003-745 · Cited by 18 · cited 1×
“Digital Object Identi.er 10.1109/TASLP.2015.2438544 abundant variations presented in the training samples under the same labels and make classi.cations that are invariant to such variations.”
BackgroundInfluentialBut neural network features for spontaneous Vietnamese in BABEL
Karafiát, M., Grézl, F., Hannemann, M., & Černocký, J. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6854679 · Cited by 14
Hidden Markov Acoustic Modeling With Bootstrap and Restructuring for Low-Resourced Languages
Cui, X., Xue, J., Chen, X., Olsen, P. A., Dognin, P., Chaudhari, U. V., Hershey, J. R., & Zhou, B. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2012.2199982 · Cited by 13 · cited 1×
“In the LLP configuration, the output layer has 2,000 units derived from the context-dependent states of a GMM-HMM acoustic model after bootstrap and restructuring [26][35].”
MethodSelf-citationCui, X., Kingsbury, B. E. D., Cui, J., Ramabhadran, B., Rosenberg, A., Rasooli, M. S., Rambow, O., Habash, N. Y., & Goel, V. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-477 · Cited by 9 · cited 1×
“Two data augmentation approaches, vocal tract length perturbation (VTLP) and stochasticfeature mapping (SFM), are investigated for both deep neural networks (DNNs) and convolutional neural networks (CNNs).”
BackgroundSelf-citation
Research works printed but not linked: 1
- fiscus, 2007, Results of the 2006 spoken term detection evaluation, Proc SIGIR Workshop Search Spontan Conversat Speech, 51
Cited by
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239
“(AVSR) systems [125], and data augmentation on the basis of l abel-preserving transformations [27].”
BackgroundEDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
Wei, J., & Zou, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1670 · Cited by 1,911
Convolutional neural network for earthquake detection and location
Perol, T., Gharbi, M., & Denolle, M. (2018) · Science Advances · doi:10.1126/sciadv.1700578 · Cited by 921
Um, T. T., Pfister, F. M. J., Pichler, D., Endo, S., Lang, M., Hirche, S., Fietzek, U., & Kulić, D. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136817 · Cited by 729
SMILES Enumeration as Data Augmentation for Neural Network Modeling of Molecules
Bjerrum, E. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07076 · Cited by 273
Metadata
- Authors
- Cui, X., Goel, V., & Kingsbury, B.
- Affiliations
- IBM (United States) (US)
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2015-09
- Volume
- 23
- Issue
- 9
- Pages
- 1469-1477
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W1979651826
- Cited by
- 376
- References
- 44
- Citations by year
- 2026: 8 · 2025: 18 · 2024: 20 · 2023: 39 · 2022: 52 · 2021: 56 · 2020: 55 · 2019: 52 · 2018: 31 · 2017: 26 · 2016: 9 · 2015: 6
- FWCI
- 23.27
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Keywords
- data augmentation, acoustic modeling, data sparsity, speaker variability, speech variation, Assamese language, Haitian Creole, low-resource languages, speech recognition, keyword search
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.