Emotional voice conversion: Theory, databases and ESD
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
In this paper, we first provide a review of the state-of-the-art emotional voice conversion research, and the existing emotional speech databases. We then motivate the development of a novel emotional speech database (ESD) that addresses the increasing research need. With this paper, the ESD database1 is now made available to the research community. The ESD database consists of 350 parallel utterances spoken by 10 native English and 10 native Chinese speakers and covers 5 emotion categories (neutral, happy, angry, sad and surprise). More than 29 h of speech data were recorded in a controlled acoustic environment. The database is suitable for multi-speaker and cross-lingual emotional voice conversion studies. As case studies, we implement several state-of-the-art emotional voice conversion systems on the ESD database. This paper provides a reference study on ESD in conjunction with its release.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334
An argument for basic emotions
Ekman, P. (1992) · Cognition and Emotion · doi:10.1080/02699939208411068 · Cited by 9,646
StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., & Choo, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00916 · Cited by 4,163
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Opensmile: the munich versatile and fast open-source audio feature extractor
Eyben, F., Wöllmer, M., & Schuller, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874246 · Cited by 2,642
POSNER, J., RUSSELL, J. A., & PETERSON, B. S. (2005) · Development and Psychopathology · doi:10.1017/s0954579405050340 · Cited by 2,602
A database of German emotional speech
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-446 · Cited by 2,365
The eNTERFACE’05 Audio-Visual Emotion Database
Martin, O., Kotsia, I., Macq, B., & Pitas, I. (2006) · 22nd International Conference on Data Engineering Workshops (ICDEW'06) · doi:10.1109/icdew.2006.145 · Cited by 615
MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception
Busso, C., Parthasarathy, S., Burmania, A., AbdelWahab, M., Sadoughi, N., & Provost, E. M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2515617 · Cited by 391
“StarGAN-VC: non-parallel many-to-many voice conversion using star generative adversarial networks,”
Kameoka, H., Kaneko, T., Tanaka, K., & Hojo, N. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639535 · Cited by 371
Gunes, H., & Schuller, B. (2013) · Image and Vision Computing · doi:10.1016/j.imavis.2012.06.016 · Cited by 360
CycleGAN-VC: Non-parallel Voice Conversion Using Cycle-Consistent Adversarial Networks
Kaneko, T., & Kameoka, H. (2018) · 2018 26th European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2018.8553236 · Cited by 358
CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92)
Yamagishi, J., Veaux, C., & MacDonald, K. (2019) · University of Edinburgh. The Centre for Speech Technology Research (CSTR) · doi:10.7488/ds/2645 · Cited by 356
The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods
Lorenzo-Trueba, J., Yamagishi, J., Toda, T., Saito, D., Villavicencio, F., Kinnunen, T., & Ling, Z. (2018) · The Speaker and Language Recognition Workshop (Odyssey 2018) · doi:10.21437/odyssey.2018-28 · Cited by 305
Bachorowski, J.-A., & Owren, M. J. (1995) · Psychological Science · doi:10.1111/j.1467-9280.1995.tb00596.x · Cited by 259
“High-quality nonparallel voice conversion based on cycle-consistent adversarial network,”
Fang, F., Yamagishi, J., Echizen, I., & Lorenzo-Trueba, J. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8462342 · Cited by 128
Voice Conversion Challenge 2020 –- Intra-lingual semi-parallel and cross-lingual voice conversion –-
Yi, Z., Huang, W.-C., Tian, X., Yamagishi, J., Das, R. K., Kinnunen, T., Ling, Z.-H., & Toda, T. (2020) · Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020 · doi:10.21437/vcc_bc.2020-14 · Cited by 126
Group sparse representation with wavenet vocoder adaptation for spectrum and prosody conversion
Sisman, B., Zhang, M., & Li, H. (2019) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2910637 · Cited by 52
Self-citationAdaptive Wavenet Vocoder for Residual Compensation in GAN-Based Voice Conversion
Sisman, B., Zhang, M., Sakti, S., Li, H., & Nakamura, S. (2018) · 2018 IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt.2018.8639507 · Cited by 36
Self-citationLibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech
Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z., & Wu, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.02882 · Cited by 23
Research works printed but not linked: 78
- Abe, 1991, Statistical analysis of bilingual speaker’s speech for cross-language voice conversion, J. Acoust. Soc. Am., 90, 76, 10.1121/1.402284 · DOI
- Aihara, 2012, Gmm-based emotional voice conversion using spectrum and prosody features, Amer. J. Signal Process., 10.5923/j.ajsp.20120205.06 · doi:10.5923/j.ajsp.20120205.06
- Ak, K.E., Lim, J.H., Tham, J.Y., Kassim, A.A., 2019. Attribute manipulation generative adversarial networks for fashion images. In: Proceedings of the IEEE International Conference on Computer Vision. · DOI
- Ak, 2020, Semantically consistent text to fashion image synthesis with an enhanced attentional generative adversarial network, Pattern Recognit. Lett., 10.1016/j.patrec.2020.02.030 · DOI
- Ak, K.E., Sun, Y., Lim, J.H., 2020b. Learning cross-modal representations for language-based image manipulation. In: Proceedings of the IEEE ICIP.
- Akçay, 2020, Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers, Speech Commun, 10.1016/j.specom.2019.12.001 · DOI · Paper page
- Arias, 2020, Beyond correlation: acoustic transformation methods for the experimental study of emotional voice and speech, Emot. Rev.
- Banse, 1996, Acoustic profiles in vocal emotion expression, J. Personal. Soc. Psychol., 70, 614, 10.1037/0022-3514.70.3.614 · DOI · Paper page
- Barra-Chicote, 2010, Analysis of statistical parametric and unit selection speech synthesis systems applied to emotional speech, Speech Commun., 52, 394, 10.1016/j.specom.2009.12.007 · DOI
- Biassoni, 2016, Hot or cold anger? Verbal and vocal expression of anger while driving in a simulated anger-provoking scenario, Sage Open, 6, 10.1177/2158244016658084 · DOI
- Brunswik, 1956, Historical and thematic relations of psychology to other sciences, Sci. Mon., 83, 151
- Cao, 2014, Crema-d: Crowd-sourced emotional multimodal actors dataset, IEEE Trans. Affect. Comput., 5, 377, 10.1109/TAFFC.2014.2336244 · DOI · Paper page
- Cao, Y., Liu, Z., Chen, M., Ma, J., Wang, S., Xiao, J., 2020. Nonparallel emotional speech conversion using VAE-GAN. In: Proc. Interspeech 2020, pp. 3406–3410. · DOI
- Chen, 2014, Voice conversion using deep neural networks with layer-wise generative training, IEEE/ACM Trans. Audio Speech Lang. Process., 22, 1859, 10.1109/TASLP.2014.2353991 · DOI · Paper page
- Childers, 1989, Voice conversion, Speech Commun., 8, 147, 10.1016/0167-6393(89)90041-1 · DOI
- Choi, 2021, Sequence-to-sequence emotional voice conversion with strength control, IEEE Access, 9, 42674, 10.1109/ACCESS.2021.3065460 · DOI
- Çişman, B., Li, H., Tan, K.C., 2017. Sparse representation of phonetic features for voice conversion with and without parallel data. In: 2017 IEEE Automatic Speech Recognition and Understanding Workshop. · DOI
- Crumpton, 2016, A survey of using vocal prosody to convey emotion in robot speech, Int. J. Soc. Robot., 8, 271, 10.1007/s12369-015-0329-4 · DOI
- Desai, 2010, Spectral mapping using artificial neural networks for voice conversion, IEEE Trans. Audio Speech Lang. Process., 18, 954, 10.1109/TASL.2010.2047683 · DOI · Paper page
- El Ayadi, 2011, Survey on speech emotion recognition: Features, classification schemes, and databases, Pattern Recognit., 44, 572, 10.1016/j.patcog.2010.09.020 · DOI · Paper page
- Elbarougy, 2014, Improving speech emotion dimensions estimation using a three-layer model of human perception, Acoust. Sci. Technol., 35, 86, 10.1250/ast.35.86 · DOI
- Emir Ak, K., Hwee Lim, J., Yew Tham, J., Kassim, A., 2019. Semantically consistent hierarchical text to fashion image synthesis with an enhanced-attentional generative adversarial network. In: Proceedings of the IEEE International Conference on Computer Vision Workshops.
- Erickson, 2005, Expressive speech: Production, perception and application to speech synthesis, Acoust. Sci. Technol., 26, 317, 10.1250/ast.26.317 · DOI
- Erro, 2009, Voice conversion based on weighted frequency warping, IEEE Trans. Audio Speech Lang. Process., 18, 922, 10.1109/TASL.2009.2038663 · DOI · Paper page
- Erro, 2009, Emotion conversion based on prosodic unit selection, IEEE Trans. Audio Speech Lang. Process., 18, 974, 10.1109/TASL.2009.2038658 · doi:10.1109/tasl.2009.2038658
- Eyben, F., Weninger, F., Gross, F., Schuller, B., 2013. Recent developments in opensmile, the munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM International Conference on Multimedia, pp. 835–838. · DOI · Paper page
- Gao, J., Chakraborty, D., Tembine, H., Olaleye, O., 2019. Nonparallel emotional speech conversion. In: Proc. Interspeech 2019, pp. 2858–2862. · DOI
- Helander, 2010, Voice conversion using partial least squares regression, IEEE Trans. Audio Speech Lang. Process., 10.1109/TASL.2010.2041699 · DOI · Paper page
- Hirschberg, 2004, Pragmatics and intonation, Handb. Pragmat., 515
- Huang, 2008, A three-layered model for expressive speech perception, Speech Commun., 50, 810, 10.1016/j.specom.2008.05.017 · DOI
- Inanoglu, 2009, Data-driven emotion conversion in spoken english, Speech Commun., 10.1016/j.specom.2008.09.006 · DOI
- Johnstone, 2000, Vocal communication of emotion, Handb. Emot., 2, 220
- Juslin, 2001, Impact of intended emotion intensity on cue utilization and decoding accuracy in vocal expression of emotion, Emotion, 1, 381, 10.1037/1528-3542.1.4.381 · doi:10.1037/1528-3542.1.4.381 · Paper page
- Kane, 2014, Phonetic feature extraction for context-sensitive glottal source processing, Speech Commun., 59, 10, 10.1016/j.specom.2013.12.003 · doi:10.1016/j.specom.2013.12.003
- King, S., Karaiskos, V., 2013. The blizzard challenge 2013. In: Proc. Blizzard Challenge Workshop, Vol. 2013. · DOI
- Kotti, 2012, Speaker-independent emotion recognition exploiting a psychologically-inspired binary cascade classification schema, Int. J. Speech Technol., 15, 131, 10.1007/s10772-012-9127-7 · DOI
- Kwon, 2019, An effective style token weight control technique for end-to-end emotional speech synthesis, IEEE Signal Process. Lett., 26, 1383, 10.1109/LSP.2019.2931673 · DOI
- Le Moine, C., Obin, N., Roebel, A., 2021. Towards end-to-end f0 voice conversion based on dual-gan with convolutional wavelet kernels. · DOI
- Li, 2017, Cheavd: a Chinese natural emotional audio–visual database, J. Ambient Intell. Humaniz. Comput., 8, 913, 10.1007/s12652-016-0406-z · DOI
- Liu, S., Cao, Y., Kang, S., Hu, N., Liu, X., Su, D., Yu, D., Meng, H., 2020a. Transferring source style in non-parallel voice conversion. In: Proc. Interspeech 2020, pp. 4721–4725. · DOI
- Livingstone, 2018, The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english, PLoS One, 13, 10.1371/journal.pone.0196391 · DOI · Paper page
- Lorenzo-Trueba, 2018, Investigating different representations for modeling and controlling multiple emotions in dnn-based speech synthesis, Speech Commun., 10.1016/j.specom.2018.03.002 · DOI
- Luo, 2019, Emotional voice conversion using dual supervised adversarial networks with continuous wavelet transform f0 features, IEEE/ACM Trans. Audio Speech Lang. Process., 27, 1535, 10.1109/TASLP.2019.2923951 · DOI
- Maaten, 2008, Visualizing data using t-SNE, J. Mach. Learn. Res., 9, 2579
- Mehrabian, 1967, Decoding of inconsistent communications, J. Personal. Soc. Psychol., 6, 109, 10.1037/h0024532 · DOI · Paper page
- Mohammadi, 2017, An overview of voice conversion systems, Speech Commun., 88, 65, 10.1016/j.specom.2017.01.008 · DOI · Paper page
- Morise, 2016, World: a vocoder-based high-quality speech synthesis system for real-time applications, IEICE Trans. Inf. Syst., 99, 1877, 10.1587/transinf.2015EDP7457 · DOI · Paper page
- Müller, 2007, Dynamic time warping, Inf. Retr. Music Motion, 69, 10.1007/978-3-540-74048-3_4 · DOI · Paper page
- Nagrani, 2020, Voxceleb: Large-scale speaker verification in the wild, Comput. Speech Lang., 60, 10.1016/j.csl.2019.101027 · DOI · Paper page
- Nekvinda, T., Dušek, O., 2020. One model, many languages: Meta-learning for multilingual text-to-speech. In: Proc. Interspeech 2020, pp. 2972–2976. · DOI
- Nose, 2007, A style control technique for hmm-based expressive speech synthesis, IEICE Trans. Inf. Syst., 90, 1406, 10.1093/ietisy/e90-d.9.1406 · doi:10.1093/ietisy/e90-d.9.1406
- Parada-Cabaleiro, 2020, Demos: an Italian emotional speech corpus, Lang. Resour. Eval., 54, 341, 10.1007/s10579-019-09450-y · DOI
- Schroder, 2006, Expressing degree of activation in synthetic speech, IEEE Trans. Audio Speech Lang. Process., 14, 1128, 10.1109/TASL.2006.876118 · DOI
- Schuller, 2018, Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends, Commun. ACM, 61, 90, 10.1145/3129340 · DOI · Paper page
- Schuller, 2018, The age of artificial emotional intelligence, Computer, 51, 38, 10.1109/MC.2018.3620963 · DOI · Paper page
- Schuller, 2020, A review on five recent and near-future developments in computational processing of emotion in the human voice, Emot. Rev.
- Schuller, 2013, Paralinguistics in speech and language—State-of-the-art and the challenge, Comput. Speech Lang., 27, 4, 10.1016/j.csl.2012.02.005 · DOI · Paper page
- Shankar, R., Hsieh, H.-W., Charon, N., Venkataraman, A., 2020. Multi-speaker emotion conversion via latent variable regularization and a chained encoder-decoder-predictor network. In: Proc. Interspeech 2020, pp. 3391–3395. · DOI
- Shankar, R., Sager, J., Venkataraman, A., 2019. A multi-speaker emotion morphing model using highway networks and maximum likelihood objective. In: Proc. Interspeech 2019. · DOI
- Sisman, 2020, An overview of voice conversion and its challenges: From statistical modeling to deep learning, IEEE/ACM Trans. Audio Speech Lang. Process.
- Sisman, 2019, On the study of generative adversarial networks for cross-lingual voice conversion, IEEE ASRU
- Soleymani, 2017, A survey of multimodal sentiment analysis, Image Vis. Comput., 65, 3, 10.1016/j.imavis.2017.08.003 · DOI · Paper page
- Tao, 2006, Prosody conversion from neutral speech to emotional speech, IEEE Trans. Audio Speech Lang. Process.
- Tits, N., Wang, F., El Haddad, K., Pagel, V., Dutoit, T., 2019b. Visualization and interpretation of latent spaces for controlling expressive speech synthesis through audio analysis. In: Proc. Interspeech 2019, pp. 4475–4479. · DOI
- Toda, 2007, Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory, IEEE Trans. Audio Speech Lang. Process., 15, 2222, 10.1109/TASL.2007.907344 · DOI · Paper page
- Vaswani, 2017, Attention is all you need, Adv. Neural Inf. Process. Syst., 30, 5998
- Wu, 2009, Hierarchical prosody conversion using regression-based clustering for emotional speech synthesis, IEEE Trans. Audio Speech Lang. Process., 18, 1394
- Wu, Z., Xie, Z., King, S., 2019. The blizzard challenge 2019. In: Proc. Blizzard Challenge Workshop, Vol. 2019. · doi:10.21437/blizzard.2019-1
- Xu, 2011, Speech prosody: A methodological review, J. Speech Sci., 1, 85, 10.20396/joss.v1i1.15014 · DOI
- Xue, 2018, Voice conversion for emotional speech: Rule-based synthesis with degree of emotion controllable in dimensional space, Speech Commun., 10.1016/j.specom.2018.06.006 · DOI
- Yamagishi, 2005, Acoustic modeling of speaking styles and emotional expressions in hmm-based speech synthesis, IEICE Trans. Inf. Syst., 88, 502, 10.1093/ietisy/e88-d.3.502 · doi:10.1093/ietisy/e88-d.3.502
- Zhang, 2019, Non-parallel sequence-to-sequence voice conversion with disentangled linguistic and speaker representations, IEEE/ACM Trans. Audio Speech Lang. Process., 28, 540, 10.1109/TASLP.2019.2960721 · DOI · Paper page
- Zhang, 2019, Sequence-to-sequence acoustic modeling for voice conversion, IEEE/ACM Trans. Audio Speech Lang. Process., 27, 631, 10.1109/TASLP.2019.2892235 · DOI · Paper page
- Zhang, M., Wang, X., Fang, F., Li, H., Yamagishi, J., 2019. Joint training framework for text-to-speech and voice conversion using multi-source tacotron and wavenet. In: Proc. Interspeech 2019, pp. 1298–1302. · DOI
- Zhang, 2021, Transfer learning from speech synthesis to voice conversion with non-parallel training data, IEEE/ACM Trans. Audio Speech Lang. Process., 29, 1290, 10.1109/TASLP.2021.3066047 · DOI
- Zhou, X., Ling, Z.-H., King, S., 2020. The Blizzard Challenge 2020. In: Proc. Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020, pp. 1–18. · DOI
- Zhou, K., Sisman, B., Li, H., 2020. Transforming spectrum and prosody for emotional voice conversion with non-parallel training data. In: Proc. Odyssey 2020 the Speaker and Language Recognition Workshop, pp. 230–237. · DOI
- Zhou, K., Sisman, B., Zhang, M., Li, H., 2020. Converting anyone’s emotion: Towards speaker-independent emotional voice conversion. In: Proc. Interspeech 2020, pp. 3416–3420. · DOI
Cited by
Domain Generalization via Aggregation and Separation for Audio Deepfake Detection
Xie, Y., Cheng, H., Wang, Y., & Ye, L. (2024) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2023.3324724 · Cited by 59
Metadata
- Authors
- Zhou, K., Sisman, B., Liu, R., & Li, H.
- Venue
- Speech Communication
- Publisher
- Elsevier BV
- Published
- 2022-02
- Volume
- 137
- Pages
- 1-18
- ISSN
- 0167-6393 (ISSN-L)
- OpenAlex
- W4205742757
- Cited by
- 181
- References
- 21
- FWCI
- 8.82
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.