CoChat

Emotional voice conversion: Theory, databases and ESD

Open access
Zhou, Kun, Sisman, Berrak, Liu, Rui, Li, Haizhou · 2022 · Speech Communication
Cited by 181

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

Abstract

In this paper, we first provide a review of the state-of-the-art emotional voice conversion research, and the existing emotional speech databases. We then motivate the development of a novel emotional speech database (ESD) that addresses the increasing research need. With this paper, the ESD database1 is now made available to the research community. The ESD database consists of 350 parallel utterances spoken by 10 native English and 10 native Chinese speakers and covers 5 emotion categories (neutral, happy, angry, sad and surprise). More than 29 h of speech data were recorded in a controlled acoustic environment. The database is suitable for multi-speaker and cross-lingual emotional voice conversion studies. As case studies, we implement several state-of-the-art emotional voice conversion systems on the ESD database. This paper provides a reference study on ESD in conjunction with its release.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.specom.2021.11.006

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 78

  1. Abe, 1991, Statistical analysis of bilingual speaker’s speech for cross-language voice conversion, J. Acoust. Soc. Am., 90, 76, 10.1121/1.402284 · DOI
  2. Aihara, 2012, Gmm-based emotional voice conversion using spectrum and prosody features, Amer. J. Signal Process., 10.5923/j.ajsp.20120205.06 · doi:10.5923/j.ajsp.20120205.06
  3. Ak, K.E., Lim, J.H., Tham, J.Y., Kassim, A.A., 2019. Attribute manipulation generative adversarial networks for fashion images. In: Proceedings of the IEEE International Conference on Computer Vision. · DOI
  4. Ak, 2020, Semantically consistent text to fashion image synthesis with an enhanced attentional generative adversarial network, Pattern Recognit. Lett., 10.1016/j.patrec.2020.02.030 · DOI
  5. Ak, K.E., Sun, Y., Lim, J.H., 2020b. Learning cross-modal representations for language-based image manipulation. In: Proceedings of the IEEE ICIP.
  6. Akçay, 2020, Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers, Speech Commun, 10.1016/j.specom.2019.12.001 · DOI · Paper page
  7. Arias, 2020, Beyond correlation: acoustic transformation methods for the experimental study of emotional voice and speech, Emot. Rev.
  8. Banse, 1996, Acoustic profiles in vocal emotion expression, J. Personal. Soc. Psychol., 70, 614, 10.1037/0022-3514.70.3.614 · DOI · Paper page
  9. Barra-Chicote, 2010, Analysis of statistical parametric and unit selection speech synthesis systems applied to emotional speech, Speech Commun., 52, 394, 10.1016/j.specom.2009.12.007 · DOI
  10. Biassoni, 2016, Hot or cold anger? Verbal and vocal expression of anger while driving in a simulated anger-provoking scenario, Sage Open, 6, 10.1177/2158244016658084 · DOI
  11. Brunswik, 1956, Historical and thematic relations of psychology to other sciences, Sci. Mon., 83, 151
  12. Cao, 2014, Crema-d: Crowd-sourced emotional multimodal actors dataset, IEEE Trans. Affect. Comput., 5, 377, 10.1109/TAFFC.2014.2336244 · DOI · Paper page
  13. Cao, Y., Liu, Z., Chen, M., Ma, J., Wang, S., Xiao, J., 2020. Nonparallel emotional speech conversion using VAE-GAN. In: Proc. Interspeech 2020, pp. 3406–3410. · DOI
  14. Chen, 2014, Voice conversion using deep neural networks with layer-wise generative training, IEEE/ACM Trans. Audio Speech Lang. Process., 22, 1859, 10.1109/TASLP.2014.2353991 · DOI · Paper page
  15. Childers, 1989, Voice conversion, Speech Commun., 8, 147, 10.1016/0167-6393(89)90041-1 · DOI
  16. Choi, 2021, Sequence-to-sequence emotional voice conversion with strength control, IEEE Access, 9, 42674, 10.1109/ACCESS.2021.3065460 · DOI
  17. Çişman, B., Li, H., Tan, K.C., 2017. Sparse representation of phonetic features for voice conversion with and without parallel data. In: 2017 IEEE Automatic Speech Recognition and Understanding Workshop. · DOI
  18. Crumpton, 2016, A survey of using vocal prosody to convey emotion in robot speech, Int. J. Soc. Robot., 8, 271, 10.1007/s12369-015-0329-4 · DOI
  19. Desai, 2010, Spectral mapping using artificial neural networks for voice conversion, IEEE Trans. Audio Speech Lang. Process., 18, 954, 10.1109/TASL.2010.2047683 · DOI · Paper page
  20. El Ayadi, 2011, Survey on speech emotion recognition: Features, classification schemes, and databases, Pattern Recognit., 44, 572, 10.1016/j.patcog.2010.09.020 · DOI · Paper page
  21. Elbarougy, 2014, Improving speech emotion dimensions estimation using a three-layer model of human perception, Acoust. Sci. Technol., 35, 86, 10.1250/ast.35.86 · DOI
  22. Emir Ak, K., Hwee Lim, J., Yew Tham, J., Kassim, A., 2019. Semantically consistent hierarchical text to fashion image synthesis with an enhanced-attentional generative adversarial network. In: Proceedings of the IEEE International Conference on Computer Vision Workshops.
  23. Erickson, 2005, Expressive speech: Production, perception and application to speech synthesis, Acoust. Sci. Technol., 26, 317, 10.1250/ast.26.317 · DOI
  24. Erro, 2009, Voice conversion based on weighted frequency warping, IEEE Trans. Audio Speech Lang. Process., 18, 922, 10.1109/TASL.2009.2038663 · DOI · Paper page
  25. Erro, 2009, Emotion conversion based on prosodic unit selection, IEEE Trans. Audio Speech Lang. Process., 18, 974, 10.1109/TASL.2009.2038658 · doi:10.1109/tasl.2009.2038658
  26. Eyben, F., Weninger, F., Gross, F., Schuller, B., 2013. Recent developments in opensmile, the munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM International Conference on Multimedia, pp. 835–838. · DOI · Paper page
  27. Gao, J., Chakraborty, D., Tembine, H., Olaleye, O., 2019. Nonparallel emotional speech conversion. In: Proc. Interspeech 2019, pp. 2858–2862. · DOI
  28. Helander, 2010, Voice conversion using partial least squares regression, IEEE Trans. Audio Speech Lang. Process., 10.1109/TASL.2010.2041699 · DOI · Paper page
  29. Hirschberg, 2004, Pragmatics and intonation, Handb. Pragmat., 515
  30. Huang, 2008, A three-layered model for expressive speech perception, Speech Commun., 50, 810, 10.1016/j.specom.2008.05.017 · DOI
  31. Inanoglu, 2009, Data-driven emotion conversion in spoken english, Speech Commun., 10.1016/j.specom.2008.09.006 · DOI
  32. Johnstone, 2000, Vocal communication of emotion, Handb. Emot., 2, 220
  33. Juslin, 2001, Impact of intended emotion intensity on cue utilization and decoding accuracy in vocal expression of emotion, Emotion, 1, 381, 10.1037/1528-3542.1.4.381 · doi:10.1037/1528-3542.1.4.381 · Paper page
  34. Kane, 2014, Phonetic feature extraction for context-sensitive glottal source processing, Speech Commun., 59, 10, 10.1016/j.specom.2013.12.003 · doi:10.1016/j.specom.2013.12.003
  35. King, S., Karaiskos, V., 2013. The blizzard challenge 2013. In: Proc. Blizzard Challenge Workshop, Vol. 2013. · DOI
  36. Kotti, 2012, Speaker-independent emotion recognition exploiting a psychologically-inspired binary cascade classification schema, Int. J. Speech Technol., 15, 131, 10.1007/s10772-012-9127-7 · DOI
  37. Kwon, 2019, An effective style token weight control technique for end-to-end emotional speech synthesis, IEEE Signal Process. Lett., 26, 1383, 10.1109/LSP.2019.2931673 · DOI
  38. Le Moine, C., Obin, N., Roebel, A., 2021. Towards end-to-end f0 voice conversion based on dual-gan with convolutional wavelet kernels. · DOI
  39. Li, 2017, Cheavd: a Chinese natural emotional audio–visual database, J. Ambient Intell. Humaniz. Comput., 8, 913, 10.1007/s12652-016-0406-z · DOI
  40. Liu, S., Cao, Y., Kang, S., Hu, N., Liu, X., Su, D., Yu, D., Meng, H., 2020a. Transferring source style in non-parallel voice conversion. In: Proc. Interspeech 2020, pp. 4721–4725. · DOI
  41. Livingstone, 2018, The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english, PLoS One, 13, 10.1371/journal.pone.0196391 · DOI · Paper page
  42. Lorenzo-Trueba, 2018, Investigating different representations for modeling and controlling multiple emotions in dnn-based speech synthesis, Speech Commun., 10.1016/j.specom.2018.03.002 · DOI
  43. Luo, 2019, Emotional voice conversion using dual supervised adversarial networks with continuous wavelet transform f0 features, IEEE/ACM Trans. Audio Speech Lang. Process., 27, 1535, 10.1109/TASLP.2019.2923951 · DOI
  44. Maaten, 2008, Visualizing data using t-SNE, J. Mach. Learn. Res., 9, 2579
  45. Mehrabian, 1967, Decoding of inconsistent communications, J. Personal. Soc. Psychol., 6, 109, 10.1037/h0024532 · DOI · Paper page
  46. Mohammadi, 2017, An overview of voice conversion systems, Speech Commun., 88, 65, 10.1016/j.specom.2017.01.008 · DOI · Paper page
  47. Morise, 2016, World: a vocoder-based high-quality speech synthesis system for real-time applications, IEICE Trans. Inf. Syst., 99, 1877, 10.1587/transinf.2015EDP7457 · DOI · Paper page
  48. Müller, 2007, Dynamic time warping, Inf. Retr. Music Motion, 69, 10.1007/978-3-540-74048-3_4 · DOI · Paper page
  49. Nagrani, 2020, Voxceleb: Large-scale speaker verification in the wild, Comput. Speech Lang., 60, 10.1016/j.csl.2019.101027 · DOI · Paper page
  50. Nekvinda, T., Dušek, O., 2020. One model, many languages: Meta-learning for multilingual text-to-speech. In: Proc. Interspeech 2020, pp. 2972–2976. · DOI
  51. Nose, 2007, A style control technique for hmm-based expressive speech synthesis, IEICE Trans. Inf. Syst., 90, 1406, 10.1093/ietisy/e90-d.9.1406 · doi:10.1093/ietisy/e90-d.9.1406
  52. Parada-Cabaleiro, 2020, Demos: an Italian emotional speech corpus, Lang. Resour. Eval., 54, 341, 10.1007/s10579-019-09450-y · DOI
  53. Schroder, 2006, Expressing degree of activation in synthetic speech, IEEE Trans. Audio Speech Lang. Process., 14, 1128, 10.1109/TASL.2006.876118 · DOI
  54. Schuller, 2018, Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends, Commun. ACM, 61, 90, 10.1145/3129340 · DOI · Paper page
  55. Schuller, 2018, The age of artificial emotional intelligence, Computer, 51, 38, 10.1109/MC.2018.3620963 · DOI · Paper page
  56. Schuller, 2020, A review on five recent and near-future developments in computational processing of emotion in the human voice, Emot. Rev.
  57. Schuller, 2013, Paralinguistics in speech and language—State-of-the-art and the challenge, Comput. Speech Lang., 27, 4, 10.1016/j.csl.2012.02.005 · DOI · Paper page
  58. Shankar, R., Hsieh, H.-W., Charon, N., Venkataraman, A., 2020. Multi-speaker emotion conversion via latent variable regularization and a chained encoder-decoder-predictor network. In: Proc. Interspeech 2020, pp. 3391–3395. · DOI
  59. Shankar, R., Sager, J., Venkataraman, A., 2019. A multi-speaker emotion morphing model using highway networks and maximum likelihood objective. In: Proc. Interspeech 2019. · DOI
  60. Sisman, 2020, An overview of voice conversion and its challenges: From statistical modeling to deep learning, IEEE/ACM Trans. Audio Speech Lang. Process.
  61. Sisman, 2019, On the study of generative adversarial networks for cross-lingual voice conversion, IEEE ASRU
  62. Soleymani, 2017, A survey of multimodal sentiment analysis, Image Vis. Comput., 65, 3, 10.1016/j.imavis.2017.08.003 · DOI · Paper page
  63. Tao, 2006, Prosody conversion from neutral speech to emotional speech, IEEE Trans. Audio Speech Lang. Process.
  64. Tits, N., Wang, F., El Haddad, K., Pagel, V., Dutoit, T., 2019b. Visualization and interpretation of latent spaces for controlling expressive speech synthesis through audio analysis. In: Proc. Interspeech 2019, pp. 4475–4479. · DOI
  65. Toda, 2007, Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory, IEEE Trans. Audio Speech Lang. Process., 15, 2222, 10.1109/TASL.2007.907344 · DOI · Paper page
  66. Vaswani, 2017, Attention is all you need, Adv. Neural Inf. Process. Syst., 30, 5998
  67. Wu, 2009, Hierarchical prosody conversion using regression-based clustering for emotional speech synthesis, IEEE Trans. Audio Speech Lang. Process., 18, 1394
  68. Wu, Z., Xie, Z., King, S., 2019. The blizzard challenge 2019. In: Proc. Blizzard Challenge Workshop, Vol. 2019. · doi:10.21437/blizzard.2019-1
  69. Xu, 2011, Speech prosody: A methodological review, J. Speech Sci., 1, 85, 10.20396/joss.v1i1.15014 · DOI
  70. Xue, 2018, Voice conversion for emotional speech: Rule-based synthesis with degree of emotion controllable in dimensional space, Speech Commun., 10.1016/j.specom.2018.06.006 · DOI
  71. Yamagishi, 2005, Acoustic modeling of speaking styles and emotional expressions in hmm-based speech synthesis, IEICE Trans. Inf. Syst., 88, 502, 10.1093/ietisy/e88-d.3.502 · doi:10.1093/ietisy/e88-d.3.502
  72. Zhang, 2019, Non-parallel sequence-to-sequence voice conversion with disentangled linguistic and speaker representations, IEEE/ACM Trans. Audio Speech Lang. Process., 28, 540, 10.1109/TASLP.2019.2960721 · DOI · Paper page
  73. Zhang, 2019, Sequence-to-sequence acoustic modeling for voice conversion, IEEE/ACM Trans. Audio Speech Lang. Process., 27, 631, 10.1109/TASLP.2019.2892235 · DOI · Paper page
  74. Zhang, M., Wang, X., Fang, F., Li, H., Yamagishi, J., 2019. Joint training framework for text-to-speech and voice conversion using multi-source tacotron and wavenet. In: Proc. Interspeech 2019, pp. 1298–1302. · DOI
  75. Zhang, 2021, Transfer learning from speech synthesis to voice conversion with non-parallel training data, IEEE/ACM Trans. Audio Speech Lang. Process., 29, 1290, 10.1109/TASLP.2021.3066047 · DOI
  76. Zhou, X., Ling, Z.-H., King, S., 2020. The Blizzard Challenge 2020. In: Proc. Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020, pp. 1–18. · DOI
  77. Zhou, K., Sisman, B., Li, H., 2020. Transforming spectrum and prosody for emotional voice conversion with non-parallel training data. In: Proc. Odyssey 2020 the Speaker and Language Recognition Workshop, pp. 230–237. · DOI
  78. Zhou, K., Sisman, B., Zhang, M., Li, H., 2020. Converting anyone’s emotion: Towards speaker-independent emotional voice conversion. In: Proc. Interspeech 2020, pp. 3416–3420. · DOI

Cited by

See all 181

Metadata

Authors
Zhou, K., Sisman, B., Liu, R., & Li, H.
Venue
Speech Communication
Publisher
Elsevier BV
Published
2022-02
Volume
137
Pages
1-18
ISSN
0167-6393 (ISSN-L)
OpenAlex
W4205742757
Cited by
181
References
21
FWCI
8.82
Type
journal article
Language
en
Open access
hybrid
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.