Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 181

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Emotional voice conversion: Theory, databases and ESD

Open access
Zhou, Kun, Sisman, Berrak, Liu, Rui, Li, Haizhou · 2022 · Speech Communication

Abstract

In this paper, we first provide a review of the state-of-the-art emotional voice conversion research, and the existing emotional speech databases. We then motivate the development of a novel emotional speech database (ESD) that addresses the increasing research need. With this paper, the ESD database1 is now made available to the research community. The ESD database consists of 350 parallel utterances spoken by 10 native English and 10 native Chinese speakers and covers 5 emotion categories (neutral, happy, angry, sad and surprise). More than 29 h of speech data were recorded in a controlled acoustic environment. The database is suitable for multi-speaker and cross-lingual emotional voice conversion studies. As case studies, we implement several state-of-the-art emotional voice conversion systems on the ESD database. This paper provides a reference study on ESD in conjunction with its release.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.specom.2021.11.006

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 18 references

Research works printed but not linked: 80

  1. Abe, 1991, Statistical analysis of bilingual speaker’s speech for cross-language voice conversion, J. Acoust. Soc. Am., 90, 76, 10.1121/1.402284 · DOI
  2. Aihara, 2012, Gmm-based emotional voice conversion using spectrum and prosody features, Amer. J. Signal Process., 10.5923/j.ajsp.20120205.06 · doi:10.5923/j.ajsp.20120205.06
  3. Ak, K.E., Lim, J.H., Tham, J.Y., Kassim, A.A., 2019. Attribute manipulation generative adversarial networks for fashion images. In: Proceedings of the IEEE International Conference on Computer Vision. · DOI
  4. Ak, 2020, Semantically consistent text to fashion image synthesis with an enhanced attentional generative adversarial network, Pattern Recognit. Lett., 10.1016/j.patrec.2020.02.030 · DOI
  5. Ak, K.E., Sun, Y., Lim, J.H., 2020b. Learning cross-modal representations for language-based image manipulation. In: Proceedings of the IEEE ICIP.
  6. Akçay, 2020, Speech emotion recognition: Emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers, Speech Commun, 10.1016/j.specom.2019.12.001 · DOI · Paper page
  7. Arias, 2020, Beyond correlation: acoustic transformation methods for the experimental study of emotional voice and speech, Emot. Rev.
  8. Banse, 1996, Acoustic profiles in vocal emotion expression, J. Personal. Soc. Psychol., 70, 614, 10.1037/0022-3514.70.3.614 · DOI · Paper page
  9. Barra-Chicote, 2010, Analysis of statistical parametric and unit selection speech synthesis systems applied to emotional speech, Speech Commun., 52, 394, 10.1016/j.specom.2009.12.007 · DOI
  10. Biassoni, 2016, Hot or cold anger? Verbal and vocal expression of anger while driving in a simulated anger-provoking scenario, Sage Open, 6, 10.1177/2158244016658084 · DOI
  11. Brunswik, 1956, Historical and thematic relations of psychology to other sciences, Sci. Mon., 83, 151
  12. Busso, 2008, Iemocap: Interactive emotional dyadic motion capture database, Lang. Resour. Eval., 42, 335, 10.1007/s10579-008-9076-6 · DOI · Paper page
  13. Cao, 2014, Crema-d: Crowd-sourced emotional multimodal actors dataset, IEEE Trans. Affect. Comput., 5, 377, 10.1109/TAFFC.2014.2336244 · DOI · Paper page
  14. Cao, Y., Liu, Z., Chen, M., Ma, J., Wang, S., Xiao, J., 2020. Nonparallel emotional speech conversion using VAE-GAN. In: Proc. Interspeech 2020, pp. 3406–3410. · DOI
  15. Chen, 2014, Voice conversion using deep neural networks with layer-wise generative training, IEEE/ACM Trans. Audio Speech Lang. Process., 22, 1859, 10.1109/TASLP.2014.2353991 · DOI · Paper page
  16. Childers, 1989, Voice conversion, Speech Commun., 8, 147, 10.1016/0167-6393(89)90041-1 · DOI
  17. Choi, 2021, Sequence-to-sequence emotional voice conversion with strength control, IEEE Access, 9, 42674, 10.1109/ACCESS.2021.3065460 · DOI
  18. Çişman, B., Li, H., Tan, K.C., 2017. Sparse representation of phonetic features for voice conversion with and without parallel data. In: 2017 IEEE Automatic Speech Recognition and Understanding Workshop. · DOI
  19. Crumpton, 2016, A survey of using vocal prosody to convey emotion in robot speech, Int. J. Soc. Robot., 8, 271, 10.1007/s12369-015-0329-4 · DOI
  20. Desai, 2010, Spectral mapping using artificial neural networks for voice conversion, IEEE Trans. Audio Speech Lang. Process., 18, 954, 10.1109/TASL.2010.2047683 · DOI · Paper page
  21. El Ayadi, 2011, Survey on speech emotion recognition: Features, classification schemes, and databases, Pattern Recognit., 44, 572, 10.1016/j.patcog.2010.09.020 · DOI · Paper page
  22. Elbarougy, 2014, Improving speech emotion dimensions estimation using a three-layer model of human perception, Acoust. Sci. Technol., 35, 86, 10.1250/ast.35.86 · DOI
  23. Emir Ak, K., Hwee Lim, J., Yew Tham, J., Kassim, A., 2019. Semantically consistent hierarchical text to fashion image synthesis with an enhanced-attentional generative adversarial network. In: Proceedings of the IEEE International Conference on Computer Vision Workshops.
  24. Erickson, 2005, Expressive speech: Production, perception and application to speech synthesis, Acoust. Sci. Technol., 26, 317, 10.1250/ast.26.317 · DOI
  25. Erro, 2009, Voice conversion based on weighted frequency warping, IEEE Trans. Audio Speech Lang. Process., 18, 922, 10.1109/TASL.2009.2038663 · DOI · Paper page
  26. Erro, 2009, Emotion conversion based on prosodic unit selection, IEEE Trans. Audio Speech Lang. Process., 18, 974, 10.1109/TASL.2009.2038658 · doi:10.1109/tasl.2009.2038658
  27. Eyben, F., Weninger, F., Gross, F., Schuller, B., 2013. Recent developments in opensmile, the munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM International Conference on Multimedia, pp. 835–838. · DOI · Paper page
  28. Gao, J., Chakraborty, D., Tembine, H., Olaleye, O., 2019. Nonparallel emotional speech conversion. In: Proc. Interspeech 2019, pp. 2858–2862. · DOI
  29. Helander, 2010, Voice conversion using partial least squares regression, IEEE Trans. Audio Speech Lang. Process., 10.1109/TASL.2010.2041699 · DOI · Paper page
  30. Hirschberg, 2004, Pragmatics and intonation, Handb. Pragmat., 515
  31. Huang, 2008, A three-layered model for expressive speech perception, Speech Commun., 50, 810, 10.1016/j.specom.2008.05.017 · DOI
  32. Inanoglu, 2009, Data-driven emotion conversion in spoken english, Speech Commun., 10.1016/j.specom.2008.09.006 · DOI
  33. Johnstone, 2000, Vocal communication of emotion, Handb. Emot., 2, 220
  34. Juslin, 2001, Impact of intended emotion intensity on cue utilization and decoding accuracy in vocal expression of emotion, Emotion, 1, 381, 10.1037/1528-3542.1.4.381 · doi:10.1037/1528-3542.1.4.381 · Paper page
  35. Kane, 2014, Phonetic feature extraction for context-sensitive glottal source processing, Speech Commun., 59, 10, 10.1016/j.specom.2013.12.003 · doi:10.1016/j.specom.2013.12.003
  36. King, S., Karaiskos, V., 2013. The blizzard challenge 2013. In: Proc. Blizzard Challenge Workshop, Vol. 2013. · DOI
  37. Kotti, 2012, Speaker-independent emotion recognition exploiting a psychologically-inspired binary cascade classification schema, Int. J. Speech Technol., 15, 131, 10.1007/s10772-012-9127-7 · DOI
  38. Kwon, 2019, An effective style token weight control technique for end-to-end emotional speech synthesis, IEEE Signal Process. Lett., 26, 1383, 10.1109/LSP.2019.2931673 · DOI
  39. Le Moine, C., Obin, N., Roebel, A., 2021. Towards end-to-end f0 voice conversion based on dual-gan with convolutional wavelet kernels. · DOI
  40. Li, 2017, Cheavd: a Chinese natural emotional audio–visual database, J. Ambient Intell. Humaniz. Comput., 8, 913, 10.1007/s12652-016-0406-z · DOI
  41. Liu, S., Cao, Y., Kang, S., Hu, N., Liu, X., Su, D., Yu, D., Meng, H., 2020a. Transferring source style in non-parallel voice conversion. In: Proc. Interspeech 2020, pp. 4721–4725. · DOI
  42. Livingstone, 2018, The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english, PLoS One, 13, 10.1371/journal.pone.0196391 · DOI · Paper page
  43. Lorenzo-Trueba, 2018, Investigating different representations for modeling and controlling multiple emotions in dnn-based speech synthesis, Speech Commun., 10.1016/j.specom.2018.03.002 · DOI
  44. Luo, 2019, Emotional voice conversion using dual supervised adversarial networks with continuous wavelet transform f0 features, IEEE/ACM Trans. Audio Speech Lang. Process., 27, 1535, 10.1109/TASLP.2019.2923951 · DOI
  45. Maaten, 2008, Visualizing data using t-SNE, J. Mach. Learn. Res., 9, 2579
  46. Mehrabian, 1967, Decoding of inconsistent communications, J. Personal. Soc. Psychol., 6, 109, 10.1037/h0024532 · DOI · Paper page
  47. Mohammadi, 2017, An overview of voice conversion systems, Speech Commun., 88, 65, 10.1016/j.specom.2017.01.008 · DOI · Paper page
  48. Morise, 2016, World: a vocoder-based high-quality speech synthesis system for real-time applications, IEICE Trans. Inf. Syst., 99, 1877, 10.1587/transinf.2015EDP7457 · DOI · Paper page
  49. Müller, 2007, Dynamic time warping, Inf. Retr. Music Motion, 69, 10.1007/978-3-540-74048-3_4 · DOI · Paper page
  50. Nagrani, 2020, Voxceleb: Large-scale speaker verification in the wild, Comput. Speech Lang., 60, 10.1016/j.csl.2019.101027 · DOI · Paper page
  51. Nekvinda, T., Dušek, O., 2020. One model, many languages: Meta-learning for multilingual text-to-speech. In: Proc. Interspeech 2020, pp. 2972–2976. · DOI
  52. Nose, 2007, A style control technique for hmm-based expressive speech synthesis, IEICE Trans. Inf. Syst., 90, 1406, 10.1093/ietisy/e90-d.9.1406 · doi:10.1093/ietisy/e90-d.9.1406
  53. Parada-Cabaleiro, 2020, Demos: an Italian emotional speech corpus, Lang. Resour. Eval., 54, 341, 10.1007/s10579-019-09450-y · DOI
  54. Schroder, 2006, Expressing degree of activation in synthetic speech, IEEE Trans. Audio Speech Lang. Process., 14, 1128, 10.1109/TASL.2006.876118 · DOI
  55. Schuller, 2018, Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends, Commun. ACM, 61, 90, 10.1145/3129340 · DOI · Paper page
  56. Schuller, 2018, The age of artificial emotional intelligence, Computer, 51, 38, 10.1109/MC.2018.3620963 · DOI
  57. Schuller, 2020, A review on five recent and near-future developments in computational processing of emotion in the human voice, Emot. Rev.
  58. Schuller, 2013, Paralinguistics in speech and language—State-of-the-art and the challenge, Comput. Speech Lang., 27, 4, 10.1016/j.csl.2012.02.005 · DOI · Paper page
  59. Shankar, R., Hsieh, H.-W., Charon, N., Venkataraman, A., 2020. Multi-speaker emotion conversion via latent variable regularization and a chained encoder-decoder-predictor network. In: Proc. Interspeech 2020, pp. 3391–3395. · DOI
  60. Shankar, R., Sager, J., Venkataraman, A., 2019. A multi-speaker emotion morphing model using highway networks and maximum likelihood objective. In: Proc. Interspeech 2019. · DOI
  61. Sisman, 2020, An overview of voice conversion and its challenges: From statistical modeling to deep learning, IEEE/ACM Trans. Audio Speech Lang. Process.
  62. Sisman, 2019, On the study of generative adversarial networks for cross-lingual voice conversion, IEEE ASRU
  63. Soleymani, 2017, A survey of multimodal sentiment analysis, Image Vis. Comput., 65, 3, 10.1016/j.imavis.2017.08.003 · DOI · Paper page
  64. Tao, 2006, Prosody conversion from neutral speech to emotional speech, IEEE Trans. Audio Speech Lang. Process.
  65. Tits, N., Wang, F., El Haddad, K., Pagel, V., Dutoit, T., 2019b. Visualization and interpretation of latent spaces for controlling expressive speech synthesis through audio analysis. In: Proc. Interspeech 2019, pp. 4475–4479. · DOI
  66. Toda, 2007, Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory, IEEE Trans. Audio Speech Lang. Process., 15, 2222, 10.1109/TASL.2007.907344 · DOI · Paper page
  67. Vaswani, 2017, Attention is all you need, Adv. Neural Inf. Process. Syst., 30, 5998
  68. Wu, 2009, Hierarchical prosody conversion using regression-based clustering for emotional speech synthesis, IEEE Trans. Audio Speech Lang. Process., 18, 1394
  69. Wu, Z., Xie, Z., King, S., 2019. The blizzard challenge 2019. In: Proc. Blizzard Challenge Workshop, Vol. 2019. · doi:10.21437/blizzard.2019-1
  70. Xu, 2011, Speech prosody: A methodological review, J. Speech Sci., 1, 85, 10.20396/joss.v1i1.15014 · DOI
  71. Xue, 2018, Voice conversion for emotional speech: Rule-based synthesis with degree of emotion controllable in dimensional space, Speech Commun., 10.1016/j.specom.2018.06.006 · DOI
  72. Yamagishi, 2005, Acoustic modeling of speaking styles and emotional expressions in hmm-based speech synthesis, IEICE Trans. Inf. Syst., 88, 502, 10.1093/ietisy/e88-d.3.502 · doi:10.1093/ietisy/e88-d.3.502
  73. Zen, 2019, LibriTTS: A Corpus derived from LibriSpeech for text-to-speech, 10.21437/Interspeech.2019-2441 · DOI · Paper page
  74. Zhang, 2019, Non-parallel sequence-to-sequence voice conversion with disentangled linguistic and speaker representations, IEEE/ACM Trans. Audio Speech Lang. Process., 28, 540, 10.1109/TASLP.2019.2960721 · DOI
  75. Zhang, 2019, Sequence-to-sequence acoustic modeling for voice conversion, IEEE/ACM Trans. Audio Speech Lang. Process., 27, 631, 10.1109/TASLP.2019.2892235 · DOI
  76. Zhang, M., Wang, X., Fang, F., Li, H., Yamagishi, J., 2019. Joint training framework for text-to-speech and voice conversion using multi-source tacotron and wavenet. In: Proc. Interspeech 2019, pp. 1298–1302. · DOI
  77. Zhang, 2021, Transfer learning from speech synthesis to voice conversion with non-parallel training data, IEEE/ACM Trans. Audio Speech Lang. Process., 29, 1290, 10.1109/TASLP.2021.3066047 · DOI
  78. Zhou, X., Ling, Z.-H., King, S., 2020. The Blizzard Challenge 2020. In: Proc. Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020, pp. 1–18. · DOI
  79. Zhou, K., Sisman, B., Li, H., 2020. Transforming spectrum and prosody for emotional voice conversion with non-parallel training data. In: Proc. Odyssey 2020 the Speaker and Language Recognition Workshop, pp. 230–237. · DOI
  80. Zhou, K., Sisman, B., Zhang, M., Li, H., 2020. Converting anyone’s emotion: Towards speaker-independent emotional voice conversion. In: Proc. Interspeech 2020, pp. 3416–3420. · DOI

Cited by

See all 181

Metadata

Authors
Zhou, K., Sisman, B., Liu, R., & Li, H.
Venue
Speech Communication
Publisher
Elsevier BV
Published
2022-02
Volume
137
Pages
1-18
ISSN
0167-6393 (ISSN-L)
OpenAlex
W4205742757
Cited by
181
References
18
FWCI
8.82
Type
journal article
Language
en
Open access
hybrid
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Emotional voice conversion: Theory, databases and ESD. https://app.cochat.ai/papers/10.1016/j.specom.2021.11.006
The paper
Zhou, K., Sisman, B., Liu, R., & Li, H. (2022). Emotional voice conversion: Theory, databases and ESD. Speech Communication. https://doi.org/10.1016/j.specom.2021.11.006

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.