Transfer learning emotion manifestation across music and speech
Abstract
In this article, we focus on time-continuous predictions of emotion in music and speech, and the transfer of learning from one domain to the other. First, we compare the use of Recurrent Neural Networks (RNN) with standard hidden units (Simple Recurrent Network - SRN) and Long-Short Term Memory (LSTM) blocks for intra-domain acoustic emotion recognition. We show that LSTM networks outperform SRN, and we explain, in average, 74%/59% (music) and 42%/29% (speech) of the variance in Arousal/Valence. Next, we evaluate whether cross-domain predictions of emotion are a viable option for acoustic emotion recognition, and we test the use of Transfer Learning (TL) for feature space adaptation. In average, our models are able to explain 70%/43% (music) and 28%/ll% (speech) of the variance in Arousal/Valence. Overall, results indicate a good cross-domain generalization performance, particularly for the model trained on speech and tested on music without pre-encoding of the input features. To our best knowledge, this is the first demonstration of cross-modal time-continuous predictions of emotion in the acoustic domain.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Knowledge transfer from domains like music and visual object recognition is also possible.
“Knowledge transfer from domains like music [141] and visual object recognition [132], [85] is also possible.”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [141]
Cites
Russell, J. A. (1980) · Journal of Personality and Social Psychology · doi:10.1037/h0077714 · Cited by 16,334 · cited 1× · see citing works
“Emotions were quantified using a two-dimensional model of affect consisting of Arousal and Valence dimensions ([22]).”
MethodRepresentation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · see citing works
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,882 · see citing works
Learning to Forget: Continual Prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 5,869 · see citing works
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120 · see citing works
Communication of emotions in vocal expression and music performance: Different channels, same code?
Juslin, P. N., & Laukka, P. (2003) · Psychological Bulletin · doi:10.1037/0033-2909.129.5.770 · Cited by 1,828 · see citing works
Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach
Glorot, X., Bordes, A., & Bengio, Y. (2011) · Cited by 1,569 · see citing works
Handbook of music and emotion: theory, research, applications
(2010) · Choice Reviews Online · doi:10.5860/choice.48-0770 · Cited by 1,023 · see citing works
Praat script to detect syllable nuclei and measure speech rate automatically
de Jong, N. H., & Wempe, T. G. (2009) · Behavior Research Methods · doi:10.3758/brm.41.2.385 · Cited by 599 · see citing works
Handbook of Music and Emotion: Theory, Research, Applications
Juslin, P. N. (2010) · doi:10.1093/acprof:oso/9780199230143.001.0001 · Cited by 516 · see citing works
Cross-Corpus Acoustic Emotion Recognition: Variances and Strategies
Schuller, B. W., Vlasenko, B., Eyben, F., Wöllmer, M., Stuhlsatz, A., Wendemuth, A., & Rigoll, G. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.8 · Cited by 386 · see citing works
Self-citationSparse Autoencoder-Based Feature Transfer Learning for Speech Emotion Recognition
Deng, J., Zhang, Z., Marchi, E., & Schuller, B. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.90 · Cited by 366 · see citing works
Self-citationModeling Perceived Emotion With Continuous Musical Features
Schubert, E. (2004) · Music Perception An Interdisciplinary Journal · doi:10.1525/mp.2004.21.4.561 · Cited by 345 · see citing works
Wöllmer, M., Eyben, F., Reiter, S. A., Schuller, B. W., Cox, C., Douglas‐Cowie, E., & Cowie, R. (2008) · Interspeech (USB) · doi:10.21437/interspeech.2008-192 · Cited by 329 · see citing works
Self-citationDouglas‐Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O. M., McRorie, M., Martin, J., Devillers, L. Y., Abrilian, S., Batliner, A. M., et al. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74889-2_43 · Cited by 323 · see citing works
The Role of Structure in the Musical Expression of Emotions
Gabrielsson, A., & Lindström, E. (2010) · doi:10.1093/acprof:oso/9780199230143.003.0014 · Cited by 306 · see citing works
On the Acoustics of Emotion in Audio: What Speech, Music, and Sound have in Common
Weninger, F. J., Eyben, F., Schuller, B. W., Mortillaro, M., & Scherer, K. R. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00292 · Cited by 277 · see citing works
Self-citationThe prosogram: semi-automatic transcription of prosody based on a tonal perception model
Mertens, P. (2004) · Speech prosody · doi:10.21437/speechprosody.2004-127 · Cited by 234 · see citing works
Schuller, B. W., Valstar, M., Eyben, F., Cowie, R., & Pantić, M. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388776 · Cited by 225 · see citing works
Self-citationOnline Driver Distraction Detection Using Long Short-Term Memory
Wöllmer, M., Blaschke, C., Schindl, T., Schuller, B. W., Färber, B., Mayer, S., & Trefflich, B. (2011) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2011.2119483 · Cited by 177 · see citing works
Self-citationPsychoacoustic cues to emotion in speech prosody and music
Coutinho, E., & Dibben, N. (2012) · Cognition & Emotion · doi:10.1080/02699931.2012.732559 · Cited by 165 · see citing works
Self-citationWöllmer, M., Schuller, B. W., Eyben, F., & Rigoll, G. (2010) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2010.2057200 · Cited by 164 · see citing works
Self-citationEvaluation of the Audio Beat Tracking System BeatRoot
Dixon, S. (2007) · Journal of New Music Research · doi:10.1080/09298210701653310 · Cited by 157 · see citing works
Coutinho, E., & Cangelosi, A. (2011) · Emotion · doi:10.1037/a0024700 · Cited by 153 · see citing works
Self-citationDynamic Loudness Model (DLM) for Normal and Hearing-Impaired Listeners
Chalupper, J., & Fastl, H. (2002) · Cited by 129 · see citing works
Experiential and Cognitive Changes Following Seven Minutes Exposure to Music and Speech
Ilie, G., & Thompson, W. F. (2011) · Music Perception An Interdisciplinary Journal · doi:10.1525/mp.2011.28.3.247 · Cited by 73 · see citing works
Ein Berechnungsverfahren der Rauhigkeit
Aures, W. (1985) · Acustica · Cited by 63 · see citing works
The Use of Spatio-Temporal Connectionist Models in Psychological Studies of Musical Emotions
Coutinho, E., & Cangelosi, A. (2009) · Music Perception An Interdisciplinary Journal · doi:10.1525/mp.2009.27.1.1 · Cited by 49 · see citing works
Self-citationTandem decoding of children's speech for keyword detection in a child-robot interaction scenario
Wöllmer, M., Schuller, B., Batliner, A., Steidl, S., & Seppi, D. (2011) · ACM Transactions on Speech and Language Processing · doi:10.1145/1998384.1998386 · Cited by 18 · see citing works
Self-citationComputational and Psycho-Physiological Investigations of Musical Emotions
Coutinho, E. (2008) · PEARL (University of Plymouth) · doi:10.17638/03002888 · Cited by 7 · see citing works
Self-citationA toolbox for automatic transcription of polyphonic music
Dittmar, C., Dressler, K., & Rosenbauer, K. (2007) · Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) · Cited by 7 · see citing works
Music, Speech and Emotion: Psycho-physiological and Computational Investigations
Coutinho, E., & Dibben, N. (2010) · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
- Authors
- Coutinho, E., Deng, J., & Schuller, B.
- Affiliations
- University of Liverpool (GB)
- Technical University of Munich (DE)
- Venue
- 2014 International Joint Conference on Neural Networks (IJCNN)
- Publisher
- IEEE
- Published
- 2014-07
- Pages
- 3592-3598
- ISSN
- 2161-4407 (ISSN-L)
- OpenAlex
- W1992182263
- Cited by
- 36
- References
- 33
- Citations by year
- 2024: 2 · 2022: 2 · 2021: 4 · 2019: 4 · 2018: 5 · 2017: 9 · 2016: 2 · 2015: 8
- FWCI
- 10.37
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Joint Conference on Neural Networks (IJCNN) (publishedVersion)
- OPUS (Augsburg University) (submittedVersion)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- acoustic emotion recognition, transfer learning, music and emotion, speech emotion, long short-term memory, Elman neural network, valence and arousal
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
