Progressive Neural Networks for Transfer Learning in Emotion Recognition
Open accessAbstract
Many paralinguistic tasks are closely related and thus representations learned in one domain can be leveraged for another. In this paper, we investigate how knowledge can be transferred between three paralinguistic tasks: speaker, emotion, and gender recognition. Further, we extend this problem to cross-dataset tasks, asking how knowledge captured in one emotion dataset can be transferred to another. We focus on progressive neural networks and compare these networks to the conventional deep learning method of pre-training and fine-tuning. Progressive neural networks provide a way to transfer knowledge and avoid the forgetting effect present when pre-training neural networks on different tasks. Our experiments demonstrate that: (1) emotion recognition can benefit from using representations originally learned for different paralinguistic tasks and (2) transfer learning can effectively leverage additional datasets to improve the performance of emotion recognition systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multiple studies have shown that transfer learning can improve model accuracy by leveraging additional sources of related knowledge.
“Multiple studies have shown that transfer learning can improve model accuracy by leveraging additional sources of related knowledge (e.g., from other paralinguistic tasks [136], various standard databases [137], and different affect representations [135]).”
From Deep learning for human affect recognition: Insights and new developments (2021) · cites this paper as [136]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Cascade-Correlation learning architecture
Fahlman, S. E., & Lebiere, C. J. (2018) · doi:10.1184/r1/6610403 · Cited by 2,391 · see citing works
The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for Voice Research and Affective Computing
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg, J., Andre, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2457417 · Cited by 1,818 · see citing works
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1606.04671 · Cited by 1,307 · see citing works
The INTERSPEECH 2009 emotion challenge
Schuller, B., Steidl, S., & Batliner, A. (2009) · Interspeech 2009 · doi:10.21437/interspeech.2009-103 · Cited by 938 · see citing works
Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning
Ng, H.-W., Nguyen, V. D., Vonikakis, V., & Winkler, S. (2015) · Proceedings of the 2015 ACM on International Conference on Multimodal Interaction · doi:10.1145/2818346.2830593 · Cited by 657 · cited 2× · see citing works
“[10] studied transfer learning in the context of facial expression recognition.”
BackgroundMSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception
Busso, C., Parthasarathy, S., Burmania, A., AbdelWahab, M., Sadoughi, N., & Provost, E. M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2515617 · Cited by 391 · see citing works
Self-citationSim-to-Real Robot Learning from Pixels with Progressive Nets
Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., & Hadsell, R. (2016) · arXiv · doi:10.48550/arxiv.1610.04286 · Cited by 385 · see citing works
Sparse Autoencoder-Based Feature Transfer Learning for Speech Emotion Recognition
Deng, J., Zhang, Z., Marchi, E., & Schuller, B. (2013) · 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction · doi:10.1109/acii.2013.90 · Cited by 366 · see citing works
A Multi-Task Learning Framework for Emotion Recognition Using 2D Continuous Space
Xia, R., & Liu, Y. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2512598 · Cited by 182 · see citing works
Improving Automatic Emotion Recognition from Speech via Gender Differentiaion
Vogt, T., & André, E. (2006) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/32rkhvcgrj6f · Cited by 137 · see citing works
How Transferable are Neural Networks in NLP Applications?Preprint
Mou, L., Zhao, M., Yan, R., Li, G., Xu, Y. H., Zhang, L., & Jin, Z. G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06111 · Cited by 83 · see citing works
Cross-corpus acoustic emotion recognition from singing and speaking: A multi-task learning approach
Zhang, B., Provost, E. M., & Essi, G. (2016) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2016.7472790 · Cited by 71 · see citing works
Self-citationCross-lingual transfer learning during supervised training in low resource scenarios
Das, A. K., & Hasegawa‐Johnson, M. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-700 · Cited by 35 · see citing works
Emotion spotting: discovering regions of evidence in audio-visual emotion expressions
Kim, Y., & Provost, E. M. (2016) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2993148.2993151 · Cited by 34 · see citing works
Self-citationLotfian, R., & Busso, C. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-1052 · Cited by 27 · see citing works
Comparison of Gender- and Speaker-adaptive Emotion Recognition
Yuryevich, S. M., Ultes, S., & Schmitt, A. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4wxqwzrhi4t2 · Cited by 10 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- D. Ververidis and C. Kotropoulos, "Automatic speech classifica- tion to five emotional states based on gender information," in Sig- nal Processing Conference, 2004 12th European. IEEE, 2004, pp. 341-344.
- C. M. Lee and S. S. Narayanan, "Toward detecting emotions in spoken dialogs," IEEE transactions on speech and audio process- ing, vol. 13, no. 2, pp. 293-303, 2005.
- B. Schuller, S. Steidl, A. Batliner, F. Burkhardt, L. Devillers, C. MüLler, and S. Narayanan, "Paralinguistics in speech and languagestate-of-the-art and the challenge," Computer Speech & Language, vol. 27, no. 1, pp. 4-39, 2013.
- C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, "Iemocap: Interactive emotional dyadic motion capture database," Language resources and evaluation, vol. 42, no. 4, p. 335, 2008.
- R. R. Bouckaert and E. Frank, "Evaluating the replicability of sig- nificance tests for comparing learning algorithms," in Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, 2004, pp. 3-12.
Cited by
Sorted by citations
Deep learning for human affect recognition: Insights and new developments
Rouast, P. V., Adam, M. T. P., & Chiong, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2018.2890471 · Cited by 243 · see citing works
- Authors
- Gideon, J., Khorram, S., Aldeneh, Z., Dimitriadis, D., & Provost, E. M.
- Affiliations
- University of Michigan (US)
- Venue
- Interspeech 2017
- Publisher
- ISCA
- Published
- 2017-08-20
- Pages
- 1098-1102
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2625435062
- Cited by
- 129
- References
- 16
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 8 · 2023: 5 · 2022: 19 · 2021: 19 · 2020: 21 · 2019: 32 · 2018: 13 · 2017: 1
- FWCI
- 21.04
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- progressive neural networks, transfer learning, emotion recognition, paralinguistic features, speaker recognition, gender classification, cross-dataset transfer, pre-training and fine-tuning, knowledge forgetting
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
