CoChat
Sign Up
Cited by 512Sentiment Analysis and Opinion MiningMultimodal Machine Learning ApplicationsEmotion and Mood Recognition

Found in Translation: Learning Robust Joint Representations by Cyclic Translations between Modalities

Open access
Pham, Hai, Liang, Paul Pu, Manzini, Thomas, Morency, Louis-Philippe, Póczos, Barnabás·2019·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v33i01.33016892
Open this paper in CoChat — free account

Abstract

Multimodal sentiment analysis is a core research area that studies speaker sentiment expressed from the language, visual, and acoustic modalities. The central challenge in multimodal learning involves inferring joint representations that can process and relate information from these modalities. However, existing work learns joint representations by requiring all modalities as input and as a result, the learned representations may be sensitive to noisy or missing modalities at test time. With the recent success of sequence to sequence (Seq2Seq) models in machine translation, there is an opportunity to explore new ways of learning joint representations that may not require all input modalities at test time. In this paper, we propose a method to learn robust joint representations by translating between modalities. Our method is based on the key insight that translation from a source to a target modality provides a method of learning joint representations using only the source modality as input. We augment modality translations with a cycle consistency loss to ensure that our joint representations retain maximal information from all modalities. Once our translation model is trained with paired multimodal data, we only need data from the source modality at test time for final sentiment prediction. This ensures that our model remains robust from perturbations or missing information in the other modalities. We train our model with a coupled translationprediction objective and it achieves new state-of-the-art results on multimodal sentiment analysis datasets: CMU-MOSI, ICTMMMO, and YouTube. Additional experiments show that our model learns increasingly discriminative joint representations with more input modalities while maintaining robustness to missing or perturbed modalities.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v33i01.33016892

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Current benchmarks typically focus on performance without quantifying the potential drawbacks involved with increased time and space complexity and the risk of decreased robustness from imperfect modalities.

    “Furthermore, current benchmarks typically focus on performance without quantifying the potential drawbacks involved with increased time and space complexity [148], and the risk of decreased robustness from imperfect modalities [101, 123].”

    From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [123]

    The Introduction (p. 1) says existing joint-representation methods require all modalities at test time, making them sensitive to noisy or missing modalities. This supports the robustness risk attributed to this source, but the paper does not establish that current benchmarks typically focus on performance without quantifying that risk. The time-and-space-complexity point is attached to a different citation. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Real-world multimodal data is often imperfect as a result of missing entries, noise corruption, or missing modalities entirely, which calls for robust models that can still make accurate predictions despite only having access to noisy and missing signals.

    “Real-world multimodal data is often imperfect as a result of missing entries, noise corruption, or missing modalities entirely, which calls for robust models that can still make accurate predictions despite only having access to noisy and missing signals [101, 123].”

    From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [123]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • MCTN uses cycle-consistent translation to predict the noisy/missing modality from present ones.

    “MULTIZOO includes MCTN [123] which uses cycle-consistent translation to predict the noisy/missing modality from present ones.”

    From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [123]

    The Multimodal Cyclic Translation Network section (p. 3) confirms that MCTN uses cycle-consistent translation between modalities. But the Introduction (p. 2) says that, at test time, it uses the available source modality to infer a joint representation and label—not to predict the noisy or missing modality. The Related Work section (p. 2) distinguishes MCTN from methods that infer missing modalities before prediction. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Sequential translation-based models require translation and generation of videos, audios, and text, which are difficult to train especially with limited training samples.

    “Additionally, the sequential translation-based models require translation and generation of videos, audios, and text, which are difficult to train especially with limited training samples (Li et al., 2018; Pham et al., 2019).”

    From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Pham et al., 2019

    The Introduction (p. 2) says that translating and generating video, audio, and text is difficult, and describes MCTN as translating between modalities. It does not say this is especially difficult with limited training samples, or that every sequential translation-based model must generate all three modalities. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Existing multimodal fusion models trained on full-modality samples usually fail when partial modalities are missing.

    “Existing multimodal fusion models trained on full-modality samples usually fail when partial modalities are missing (Aguilar et al., 2019; Pham et al., 2019; Cai et al., 2018; Parthasarathy and Sundaram, 2020).”

    From Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities (2021) · cites this paper as Pham et al., 2019

    The Abstract (p. 1) says existing approaches require all modalities as input and that their representations may be sensitive to missing modalities at test time. The Introduction (p. 1) similarly says prior approaches require all modalities at test time. Neither passage establishes that they usually fail, so the claim overstates the paper's qualified point. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 64

  1. [Alku, B¨ ackstr¨ om, and Vilkman 2002] Alku, P.; B¨ ackstr¨ om, T.; and Vilkman, E. 2002. Normalized amplitude quotient for parametrization of the glottal flow. the Journal of the Acoustical Society of America 112(2):701-710.
  2. [Alku, Strik, and Vilkman 1997] Alku, P.; Strik, H.; and Vilkman, E. 1997. Parabolic spectral parameter - a new method for quantification of the glottal flow. Speech Communication 22(1):67-79.
  3. [Alku 1992] Alku, P. 1992. Glottal wave analysis with pitch synchronous iterative adaptive inverse filtering. Speech communication 11(2-3):109-118.
  4. [Bahdanau, Cho, and Bengio 2014] Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . · arXiv
  5. [Baltrusaitis, Ahuja, and Morency 2017] Baltrusaitis, T.; Ahuja, C.; and Morency, L. 2017. Multimodal machine learning: A survey and taxonomy. CoRR abs/1705.09406. · arXiv
  6. [Cai et al. 2018] Cai, L.; Wang, Z.; Gao, H.; Shen, D.; and Ji, S. 2018. Deep adversarial learning for multi-modality missing data completion. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , KDD '18, 1158-1166. New York, NY, USA: ACM.
  7. [Cheng et al. 2017] Cheng, Y.; Huang, F.; Zhou, L.; Jin, C.; Zhang, Y.; and Zhang, T. 2017. A hierarchical multimodal attentionbased neural network for image captioning. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR '17. New York, NY, USA: ACM.
  8. [Childers and Lee 1991] Childers, D. G., and Lee, C. 1991. Vocal quality factors: Analysis, synthesis, and perception. the Journal of the Acoustical Society of America 90(5):2394-2410.
  9. [Collell, Zhang, and Moens 2017] Collell, G.; Zhang, T.; and Moens, M.-F. 2017. Imagined visual representations as multimodal embeddings. AAAI .
  10. [Cortes and Vapnik 1995] Cortes, C., and Vapnik, V. 1995. Supportvector networks. Machine learning 20(3):273-297.
  11. [Degottex et al. 2014] Degottex, G.; Kane, J.; Drugman, T.; Raitio, T.; and Scherer, S. 2014. Covarep - a collaborative voice analysis repository for speech technologies. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on . IEEE.
  12. [Donahue, Kr¨ ahenb¨ uhl, and Darrell 2016] Donahue, J.; Kr¨ ahenb¨ uhl, P.; and Darrell, T. 2016. Adversarial feature learning. arXiv preprint arXiv:1605.09782 . · arXiv
  13. [Drugman and Alwan 2011] Drugman, T., and Alwan, A. 2011. Joint robust voicing detection and pitch estimation based on residual harmonics. In Interspeech , 1973-1976.
  14. [Drugman et al. 2012] Drugman, T.; Thomas, M.; Gudnason, J.; Naylor, P.; and Dutoit, T. 2012. Detection of glottal closure instants from speech signals: A quantitative review. IEEE Transactions on Audio, Speech, and Language Processing 20(3):994-1006.
  15. [Goodfellow et al. 2014] Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative adversarial nets. In NIPS .
  16. [Graves, r. Mohamed, and Hinton 2013] Graves, A.; r. Mohamed, A.; and Hinton, G. 2013. Speech recognition with deep recurrent neural networks. In 2013 IEEE ICASSP , 6645-6649.
  17. [Hill, Reichart, and Korhonen 2014] Hill, F.; Reichart, R.; and Korhonen, A. 2014. Multi-modal models for concrete and abstract concept meaning. TACL .
  18. [Hochreiter and Schmidhuber 1997] Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory. Neural Comput. 9(8):17351780.
  19. [Kane and Gobl 2013] Kane, J., and Gobl, C. 2013. Wavelet maxima dispersion for breathy to tense voice discrimination. IEEE Transactions on Audio, Speech, and Language Processing 21(6):1170-1179.
  20. [Kaushik, Sangwan, and Hansen 2013] Kaushik, L.; Sangwan, A.; and Hansen, J. H. 2013. Sentiment extraction from natural audio streams. In ICASSP . IEEE.
  21. [Kingma et al. 2014] Kingma, D. P.; Mohamed, S.; Rezende, D. J.; and Welling, M. 2014. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems , 3581-3589.
  22. [Kiros, Salakhutdinov, and Zemel 2014] Kiros, R.; Salakhutdinov, R.; and Zemel, R. S. 2014. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 . · arXiv
  23. [Lample et al. 2018] Lample, G.; Ott, M.; Conneau, A.; Denoyer, L.; and Ranzato, M. 2018. Phrase-based & neural unsupervised machine translation. CoRR abs/1804.07755. · arXiv
  24. [Lazaridou, Pham, and Baroni 2015] Lazaridou, A.; Pham, N. T.; and Baroni, M. 2015. Combining language and vision with a multimodal skip-gram model. arXiv preprint arXiv:1501.02598 . · arXiv
  25. [Li et al. 2017a] Li, C.; Xu, K.; Zhu, J.; and Zhang, B. 2017a. Triple generative adversarial nets. arXiv preprint arXiv:1703.02291 . · arXiv
  26. [Li et al. 2017b] Li, Y.; Min, M. R.; Shen, D.; Carlson, D. E.; and Carin, L. 2017b. Video generation from text. CoRR abs/1710.00421. · arXiv
  27. [Liang et al. 2018] Liang, P. P.; Liu, Z.; Zadeh, A.; and Morency, L.-P. 2018. Multimodal language analysis with recurrent multistage fusion. In Proceedings of the Conference on Empirical Methods in Natural Language Processing , EMNLP.
  28. [Liang, Zadeh, and Morency 2018] Liang, P. P.; Zadeh, A.; and Morency, L.-P. 2018. Multimodal local-global ranking fusion for emotion recognition. In Proceedings of the 20th ACM International Conference on Multimodal Interaction , ICMI.
  29. [Liu et al. 2018] Liu, Z.; Shen, Y .; Lakshminarasimhan, V. B.; Liang, P. P.; Bagher Zadeh, A.; and Morency, L.-P. 2018. Efficient low-rank multimodal fusion with modality-specific factors. In ACL .
  30. [Mario Christoudias et al. 2010] Mario Christoudias, C.; Urtasun, R.; Salzmann, M.; and Darrell, T. 2010. Learning to recognize objects from unseen modalities. In Daniilidis, K.; Maragos, P.; and Paragios, N., eds., Computer Vision - ECCV 2010 .
  31. [Morency, Mihalcea, and Doshi 2011] Morency, L.-P.; Mihalcea, R.; and Doshi, P. 2011. Towards multimodal sentiment analysis: Harvesting opinions from the web. In Proceedings of the 13th international conference on multimodal interfaces , 169-176. ACM.
  32. [Morency, Quattoni, and Darrell 2007] Morency, L.-P.; Quattoni, A.; and Darrell, T. 2007. Latent-dynamic discriminative models for continuous gesture recognition. In Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on , 1-8. IEEE.
  33. [Ngiam et al. 2011] Ngiam, J.; Khosla, A.; Kim, M.; Nam, J.; Lee, H.; and Ng, A. Y. 2011. Multimodal deep learning. In Proceedings of the 28th international conference on machine learning (ICML11) .
  34. [Nojavanasghari et al. 2016] Nojavanasghari, B.; Gopinath, D.; Koushik, J.; Baltruˇ saitis, T.; and Morency, L.-P. 2016. Deep multimodal fusion for persuasiveness prediction. In Proceedings of the 18th ACM International Conference on Multimodal Interaction , ICMI 2016.
  35. [Pandey and Dukkipati 2017] Pandey, G., and Dukkipati, A. 2017. Variational methods for conditional multimodal deep learning. In Neural Networks (IJCNN), 2017 International Joint Conference on , 308-315. IEEE.
  36. [Pang, Lee, and others 2008] Pang, B.; Lee, L.; et al. 2008. Opinion mining and sentiment analysis. Foundations and Trends® in Information Retrieval 2(1-2):1-135.
  37. [Pang, Lee, and Vaithyanathan 2002] Pang, B.; Lee, L.; and Vaithyanathan, S. 2002. Thumbs up?: Sentiment classification using machine learning techniques. EMNLP.
  38. [Pennington, Socher, and Manning 2014] Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In EMNLP , volume 14, 1532-1543.
  39. [Pham et al. 2018] Pham, H.; Manzini, T.; Liang, P. P.; and Poczos, B. 2018. Seq2seq2sentiment: Multimodal sequence to sequence models for sentiment analysis. In Proceedings of Grand Challenge and Workshop on Human Multimodal Language . ACL.
  40. [Poria et al. 2017] Poria, S.; Cambria, E.; Hazarika, D.; Majumder, N.; Zadeh, A.; and Morency, L.-P. 2017. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics .
  41. [Poria, Cambria, and Gelbukh 2015] Poria, S.; Cambria, E.; and Gelbukh, A. 2015. Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In EMNLP .
  42. [Prabhumoye et al. 2018] Prabhumoye, S.; Tsvetkov, Y.; Salakhutdinov, R.; and Black, A. W. 2018. Style transfer through backtranslation. CoRR abs/1804.09000. · arXiv
  43. [Quattoni et al. 2007] Quattoni, A.; Wang, S.; Morency, L.-P.; Collins, M.; and Darrell, T. 2007. Hidden conditional random fields. IEEE Trans. Pattern Anal. Mach. Intell. 29(10):1848-1852.
  44. [Rajagopalan et al. 2016] Rajagopalan, S. S.; Morency, L.-P.; Baltruˇ saitis, T.; and Roland, G. 2016. Extending long short-term memory for multi-view structured learning. In European Conference on Computer Vision .
  45. [Schuster and Paliwal 1997] Schuster, M., and Paliwal, K. 1997. Bidirectional recurrent neural networks. Trans. Sig. Proc. 45(11):2673-2681.
  46. [Shaffer 2018] Shaffer, I. R. 2018. Exploring the performance of facial expression recognition technologies on deaf adults and their children. In SIGACCESS Conference on Computers and Accessibility .
  47. [Socher et al. 2013] Socher, R.; Perelygin, A.; Wu, J. Y.; Chuang, J.; Manning, C. D.; Ng, A. Y.; Potts, C.; et al. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP) , volume 1631, 1642. Citeseer.
  48. [Sohn, Shang, and Lee 2014] Sohn, K.; Shang, W.; and Lee, H. 2014. Improved multimodal deep learning with variation of information. In NIPS .
  49. [Song, Morency, and Davis 2012] Song, Y.; Morency, L.-P.; and Davis, R. 2012. Multi-view latent variable discriminative models for action recognition. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on , 2120-2127. IEEE.
  50. [Song, Morency, and Davis 2013] Song, Y.; Morency, L.-P.; and Davis, R. 2013. Action recognition by hierarchical sequence summarization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 3562-3569.
  51. [Srivastava and Salakhutdinov 2014] Srivastava, N., and Salakhutdinov, R. 2014. Multimodal learning with deep boltzmann machines. JMLR 15.
  52. [Sutskever, Vinyals, and Le 2014] Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems , 3104-3112.
  53. [Tran et al. 2017] Tran, L.; Liu, X.; Zhou, J.; and Jin, R. 2017. Missing modalities imputation via cascaded residual autoencoder. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , 4971-4980.
  54. [Tsai et al. 2018] Tsai, Y.-H. H.; Liang, P. P.; Zadeh, A.; Morency, L.-P.; and Salakhutdinov, R. 2018. Learning factorized multimodal representations. arXiv preprint arXiv:1806.06176 . · arXiv
  55. [Tu et al. 2016] Tu, Z.; Liu, Y.; Shang, L.; Liu, X.; and Li, H. 2016. Neural machine translation with reconstruction. CoRR abs/1611.01874. · arXiv
  56. [van der Maaten and Hinton 2008] van der Maaten, L., and Hinton, G. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9:2579-2605.
  57. [Wang et al. 2016] Wang, H.; Meghawat, A.; Morency, L.-P.; and Xing, E. P. 2016. Select-additive learning: Improving crossindividual generalization in multimodal sentiment analysis. arXiv preprint arXiv:1609.05244 . · arXiv
  58. [W¨ ollmer et al. 2013] W¨ ollmer, M.; Weninger, F.; Knaup, T.; Schuller, B.; Sun, C.; Sagae, K.; and Morency, L.-P. 2013. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems 28(3):46-53.
  59. [Yuan and Liberman 2008] Yuan, J., and Liberman, M. 2008. Speaker identification on the scotus corpus. Journal of the Acoustical Society of America 123(5):3878.
  60. [Zadeh et al. 2016] Zadeh, A.; Zellers, R.; Pincus, E.; and Morency, L.-P. 2016. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems 31(6):82-88.
  61. [Zadeh et al. 2017] Zadeh, A.; Chen, M.; Poria, S.; Cambria, E.; and Morency, L.-P. 2017. Tensor fusion network for multimodal sentiment analysis. In EMNLP , 1114-1125.
  62. [Zadeh et al. 2018] Zadeh, A.; Liang, P. P.; Mazumder, N.; Poria, S.; Cambria, E.; and Morency, L.-P. 2018. Memory fusion network for multi-view sequential learning. arXiv preprint arXiv:1802.00927 . · arXiv
  63. [Zhu et al. 2006] Zhu, Q.; Yeh, M.-C.; Cheng, K.-T.; and Avidan, S. 2006. Fast human detection using a cascade of histograms of oriented gradients. In Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on , volume 2, 1491-1498. IEEE.
  64. [Zhu et al. 2017] Zhu, J.; Park, T.; Isola, P.; and Efros, A. A. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. CoRR abs/1703.10593. · arXiv

Cited by

Sorted by citations

Authors
Pham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B.
Affiliations
Carnegie Mellon University (US)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2019-07-17
Volume
33
Issue
01
Pages
6892-6899
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v33i01.33016892
OpenAlex
W2962931510
Cited by
512
References
1
Citations by year
2026: 64 · 2025: 93 · 2024: 96 · 2023: 94 · 2022: 60 · 2021: 62 · 2020: 35 · 2019: 8
FWCI
15.78
Type
conference paper
Language
en
Open access
Licence
www.aaai.org/ (unspecified)
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1563887)
  • Defense Advanced Research Projects Agency (D3M program)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
Keywords
multimodal sentiment analysis, joint representation learning, modality translation, cycle consistency loss, sequence-to-sequence translation, multimodal fusion

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review