CoChat
Sign Up
Cited by 22Domain Adaptation and Few-Shot LearningMultimodal Machine Learning ApplicationsTopic Modeling

Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignment

Open access
Liang, Paul Pu, Wu, Peter, Ziyin, Liu, Morency, Louis-Philippe, Salakhutdinov, Ruslan·2021·Proceedings of the 29th ACM International Conference on Multimedia
DOI 10.1145/3474085.3475247
Open this paper in CoChat — free accountOpen PDF

Abstract

How can we generalize to a new prediction task at test time when it also uses a new modality as input? More importantly, how can we do this with as little annotated data as possible? This problem of cross-modal generalization is a new research milestone with concrete impact on real-world applications. For example, can an AI system start understanding spoken language from mostly written text? Or can it learn the visual steps of a new recipe from only text descriptions? In this work, we formalize cross-modal generalization as a learning paradigm to train a model that can (1) quickly perform new tasks (from new domains) while (2) being originally trained on a different input modality. Such a learning paradigm is crucial for generalization to low-resource modalities such as spoken speech in rare languages while utilizing a different high-resource modality such as text. One key technical challenge that makes it different from other learning paradigms such as meta-learning and domain adaptation is the presence of different source and target modalities which will require different encoders. We propose an effective solution based on meta-alignment, a novel method to align representation spaces using strongly and weakly paired cross-modal data while ensuring quick generalization to new tasks across different modalities. This approach uses key ideas from cross-modal learning and meta-learning, and presents strong results on the cross-modal generalization problem. We benchmark several approaches on 3 real-world classification tasks: few-shot recipe classification from text to images of recipes, object classification from images to audio of objects, and language classification from text to spoken speech across 100 languages spanning many rare languages. Our results demonstrate strong performance even when the new target modality has only a few (1-10) labeled samples and in the presence of noisy labels, a scenario particularly prevalent in low-resource modalities.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3474085.3475247

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There has been extensive research in defining objective functions to tackle the challenge of multimodal alignment: capturing a representation space where semantically similar concepts from different modalities are close together.

    “There has been extensive research in defining objective functions to tackle the challenge of multimodal alignment: capturing a representation space where semantically similar concepts from different modalities are close together.”

    From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [104]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 62

  1. Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Apostol (Paul) Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. 2016. YouTube-8M: A Large-Scale Video Classication Benchmark. In arXiv:1609.08675. https://arxiv.org/pdf/1609.08675v1.pdf · arXiv
  2. Antreas Antoniou, Amos Storkey, and Harrison Edwards. 2017. Data augmenta- tion generative adversarial networks. arXiv preprint arXiv:1711.04340 (2017). · arXiv
  3. Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. 2019. Eective- ness of self-supervised pre-training for speech recognition. arXiv preprint arXiv:1911.03912 (2019). · arXiv
  4. Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multi- modal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence 41, 2 (2018), 423-443.
  5. A. W. Black. 2019. CMU Wilderness Multilingual Speech Dataset. In ICASSP 2019 -2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 5971-5975.
  6. Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Qiang Yang. 2017. Transitive hashing network for heterogeneous multimedia retrieval. In Proceedings of the Thirty-First AAAI Conference on Articial Intelligence. 81-87.
  7. Rich Caruana. 1997. Multitask learning. Machine learning 28, 1 (1997), 41-75.
  8. Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia- Bin Huang. 2019. A closer look at few-shot classication. arXiv preprint arXiv:1904.04232 (2019). · arXiv
  9. François Chollet. 2019. On the Measure of Intelligence. CoRR abs/1911.01547 (2019). arXiv:1911.01547 http://arxiv.org/abs/1911.01547 · arXiv
  10. Safa Cicek and Stefano Soatto. 2019. Unsupervised domain adaptation via regu- larized conditional alignment. In Proceedings of the IEEE International Conference on Computer Vision. 1416-1425.
  11. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248-255.
  12. Nanqing Dong and Eric P Xing. 2018. Domain adaption in one-shot learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 573-588.
  13. Jared Dunnmon, Alexander Ratner, Nishith Khandwala, Khaled Saab, Matthew Markert, Hersh Sagreiya, Roger E. Goldman, Christopher Lee-Messer, Matthew P. Lungren, Daniel L. Rubin, and Christopher Ré. 2019. Cross-Modal Data Program- ming Enables Rapid Medical Machine Learning. CoRR abs/1903.11101 (2019). arXiv:1903.11101 http://arxiv.org/abs/1903.11101 · arXiv
  14. Chris Dyer. 2014. Notes on noise contrastive estimation and negative sampling. arXiv preprint arXiv:1410.8251 (2014). · arXiv
  15. Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612 (2017). · arXiv
  16. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta- learning for fast adaptation of deep networks. In Proceedings of the 34th Interna- tional Conference on Machine Learning-Volume 70. JMLR. org, 1126-1135.
  17. Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Je Dean, Marc'Aurelio Ranzato, and Tomas Mikolov. 2013. Devise: A deep visual-semantic embedding model. In Advances in neural information processing systems. 2121-2129.
  18. Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 776-780.
  19. Marzyeh Ghassemi, Tristan Naumann, Peter Schulam, Andrew L. Beam, and Rajesh Ranganath. 2018. Opportunities in Machine Learning for Healthcare. CoRR abs/1806.00388 (2018). arXiv:1806.00388 http://arxiv.org/abs/1806.00388 · arXiv
  20. Edouard Grave, Armand Joulin, and Quentin Berthet. 2018. Unsupervised Align- ment of Embeddings with Wasserstein Procrustes. CoRR abs/1805.11222 (2018). arXiv:1805.11222 http://arxiv.org/abs/1805.11222 · arXiv
  21. Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor O. K. Li. 2018. Universal Neural Machine Translation for Extremely Low Resource Languages. CoRR abs/1802.05368 (2018). arXiv:1802.05368 http://arxiv.org/abs/1802.05368 · arXiv
  22. Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. 2018. Co-sampling: Training Robust Networks for Extremely Noisy Supervision. CoRR abs/1804.06872 (2018). arXiv:1804.06872 http://arxiv.org/abs/1804.06872 · arXiv
  23. Judy Homan, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. 2017. CyCADA: Cycle-Consistent Adversarial Domain Adaptation. CoRR abs/1711.03213 (2017). arXiv:1711.03213 http://arxiv. org/abs/1711.03213 · arXiv
  24. Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. 2020. Meta-learning in neural networks: A survey. arXiv preprint arXiv:2004.05439 (2020). · arXiv
  25. Kyle Hsu, Sergey Levine, and Chelsea Finn. 2018. Unsupervised Learning via Meta-Learning. CoRR abs/1810.02334 (2018). arXiv:1810.02334 http://arxiv.org/ abs/1810.02334 · arXiv
  26. Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. 2017. Learning to cluster in order to transfer across domains and tasks. arXiv preprint arXiv:1711.10125 (2017). · arXiv
  27. Xin Huang, Yuxin Peng, and Mingkuan Yuan. 2017. Cross-modal common repre- sentation learning by hybrid transfer network. In Proceedings of the 26th Interna- tional Joint Conference on Articial Intelligence. 1893-1900.
  28. Mi-Young Huh, Pulkit Agrawal, and Alexei A. Efros. 2016. What makes ImageNet good for transfer learning? CoRR abs/1608.08614 (2016). arXiv:1608.08614 http: //arxiv.org/abs/1608.08614 · arXiv
  29. Hal Daumé III. 2009. Frustratingly Easy Domain Adaptation. CoRR abs/0907.1815 (2009). arXiv:0907.1815 http://arxiv.org/abs/0907.1815 · arXiv
  30. Pratik Jawanpuria, Mayank Meghwanshi, and Bamdev Mishra. 2020. Geometry- aware Domain Adaptation for Unsupervised Alignment of Word Embeddings. arXiv preprint arXiv:2004.08243 (2020). · arXiv
  31. Annervaz K M, Somnath Basu Roy Chowdhury, and Ambedkar Dukkipati. 2018. Learning beyond Datasets: Knowledge Graph Augmented Neural Networks for Natural Language Processing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
  32. Mikhail Khodak, Maria-Florina Balcan, and Ameet Talwalkar. 2019. Provable Guarantees for Gradient-Based Meta-Learning. CoRR abs/1902.10644 (2019). arXiv:1902.10644 http://arxiv.org/abs/1902.10644 · arXiv
  33. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. 2016. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. CoRR abs/1602.07332 (2016). arXiv:1602.07332 http://arxiv.org/abs/1602.07332 · arXiv
  34. Anurag Kumar, Maksim Khadkevich, and Christian Fugen. 2018. Knowledge Transfer from Weakly Labeled Audio Using Convolutional Neural Network for Sound Events and Scenes. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 326-330.
  35. Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. 2016. Autoencoding beyond pixels using a learned similarity metric. In International conference on machine learning. PMLR, 1558-1566.
  36. Xinjian Li, Zhong Zhou, Siddharth Dalmia, Alan W Black, and Florian Metze. 2019. SANTLR: Speech Annotation Toolkit for Low Resource Languages. arXiv preprint arXiv:1908.01067 (2019). · arXiv
  37. Paul Pu Liang, Terrance Liu, Liu Ziyin, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523 (2020). · arXiv
  38. Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2018. Multimodal local-global ranking fusion for emotion recognition. In Proceedings of the 20th ACM International Conference on Multimodal Interaction. 472-476.
  39. George A. Miller. 1995. WordNet: A Lexical Database for English. Commun. ACM 38, 11 (Nov. 1995), 39-41.
  40. W. Min, S. Jiang, J. Sang, H. Wang, X. Liu, and L. Herranz. 2017. Being a Supercook: Joint Food Attributes and Multimodal Content Modeling for Recipe Retrieval and Exploration. IEEE Transactions on Multimedia 19, 5 (2017), 1100-1113. https: //doi.org/10.1109/TMM.2016.2639382 · DOI · Paper page
  41. Tsendsuren Munkhdalai and Hong Yu. 2017. Meta networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2554-2563.
  42. Alex Nichol and John Schulman. 2018. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999 2 (2018), 2. · arXiv
  43. OpenAI, Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefow- icz, Bob McGrew, Jakub W. Pachocki, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. 2018. Learning Dexter- ous In-Hand Manipulation. CoRR abs/1808.00177 (2018). arXiv:1808.00177 http://arxiv.org/abs/1808.00177 · arXiv
  44. Akila Pemasiri, Kien Nguyen, Sridha Sridharan, and Clinton Fookes. 2018. Se- mantic Correspondence: A Hierarchical Approach. CoRR abs/1806.03560 (2018). arXiv:1806.03560 http://arxiv.org/abs/1806.03560 · arXiv
  45. Veronica Perez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-Level Multimodal Sentiment Analysis. In Association for Computational Linguistics (ACL). Soa, Bulgaria.
  46. Karol J Piczak. 2015. ESC: Dataset for environmental sound classication. In Proceedings of the 23rd ACM international conference on Multimedia. 1015-1018.
  47. Anant Raj, Vinay P. Namboodiri, and Tinne Tuytelaars. 2015. Subspace Alignment Based Domain Adaptation for RCNN Detector. CoRR abs/1507.05578 (2015). arXiv:1507.05578 http://arxiv.org/abs/1507.05578 · arXiv
  48. Tal Ridnik, Hussam Lawen, Asaf Noy, and Itamar Friedman. 2020. TResNet: High Performance GPU-Dedicated Architecture. arXiv preprint arXiv:2003.13630 (2020). · arXiv
  49. Doyen Sahoo, Hung Le, Chenghao Liu, and Steven C. H. Hoi. 2019. Meta-Learning with Domain Adaptation for Few-Shot Learning under Domain Shift. https: //openreview.net/forum?id=ByGOuo0cYm
  50. Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. 2019. Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing. CoRR abs/1902.09492 (2019). arXiv:1902.09492 http://arxiv. org/abs/1902.09492 · arXiv
  51. Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Advances in neural information processing systems. 4077- 4087.
  52. Richard Socher, Milind Ganjoo, Christopher D. Manning, and Andrew Y. Ng. 2013. Zero-Shot Learning through Cross-Modal Transfer. In Proceedings of the 26th International Conference on Neural Information Processing Systems -Volume 1 (Lake Tahoe, Nevada) (NIPS'13).
  53. Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chun- fang Liu. 2018. A Survey on Deep Transfer Learning. CoRR abs/1808.01974 (2018). arXiv:1808.01974 http://arxiv.org/abs/1808.01974 · arXiv
  54. Yao-Hung Hubert Tsai and Ruslan Salakhutdinov. 2017. Improving One- Shot Learning through Fusing Side Information. CoRR abs/1710.08347 (2017). arXiv:1710.08347 http://arxiv.org/abs/1710.08347 · arXiv
  55. Eric Tzeng, Judy Homan, Kate Saenko, and Trevor Darrell. 2017. Adversar- ial discriminative domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7167-7176.
  56. Eric Tzeng, Judy Homan, Ning Zhang, Kate Saenko, and Trevor Darrell. 2014. Deep Domain Confusion: Maximizing for Domain Invariance. CoRR abs/1412.3474 (2014). arXiv:1412.3474 http://arxiv.org/abs/1412.3474 · arXiv
  57. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. In Advances in neural information processing systems. 3630-3638.
  58. Chang Wang and Sridhar Mahadevan. 2011. Heterogeneous Domain Adaptation Using Manifold Alignment. In Proceedings of the Twenty-Second International Joint Conference on Articial Intelligence -Volume Volume Two (Barcelona, Catalonia, Spain) (IJCAI'11). AAAI Press, 1541-1546.
  59. Zirui Wang, Jiateng Xie, Ruochen Xu, Yiming Yang, Graham Neubig, and Jaime Carbonell. 2019. Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unied Framework. arXiv preprint arXiv:1910.04708 (2019). · arXiv
  60. Chen Xing, Negar Rostamzadeh, Boris Oreshkin, and Pedro O O. Pinheiro. 2019. Adaptive Cross-Modal Few-shot Learning. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). http://papers.nips.cc/paper/8731-adaptive-cross- modal-few-shot-learning.pdf
  61. Zhenguo Yang, Min Cheng, Qing Li, Yukun Li, Zehang Lin, and Wenyin Liu. 2017. Cross-Domain and Cross-Modality Transfer Learning for Multi-domain and Multi-modality Event Detection. In Web Information Systems Engineering -WISE 2017, Athman Bouguettaya, Yunjun Gao, Andrey Klimenko, Lu Chen, Xiangliang Zhang, Fedor Dzerzhinskiy, Weijia Jia, Stanislav V. Klimenko, and Qing Li (Eds.). Springer International Publishing, Cham, 516-523.
  62. Qiming Zhang, Jing Zhang, Wei Liu, and Dacheng Tao. 2019. Category Anchor- Guided Unsupervised Domain Adaptation for Semantic Segmentation. In Ad- vances in Neural Information Processing Systems. 433-443.

Cited by

Sorted by citations

Authors
Liang, P. P., Wu, P., Ziyin, L., Morency, L.-P., & Salakhutdinov, R.
Affiliations
  • Carnegie Mellon University (US)
  • The University of Tokyo (JP)
ORCID
Venue
Proceedings of the 29th ACM International Conference on Multimedia
Publisher
ACM
Published
2021-10-17
Pages
2680-2689
DOI
10.1145/3474085.3475247
OpenAlex
W3110801570
Cited by
22
References
4
Citations by year
2026: 1 · 2025: 6 · 2024: 5 · 2023: 3 · 2022: 3 · 2021: 4
FWCI
1.73
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1722822, 1750439, IIS1763562)
  • Office of Naval Research (N000141812861)
  • National Institutes of Health (R01MH125740, R01MH096951, U01MH116925, U01MH116923)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
cross-modal generalization, cross-modal representation alignment, few-shot learning, label noise, rare languages

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review