CoChat
Sign Up
PreprintCited by 27Multimodal Machine Learning ApplicationsDomain Adaptation and Few-Shot LearningTopic Modeling

The DOI matches its DataCite record.

Modular and Parameter-Efficient Multimodal Fusion with Prompting

Open access
Liang, Sheng, Zhao, Mengjie, Schütze, Hinrich·2022·arXiv
DOI 10.48550/arxiv.2203.08055arXiv 2203.08055
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent research has made impressive progress in large-scale multimodal pre-training. In the context of the rapid growth of model size, it is necessary to seek efficient and flexible methods other than finetuning. In this paper, we propose to use prompt vectors to align the modalities. Our method achieves comparable performance to several other multimodal fusion methods in low-resource settings. We further show that our method is modular and parameter-efficient for processing tasks involving two or more data modalities.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2203.08055

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 48

  1. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Proceedings of the 34th Conference on Neural Information Processing Systems, volume 33.
  2. Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. OpenFace 2.0: Fa- cial behavior analysis toolkit. In 2018 13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018), pages 59-66. · DOI · Paper page
  3. Ankur Bapna, Yu-an Chung, Nan Wu, Anmol Gulati, Ye Jia, Jonathan H Clark, Melvin Johnson, Jason Riesa, Alexis Conneau, and Yu Zhang. 2021. SLAM: A unified encoder for speech and language model- ing via speech-text joint pre-training. arXiv preprint arXiv:2110.10329. · arXiv
  4. Andrew Brock, Soham De, Samuel L Smith, and Karen Simonyan. 2021. High-performance large- scale image recognition without normalization. arXiv preprint arXiv:2102.06171. · arXiv
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
  6. Curran Associates, Inc. Santiago Castro, Devamanyu Hazarika, Verónica Pérez- Rosas, Roger Zimmermann, Rada Mihalcea, and Sou- janya Poria. 2019. Towards multimodal sarcasm de- tection (an obviously perfect paper). In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), Florence, Italy. Association for Computational Linguistics.
  7. Jaemin Cho, Jie Lei, Haochen Tan, and M. Bansal. 2021. Unifying vision-and-language tasks via text genera- tion. In ICML.
  8. S. Davis and P. Mermelstein. 1980. Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences. IEEE Transactions on Acoustics, Speech, and Signal Pro- cessing, 28(4):357-366. · DOI · Paper page
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI · Paper page
  10. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
  11. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 3816-3830, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  12. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR).
  13. Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representa- tions.
  14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735-1780. · DOI · Paper page
  15. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In In- ternational Conference on Machine Learning, pages 2790-2799. PMLR.
  16. Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolu- tion or region supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5583-5594.
  17. PMLR.
  18. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
  19. Teven Le Scao and Alexander Rush. 2021. How many data points is a prompt worth? In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2627-2636, Online. Association for Computational Linguistics. · DOI · Paper page
  20. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
  21. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  22. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics. · DOI · Paper page
  23. Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, et al. 2021. M6: A chinese multimodal pretrainer. arXiv preprint arXiv:2103.00823. · arXiv
  24. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586. · arXiv
  25. Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021b. Swin Transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030. · arXiv
  26. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and-language tasks. In NeurIPS.
  27. Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. 2015. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, volume 8, pages 18-25. Citeseer.
  28. Timothy Niven and Hung-Yu Kao. 2019. Probing neu- ral network comprehension of natural language argu- ments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658-4664, Florence, Italy. Association for Compu- tational Linguistics. · DOI · Paper page
  29. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020. · arXiv
  30. Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
  31. Ander Salaberria, Gorka Azkune, Oier Lopez de La- calle, Aitor Soroa, and Eneko Agirre. 2021. Image captioning for effective use of language models in knowledge-based visual question answering. arXiv preprint arXiv:2109.08029. · arXiv
  32. Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics. · DOI · Paper page
  33. Nitish Srivastava, Ruslan Salakhutdinov, et al. 2014. Multimodal learning with deep boltzmann machines. J. Mach. Learn. Res., 15(1):2949-2980.
  34. Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. VL-BERT: Pre- training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530. · arXiv
  35. Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. VideoBERT: A joint model for video and language representation learn- ing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464-7473.
  36. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In Inter- national Conference on Machine Learning, pages 3319-3328. PMLR.
  37. Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021. Improving and simplifying pattern exploiting training. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4980-4991, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
  38. Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 5100-5111, Hong Kong, China. Association for Com- putational Linguistics. · DOI · Paper page
  39. Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. Es- lami, Oriol Vinyals, and Felix Hill. 2021. Multi- modal few-shot learning with frozen language mod- els. ArXiv, abs/2106.13884. · arXiv
  40. Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yu- lia Tsvetkov, and Yuan Cao. 2021. SimVLM: Simple visual language model pretraining with weak super- vision. arXiv preprint arXiv:2108.10904. · arXiv
  41. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
  42. Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2021. An empirical study of GPT-3 for few-shot knowledge- based VQA. arXiv preprint arXiv:2109.05014. · arXiv
  43. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Advances in Neural In- formation Processing Systems, volume 32.
  44. Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2021. CPT: Col- orful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.11797. · arXiv
  45. Elad Ben Zaken, Shauli Ravfogel, and Yoav Gold- berg. 2021. BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language- models. CoRR, abs/2106.10199. · arXiv
  46. Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hin- rich Schütze. 2020. Masking as an efficient alterna- tive to finetuning for pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2226-2241, Online. Association for Computa- tional Linguistics. · DOI
  47. Mengjie Zhao and Hinrich Schütze. 2021. Discrete and soft prompting for multilingual models. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8547-8555, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics. · DOI
  48. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2021. Learning to prompt for vision- language models. arXiv preprint arXiv:2109.01134. · arXiv

Cited by

See all 27

Metadata

Authors
Liang, S., Zhao, M., & Schütze, H.
Affiliations
Ludwig-Maximilians-Universität München (DE)
ORCID
Zhao, Mengjie
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
2976-2985
DOI
10.48550/arxiv.2203.08055
arXiv
2203.08055
OpenAlex
W4226208626
Cited by
27
References
0
Citations by year
2026: 4 · 2025: 5 · 2024: 9 · 2023: 9
FWCI
2.08
Type
conference paper
Language
en
Open access
Licence
Locations
  • Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Open access LMU (Ludwid Maxmilian's Universitat Munchen) (submittedVersion, other-oa)
  • arXiv (Cornell University)
  • Universitätsbibliothek der LMU
Funders
H2020 European Research Council (740516)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
multimodal fusion, prompting, parameter efficiency, modular model, resource-limited settings, multimodal pre-training, cross-modal alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Modular and Parameter-Efficient Multimodal Fusion with Prompting. https://app.cochat.ai/papers/10.48550/arxiv.2203.08055
The paper
Liang, S., Zhao, M., & Schütze, H. (2022). Modular and Parameter-Efficient Multimodal Fusion with Prompting. arXiv. https://doi.org/10.48550/arxiv.2203.08055