CoChat
Sign Up
PreprintCited by 1Multimodal Machine Learning ApplicationsAdversarial Robustness in Machine LearningDomain Adaptation and Few-Shot Learning

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

On Robustness in Multimodal Learning

Open access
McKinzie, Brandon, Cheng, Joseph, Shankar, Vaishaal, Yang, Yinfei, Shlens, Jonathon, Toshev, Alexander·2023·arXiv
DOI 10.48550/arxiv.2304.04385arXiv 2304.04385
Open this paper in CoChat — free accountOpen PDF

Abstract

Multimodal learning is defined as learning over multiple heterogeneous input modalities such as video, audio, and text. In this work, we are concerned with understanding how models behave as the type of modalities differ between training and deployment, a situation that naturally arises in many applications of multimodal learning to hardware platforms. We present a multimodal robustness framework to provide a systematic analysis of common multimodal representation learning methods. Further, we identify robustness short-comings of these approaches and propose two intervention techniques leading to $1.5\times$-$4\times$ robustness improvements on three datasets, AudioSet, Kinetics-400 and ImageNet-Captions. Finally, we demonstrate that these interventions better utilize additional modalities, if present, to achieve competitive results of $44.2$ mAP on AudioSet 20K.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.04385

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The performance of several existing multimodal networks significantly deteriorates when one or multiple modalities are absent at test time.

    “However, we observe that the performance of several existing multimodal networks significantly deteriorates if one or multiple modalities are absent at test time.”

    From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [16]

    Section 6.1 reports that when modalities are missing at evaluation, both methods show a large drop in robustness. It describes the drop in average performance as small, however, so the claim overstates how much performance deteriorates. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 73

  1. Akbari, H., Yuan, L., Qian, R., Chuang, W.-H., Chang, S.-F., Cui, Y., and Gong, B. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34: 24206-24221, 2021.
  2. Alayrac, J., Recasens, A., Schneider, R., Arandjelovic, R., Ramapuram, J., Fauw, J. D., Smaira, L., Dieleman, S., and Zisserman, A. Self-supervised multimodal versatile networks. CoRR, abs/2006.16228, 2020. URL https: //arxiv.org/abs/2006.16228. · arXiv
  3. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022. · arXiv
  4. Alzantot, M., Sharma, Y., Elgohary, A., Ho, B.-J., Srivas- tava, M., and Chang, K.-W. Generating natural language adversarial examples. arXiv preprint arXiv:1804.07998, 2018. · arXiv
  5. Baade, A., Peng, P., and Harwath, D. Mae-ast: Masked au- toencoding audio spectrogram transformer. arXiv preprint arXiv:2203.16691, 2022. · arXiv
  6. Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gut- freund, D., Tenenbaum, J., and Katz, B. Objectnet: A large-scale bias-controlled dataset for pushing the lim- its of object recognition models. Advances in neural information processing systems, 32, 2019.
  7. Boser, B. E., Guyon, I. M., and Vapnik, V. N. A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory, pp. 144-152, 1992.
  8. Carlini, N. and Wagner, D. Audio adversarial examples: Targeted attacks on speech-to-text. In 2018 IEEE security and privacy workshops (SPW), pp. 1-7. IEEE, 2018.
  9. Carlini, N., Tramer, F., Dvijotham, K., and Kolter, J. Z. (certified!!) adversarial robustness for free!, 2022. URL https://arxiv.org/abs/2206.10550. · arXiv
  10. Carmon, Y., Raghunathan, A., Schmidt, L., Duchi, J. C., and Liang, P. S. Unlabeled data improves adversarial robustness. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Pro- cessing Systems, volume 32.
  11. Curran Associates, Inc., 2019. URL https://proceedings. neurips.cc/paper/2019/file/ 32e0bd1497aa43e02a42f47d9d6515ad-Paper. pdf.
  12. Castro, F. M., Marín-Jiménez, M. J., Guil, N., Schmid, C., and Alahari, K. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233-248, 2018.
  13. Chong, D., Wang, H., Zhou, P., and Zeng, Q. Masked spectrogram prediction for self-supervised audio pre- training, 2022. URL https://arxiv.org/abs/ 2204.12768.
  14. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, pp. 248-255. Ieee, 2009.
  15. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. · arXiv
  16. Dou, Z.-Y., Xu, Y., Gan, Z., Wang, J., Wang, S., Wang, L., Zhu, C., Zhang, P., Yuan, L., Peng, N., Liu, Z., and Zeng, M. An empirical study of training end-to-end vision-and-language transformers, 2021. URL https: //arxiv.org/abs/2111.02387.
  17. Fang, A., Ilharco, G., Wortsman, M., Wan, Y., Shankar, V., Dave, A., and Schmidt, L. Data determines distri- butional robustness in contrastive language image pre- training (CLIP). In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Ma- chine Learning Research, pp. 6216-6234. PMLR, 17- 23 Jul 2022a. URL https://proceedings.mlr. press/v162/fang22a.html.
  18. Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva: Explor- ing the limits of masked visual representation learning at scale, 2022b. URL https://arxiv.org/abs/ 2211.07636.
  19. Feichtenhofer, C., Fan, H., Li, Y., and He, K. Masked autoencoders as spatiotemporal learners. arXiv preprint arXiv:2205.09113, 2022. · arXiv
  20. Geirhos, R., Temme, C. R. M., Rauber, J., Schütt, H. H., Bethge, M., and Wichmann, F. A. Generalisation in humans and deep neural networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa- Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31.
  21. Curran As- sociates, Inc., 2018. URL https://proceedings. neurips.cc/paper/2018/file/ 0937fb5864ed06ffb59ae5f9b5ed67a9-Paper. pdf.
  22. Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
  23. Geng, X., Liu, H., Lee, L., Schuurams, D., Levine, S., and Abbeel, P. Multimodal masked autoencoders learn trans- ferable representations. arXiv preprint arXiv:2205.14204, 2022. · arXiv
  24. Georgescu, M.-I., Fonseca, E., Ionescu, R. T., Lucic, M., Schmid, C., and Arnab, A. Audiovisual masked autoen- coders. arXiv preprint arXiv:2212.05922, 2022. · arXiv
  25. Girdhar, R., El-Nouby, A., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Omnimae: Single model masked pretraining on images and videos. arXiv preprint arXiv:2206.08356, 2022. · arXiv
  26. Gong, Y., Chung, Y.-A., and Glass, J. Ast: Audio spec- trogram transformer. arXiv preprint arXiv:2104.01778, 2021. · arXiv
  27. Gong, Y., Rouditchenko, A., Liu, A. H., Harwath, D., Karlinsky, L., Kuehne, H., and Glass, J. Con- trastive audio-visual masked autoencoder. arXiv preprint arXiv:2210.07839, 2022. · arXiv
  28. Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. Deep learning, volume 1. MIT Press, 2016.
  29. Gu, K., Yang, B., Ngiam, J., Le, Q., and Shlens, J. Using videos to evaluate image model robustness, 2019. URL https://arxiv.org/abs/1904.10076. · arXiv
  30. Gutmann, M. and Hyvärinen, A. Noise-contrastive estima- tion: A new estimation principle for unnormalized statisti- cal models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
  31. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
  32. Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and pertur- bations, 2019. URL https://arxiv.org/abs/ 1903.12261.
  33. Hessel, J. and Lee, L. Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think! arXiv preprint arXiv:2010.06572, 2020. · arXiv
  34. Hinton, G., Vinyals, O., Dean, J., et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2(7), 2015. · arXiv
  35. Hou, S., Pan, X., Loy, C. C., Wang, Z., and Lin, D. Lifelong learning via progressive distillation and retrospection. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 437-452, 2018.
  36. Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Deep networks with stochastic depth, 2016. URL https://arxiv.org/abs/1603.09382. · arXiv
  37. Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., and Feichtenhofer, C. Masked autoen- coders that listen, 2022. URL https://arxiv.org/ abs/2207.06405. · arXiv
  38. Ilharco, G., Wortsman, M., Gadre, S. Y., Song, S., Hajishirzi, H., Kornblith, S., Farhadi, A., and Schmidt, L. Patching open-vocabulary models by interpolating weights, 2022. URL https://arxiv.org/abs/2208.05592. · arXiv
  39. Jaegle, A., Borgeaud, S., Alayrac, J.-B., Doersch, C., Ionescu, C., Ding, D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., et al. Perceiver io: A general archi- tecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021. · arXiv
  40. Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. ICML, 2021.
  41. Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017. · arXiv
  42. Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
  43. Liang, P. P., Lyu, Y., Fan, X., Mo, S., Yogatama, D., Morency, L.-P., and Salakhutdinov, R. Highmmt: To- wards modality and task generalization for high-modality representation learning. arXiv preprint arXiv:2203.01311, 2022. · arXiv
  44. Liu, K., Li, Y., Xu, N., and Natarajan, P. Learn to com- bine modalities in multimodal deep learning, 2018. URL https://arxiv.org/abs/1805.11730. · arXiv
  45. Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
  46. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. · arXiv
  47. Nagrani, A., Yang, S., Arnab, A., Jansen, A., Schmid, C., and Sun, C. Attention bottlenecks for multimodal fusion. Advances in Neural Information Processing Systems, 34: 14200-14213, 2021.
  48. Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. · arXiv
  49. Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., and Le, Q. V. SpecAugment: A simple data augmentation method for automatic speech recog- nition. In Interspeech 2019. ISCA, sep 2019. doi: 10.21437/interspeech.2019-2680. URL https://doi. org/10.21437%2Finterspeech.2019-2680. · DOI · Paper page
  50. Pham, H., Dai, Z., Ghiasi, G., Kawaguchi, K., Liu, H., Yu, A. W., Yu, J., Chen, Y.-T., Luong, M.-T., Wu, Y., Tan, M., and Le, Q. V. Combined scaling for open- vocabulary image classification, 2021. URL https: //arxiv.org/abs/2111.10050.
  51. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  52. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. ICML, 2021.
  53. Rebuffi, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
  54. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In ICML, pp. 5389-5400. PMLR, 2019.
  55. Shankar, V., Dave, A., Roelofs, R., Ramanan, D., Recht, B., and Schmidt, L. Do image classifiers generalize across time? In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9661-9669, October 2021.
  56. Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. Flava: A founda- tional language and vision alignment model, 2021. URL https://arxiv.org/abs/2112.04482. · arXiv
  57. Sohn, K. Improved deep metric learning with multi-class n-pair loss objective. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 29.
  58. Curran As- sociates, Inc., 2016. URL https://proceedings. neurips.cc/paper/2016/file/ 6b180037abbebea991d8b1232f8a8ca9-Paper. pdf.
  59. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. · arXiv
  60. Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribution shifts in image classification. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18583-18599.
  61. Curran Asso- ciates, Inc., 2020. URL https://proceedings. neurips.cc/paper/2020/file/ d8330f857a17c53d217014ee776bfd50-Paper. pdf.
  62. Tong, Z., Song, Y., Wang, J., and Wang, L. Video- mae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203.12602, 2022. · arXiv
  63. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. NeurIPS, 30, 2017.
  64. Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022. · arXiv
  65. Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7959-7971, June 2022.
  66. Wu, N., Jastrzebski, S., Cho, K., and Geras, K. J. Charac- terizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In International Con- ference on Machine Learning, pp. 24043-24055.
  67. PMLR, 2022a.
  68. Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg-Kirkpatrick, T., and Dubnov, S. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. arXiv preprint arXiv:2211.06687, 2022b. · arXiv
  69. Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., Feichtenhofer, C., et al. Masked autoencoders that listen. arXiv preprint arXiv:2207.06405, 2022. · arXiv
  70. Yang, Z., Fang, Y., Zhu, C., Pryzant, R., Chen, D., Shi, Y., Xu, Y., Qian, Y., Gao, M., Chen, Y.-L., et al. i-code: An integrative and composable multimodal learning frame- work. arXiv preprint arXiv:2205.01818, 2022. · arXiv
  71. You, H., Zhou, L., Xiao, B., Codella, N., Cheng, Y., Xu, R., Chang, S.-F., and Yuan, L. Learning visual representation from modality-shared contrastive language-image pre- training, 2022. URL https://arxiv.org/abs/ 2207.12661.
  72. Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. · arXiv
  73. Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization, 2017. URL https://arxiv.org/abs/1710.09412. · arXiv

Cited by

Sorted by citations

Metadata

Authors
McKinzie, B., Cheng, J., Shankar, V., Yang, Y., Shlens, J., & Toshev, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2304.04385
arXiv
2304.04385
OpenAlex
W4365211948
Cited by
1
References
1
Citations by year
2024: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
multimodal representation learning, AudioSet, Kinetics-400, hardware deployment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review