CoChat
Sign Up
PreprintCited by 761Multimodal Machine Learning ApplicationsVisual Attention and Saliency DetectionAdvanced Image and Video Retrieval Techniques

Bilinear Attention Networks

Open access
Kim, Jin-Hwa, Jun, Jaehyun, Zhang, Byoung-Tak·2018·arXiv
DOI 10.48550/arxiv.1805.07932arXiv 1805.07932
Open this paper in CoChat — free accountOpen PDF

Abstract

Attention networks in multimodal learning provide an efficient way to utilize given visual information selectively. However, the computational cost to learn attention distributions for every pair of multimodal input channels is prohibitively expensive. To solve this problem, co-attention builds two separate attention distributions for each modality neglecting the interaction between multimodal inputs. In this paper, we propose bilinear attention networks (BAN) that find bilinear attention distributions to utilize given vision-language information seamlessly. BAN considers bilinear interactions among two groups of input channels, while low-rank bilinear pooling extracts the joint representations for each pair of channels. Furthermore, we propose a variant of multimodal residual networks to exploit eight-attention maps of the BAN efficiently. We quantitatively and qualitatively evaluate our model on visual question answering (VQA 2.0) and Flickr30k Entities datasets, showing that BAN significantly outperforms previous methods and achieves new state-of-the-arts on both datasets.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1805.07932

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 39

  1. [1][1] Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C Lawrence Zitnick, Devi Parikh, and Dhruv Batra. Vqa: Visual question answering. International Journal of Computer Vision , 123(1):4-31, 2017. -
  2. [2][2] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. arXiv preprint arXiv:1707.07998 , 2017. - · arXiv
  3. [3][3] Pablo Arbeláez, Jordi Pont-Tuset, Jonathan T Barron, Ferran Marques, and Jitendra Malik. Multiscale combinatorial grouping. In IEEE conference on computer vision and pattern recognition , pages 328-335, 2014. -
  4. [4][4] Hedi Ben-younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome. MUTAN: Multimodal Tucker Fusion for Visual Question Answering. In IEEE International Conference on Computer Vision , pages 2612-2620, 2017. -
  5. [5][5] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN EncoderDecoder for Statistical Machine Translation. In 2014 Conference on Empirical Methods in Natural Language Processing , pages 1724-1734, 2014. -
  6. [6][6] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding. arXiv preprint arXiv:1606.01847 , 2016. - · arXiv
  7. [7][7] Ross Girshick. Fast r-cnn. In IEEE International Conference on Computer Vision , pages 1440-1448, 2015. -
  8. [8][8] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. In IEEE Conference on Computer Vision and Pattern Recognition , 2017. -
  9. [10][10] Ryota Hinami and Shin'ichi Satoh. Query-Adaptive R-CNN for Open-Vocabulary Object Detection and Retrieval. arXiv preprint arXiv:1711.09509 , 2017. - · arXiv
  10. [11][11] Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In IEEE Computer Vision and Pattern Recognition , pages 4555-4564, 2016. -
  11. [13][13] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial Transformer Networks. In Advances in Neural Information Processing Systems 28 , pages 2008-2016, 2015. -
  12. [14][14] Jin-Hwa Kim, Sang-Woo Lee, Donghyun Kwak, Min-Oh Heo, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Multimodal Residual Learning for Visual QA. In Advances in Neural Information Processing Systems 29 , pages 361-369, 2016. -
  13. [15][15] Jin-Hwa Kim, Kyoung Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Ha, and Byoung-Tak Zhang. Hadamard Product for Low-rank Bilinear Pooling. In The 5th International Conference on Learning Representations , 2017. -
  14. [16][16] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations , 2015. -
  15. [17][17] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. arXiv preprint arXiv:1602.07332 , 2016. - · arXiv
  16. [18][18] Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical Question-Image CoAttention for Visual Question Answering. arXiv preprint arXiv:1606.00061 , 2016. - · arXiv
  17. [19][19] Vinod Nair and Geoffrey E Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. 27th International Conference on Machine Learning , pages 807-814, 2010. -
  18. [20][20] Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim. Dual Attention Networks for Multimodal Reasoning and Matching. In IEEE Conference on Computer Vision and Pattern Recognition , 2016. -
  19. [21][21] Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing , 2014. -
  20. [22][22] Hamed Pirsiavash, Deva Ramanan, and Charless C. Fowlkes. Bilinear classifiers for visual recognition. In Advances in Neural Information Processing Systems 22 , pages 1482-1490, 2009. -
  21. [23][23] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. International Journal of Computer Vision , 123:74-93, 2017. -
  22. [24][24] Joseph Redmon and Ali Farhadi. YOLO9000: Better, Faster, Stronger. In IEEE Computer Vision and Pattern Recognition , 2017. -
  23. [26][26] Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bernt Schiele. Grounding of textual phrases in images by reconstruction. In European Conference on Computer Vision , pages 817-834, 2016. -
  24. [27][27] Tim Salimans and Diederik P. Kingma. Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. arXiv preprint arXiv:1602.07868 , 2016. - · arXiv
  25. [28][28] Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout : A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research , 15(1):1929-1958, 2014. -
  26. [29][29] Damien Teney, Peter Anderson, Xiaodong He, and Anton van den Hengel. Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge. arXiv preprint arXiv:1708.02711 , 2017. - · arXiv
  27. [30][30] Alexander Trott, Caiming Xiong, and Richard Socher. Interpretable Counting for Visual Question Answering. In International Conference on Learning Representations , 2018. -
  28. [31][31] Andreas Veit, Michael J Wilber, and Serge Belongie. Residual Networks are Exponential Ensembles of Relatively Shallow Networks. In Advances in Neural Information Processing Systems 29 , pages 550-558, 2016. -
  29. [32][32] Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning Deep Structure-Preserving ImageText Embeddings. In IEEE Conference on Computer Vision and Pattern Recognition , pages 5005-5013, 2016. -
  30. [33][33] Mingzhe Wang, Mahmoud Azab, Noriyuki Kojima, Rada Mihalcea, and Jia Deng. Structured Matching for Phrase Localization. In European Conference on Computer Vision , volume 9908, pages 696-711, 2016. -
  31. [34][34] Peng Wang, Qi Wu, Chunhua Shen, and Anton van den Hengel. The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions. In Computer Vision and Pattern Recognition (CVPR) , pages 1173-1182, 2017. -
  32. [35][35] Lior Wolf, Hueihan Jhuang, and Tamir Hazan. Modeling appearances with low-rank SVM. IEEE Conference on Computer Vision and Pattern Recognition , 2007. -
  33. [36][36] Huijuan Xu and Kate Saenko. Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering. In European Conference on Computer Vision , 2016. -
  34. [37][37] Raymond A Yeh, Jinjun Xiong, Wen-Mei W Hwu, Minh N Do, and Alexander G Schwing. Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts. In Advances in Neural Information Processing Systems 30 , 2017. -
  35. [38][38] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2:67-78, 2014. -
  36. [39][39] Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao. Beyond Bilinear: Generalized Multi-modal Factorized High-order Pooling for Visual Question Answering. IEEE Transactions on Neural Networks and Learning Systems , 2018. -
  37. [40][40] Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-Down Neural Attention by Excitation Backprop. In European Conference on Computer Vision , volume 9908, pages 543-559, 2016. -
  38. [41][41] Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. Learning to Count Objects in Natural Images for Visual Question Answering. In International Conference on Learning Representations , 2018. -
  39. [42][42] C Lawrence Zitnick and Piotr Dollár. Edge boxes: Locating object proposals from edges. In European Conference on Computer Vision , pages 391-405, 2014.

Cited by

Sorted by citations

Authors
Kim, J.-H., Jun, J., & Zhang, B.-T.
Affiliations
Seoul National University (KR)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1805.07932
arXiv
1805.07932
OpenAlex
W2804243936
Cited by
761
References
2
Citations by year
2026: 3 · 2025: 57 · 2024: 117 · 2023: 134 · 2022: 103 · 2021: 155 · 2020: 128 · 2019: 60 · 2018: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Seoul National University
  • Korea Evaluation Institute of Industrial Technology (10060086-RISF)
  • Institute for Information and Communications Technology Promotion (2018-0-00622-RMI, 2017-0-01772-VTT, 2017-0-01772, 2018-0-00622)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
Keywords
multimodal attention, co-attention, bilinear pooling, vision-language models, visual question answering, attention maps

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review