How Transferable are Reasoning Patterns in VQA?
Open accessAbstract
Since its inception, Visual Question Answering (VQA) is notoriously known as a task, where models are prone to exploit biases in datasets to find shortcuts instead of performing high-level reasoning. Classical methods address this by removing biases from training data, or adding branches to models to detect and remove biases. In this paper, we argue that uncertainty in vision is a dominating factor preventing the successful learning of reasoning in vision and language problems. We train a visual oracle and in a large scale study provide experimental evidence that it is much less prone to exploiting spurious dataset biases compared to standard models. We propose to study the attention mechanisms at work in the visual oracle and compare them with a SOTA Transformer-based model. We provide an in-depth analysis and visualizations of reasoning patterns obtained with an online visualization tool which we make publicly available1. We exploit these insights by transferring reasoning patterns from the oracle to a SOTA Transformer-based VQA model taking standard noisy visual inputs via fine-tuning. In experiments we report higher overall accuracy, as well as accuracy on infrequent answers for each question type, which provides evidence for improved generalization and a decrease of the dependency on dataset biases.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Consequently, successive works mostly exploit pretrained models for tackling various downstream tasks.
“Consequently, successive works mostly exploit pretrained models for tackling various downstream tasks [120], [263], [264], [265], [266].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [263]
Existing multimodal Transformer models are superior only for specific MML tasks, as they are designed specifically for only a subset of specific tasks.
“existing multimodal Transformer models [9], [199], [263] are superior only for specific MML tasks, as they are designed specifically for only a subset of specific tasks [137], [142], [212], [249], [260], [261], [265], [266].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [263]
Two-stream architectures are typically preferred over one-stream ones for cross-modal retrieval-like tasks in efficiency, since the representation of each modality can be pre-computed beforehand and reused repeatedly.
“two-stream architectures [9], [263] are typically preferred over one-stream ones for cross-modal retrieval-like tasks in efficiency, since the representation of each modality can be pre-computed beforehand and reused repeatedly.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [263]
Properly tackling the noise issue is useful.
“It has been recently shown that properly tackling the noise issue is useful [263], [309].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [263]
As the learning scale expands exponentially, the training data become inevitably noisy and heterogeneous.
“As the learning scale expands exponentially, the training data become inevitably noisy and heterogeneous [9], [199], [263].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [263]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
The mythos of model interpretability
Lipton, Z. C. (2018) · Communications of the ACM · doi:10.1145/3233231 · Cited by 5,427 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2019.00686 · Cited by 3,703 · see citing works
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · see citing works
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287 · see citing works
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · see citing works
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · see citing works
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · see citing works
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781 · see citing works
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · see citing works
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · see citing works
Deep Modular Co-Attention Networks for Visual Question Answering
Zhou, Y., Yu, J., Cui, Y., Tao, D., & Tian, Q. C. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00644 · Cited by 1,013 · see citing works
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950 · see citing works
Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.12 · Cited by 933 · see citing works
MUTAN: Multimodal Tucker Fusion for Visual Question Answering
Ben-younes, H., Cadène, R., Cord, M., & Thome, N. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.285 · Cited by 733 · see citing works
A Multiscale Visualization of Attention in the Transformer Model
Vig, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3007 · Cited by 600 · see citing works
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00522 · Cited by 586 · see citing works
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582 · see citing works
(2021) · International Conference on Learning Representations · Cited by 449 · see citing works
Dynamic Fusion With Intra- and Inter-Modality Attention Flow for Visual Question Answering
Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S. C. H., Wang, X., & Li, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00680 · Cited by 415 · see citing works
Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language UnderstandingPreprint
Yi, K., Wu, J., Gan, C., Torralba, A. B., Kohli, P., & Tenenbaum, J. B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02338 · Cited by 388 · see citing works
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2020) · International Conference on Learning Representations · Cited by 276 · see citing works
From machine learning to machine reasoning
Bottou, L. (2013) · Machine Learning · doi:10.1007/s10994-013-5335-x · Cited by 249 · see citing works
Hopfield Networks is All You Need
Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G. K., et al. (2020) · arXiv · doi:10.48550/arxiv.2008.02217 · Cited by 236 · cited 8× · see citing works
“In contrast to the attention modes reported in [30], we also observed bi-modal k-number distributions, shown in Fig.”
BackgroundMethodResultInfluentialSSTVOS: Sparse Spatiotemporal Transformers for Video Object Segmentation
Duke, B., Ahmed, A. S., Wolf, C., Aarabi, P., & Taylor, G. W. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00585 · Cited by 195 · see citing works
Self-citationDumoulin, V., Perez, E., Schucher, N., Strub, F., de Vries, H., Courville, A. C., & Bengio, Y. (2018) · Distill · doi:10.23915/distill.00011 · Cited by 166 · see citing works
What Does BERT with Vision Look At?
Li, L. H., Yatskar, M., Yin, D., Hsieh, C., & Chang, K.-W. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.469 · Cited by 131 · see citing works
Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models
DeRose, J. F., Wang, J., & Berger, M. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3028976 · Cited by 113 · see citing works
Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
Cao, J., Gan, Z., Cheng, Y., Yu, L., Chen, Y.-C., & Liu, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58539-6_34 · Cited by 105 · see citing works
On the Value of Out-of-Distribution Testing: An Example of Goodhart's LawPreprint
Teney, D., Kafle, K., Shrestha, R., Abbasnejad, E., Kanan, C., & van den Hengel, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09241 · Cited by 99 · see citing works
Roses are Red, Violets are Blue… But Should VQA expect Them To?
Kervadec, C., Antipov, G., Baccouche, M., & Wolf, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00280 · Cited by 84 · see citing works
Self-citationMeta Module Network for Compositional Visual Reasoning
Chen, W., Gan, Z., Li, L., Cheng, Y., Wang, W. Y., & Liu, J. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00070 · Cited by 67 · see citing works
A negative case analysis of visual grounding methods for VQA
Shrestha, R., Kafle, K., & Kanan, C. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.727 · Cited by 48 · see citing works
Weak Supervision helps Emergence of Word-Object Alignment and improves Vision-Language TasksPreprint
Kervadec, C., Antipov, G., Baccouche, M., & Wolf, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.03063 · Cited by 9 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 4
- [23][23] Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems , pages 1564-1574, 2018. 2, 8 -
- [35][35] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visuallinguistic representations. In International Conference on Learning Representations , 2020. 3 -
- [36][36] Hao Tan and Mohit Bansal. Lxmert: Learning crossmodality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP) , pages 5103-5114, 2019. 2, 3, 4, 7, 8, 11 -
- [40][40] Jesse Vig. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations , pages 37-42, 2019. 2 -
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
VisQA: X-raying Vision and Language Reasoning in Transformers
Jaunet, T., Kervadec, C., Vuillemot, R., Antipov, G., Baccouche, M., & Wolf, C. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114683 · Cited by 30 · see citing works
Self-citation
- Authors
- Kervadec, C., Jaunet, T., Antipov, G., Baccouche, M., Vuillemot, R., & Wolf, C.
- Affiliations
- École Centrale de Lyon (FR)
- Orange (France) (FR)
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 4205-4214
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3149876404
- Cited by
- 25
- References
- 42
- Citations by year
- 2025: 1 · 2024: 5 · 2023: 11 · 2022: 4 · 2021: 4
- FWCI
- 1.96
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Agence Nationale de la Recherche (ANR-20-CHIA-0018)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- visual question answering, reasoning patterns, transfer learning, shortcut learning, attention mechanism, fine-tuning, generalization, vision-language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
