The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.
Counterfactual VQA: A Cause-Effect Look at Language Bias
Open accessAbstract
VQA models may tend to rely on language bias as a shortcut and thus fail to sufficiently learn the multi-modal knowledge from both vision and language. Recent debiasing methods proposed to exclude the language prior during inference. However, they fail to disentangle the "good" language context and "bad" language bias from the whole. In this paper, we investigate how to mitigate language bias in VQA. Motivated by causal effects, we proposed a novel counterfactual inference framework, which enables us to capture the language bias as the direct causal effect of questions on answers and reduce the language bias by subtracting the direct language effect from the total causal effect. Experiments demonstrate that our proposed counterfactual inference framework 1) is general to various VQA backbones and fusion strategies, 2) achieves competitive performance on the language-bias sensitive VQA-CP dataset while performs robustly on the balanced VQA v2 dataset without any augmented data. The code is available at https://github.com/yuleiniu/cfvqa.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Marginal Structural Models and Causal Inference in Epidemiology
Robins, J. M., Hernán, M. Á., & Brumback, B. (2000) · Epidemiology · doi:10.1097/00001648-200009000-00011 · Cited by 5,581
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
The Book of Why: The New Science of Cause and Effect
Pearl, J., & Mackenzie, D. N. (2018) · Cited by 3,781
Causality: models, reasoning, and inference
(2000) · Choice Reviews Online · doi:10.5860/choice.38-1494a · Cited by 2,873
Bayesian inference for causal effects: The role of randomization
Rubin, D. B. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344064 · Cited by 2,503
Robins, J. (1986) · Mathematical Modelling · doi:10.1016/0270-0255(86)90088-6 · Cited by 2,451
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.10 · Cited by 1,998
The Book of Why: The New Science of Cause and Effect
Goldberg, L. R. (2019) · Notices of the American Mathematical Society · doi:10.1090/noti1912 · Cited by 1,655
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187
Pearl, J. (2022) · Probabilistic and Causal Inference · doi:10.1145/3501714.3501736 · Cited by 1,033
From Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769
Unbiased Scene Graph Generation From Biased Training
Tang, K., Niu, Y., Huang, J., Shi, J., & Zhang, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00377 · Cited by 755
Self-citationVQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00522 · Cited by 566
Das, A., Kottur, S., Gupta, K., Singh, A., Jain, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · Cited by 453
Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.121 · Cited by 445 · cited 1×
“Visual Question Answering (VQA) [8, 4] has become the fundamental building block that underpins many frontier interactive AI systems, such as visual dialog [15], visionand-language navigation [6], and visual commonsense reasoning [51].”
BackgroundHuman Attention in Visual Question Answering: Do Humans and Deep Networks look at the same regions?
Das, A., Agrawal, H., Zitnick, L., Parikh, D., & Batra, D. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1092 · Cited by 427
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence
Park, D. H., Hendricks, L. A., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., & Rohrbach, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00915 · Cited by 418
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
Clark, C., Yatskar, M., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1418 · Cited by 381
Causal Intervention for Weakly-Supervised Semantic Segmentation
Zhang, D., Zhang, H., Tang, J., Hua, X., & Sun, Q. (2020) · arXiv · doi:10.48550/arxiv.2009.12547 · Cited by 353 · cited 1×
“Counterfactual thinking and causal inference have inspired several studies in computer vision, including visual explanations [20, 7, 42, 48], scene graph generation [12, 38], image recognition [38], video analysis [16, 26], zero-shot and few-shot learning [50, 49] incremental learning [22], representation learning [43…”
BackgroundSelf-citationCounterfactual Samples Synthesizing for Robust Visual Question Answering
Chen, L., Yan, X., Xiao, J., Zhang, H., Pu, S., & Zhuang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01081 · Cited by 341
Self-citationMUREL: Multimodal Relational Reasoning for Visual Question Answering
Cadene, R., Ben-younes, H., Cord, M., & Thome, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00209 · Cited by 330
Yin and Yang: Balancing and Answering Binary Visual Questions
Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., & Parikh, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.542 · Cited by 312
Analyzing the Behavior of Visual Question Answering Models
Agrawal, A., Batra, D., & Parikh, D. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1203 · Cited by 297
Wang, T., Huang, J., Zhang, H., & Sun, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01077 · Cited by 291
Self-citationInterventional Few-Shot LearningPreprint
Yue, Z., Zhang, H., Sun, Q., & Hua, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13000 · Cited by 282
Self-citationTaking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
Selvaraju, R. R., Lee, S., Shen, Y., Jin, H., Ghosh, S., Heck, L., Batra, D., & Parikh, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00268 · Cited by 251
An Analysis of Visual Question Answering Algorithms
Kafle, K., & Kanan, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.217 · Cited by 247
Semantics of causal DAG models and the identification of direct and indirect effects
Robins, J. M. (2003) · Highly Structured Stochastic Systems · doi:10.1093/oso/9780198510550.003.0007 · Cited by 237
Counterfactual Zero-Shot and Open-Set Visual Recognition
Yue, Z., Wang, T., Sun, Q., Hua, X.-S., & Zhang, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01515 · Cited by 200
Self-citationRUBi: Reducing Unimodal Biases in Visual Question Answering
Cadène, R., Dancette, C., Ben-younes, H., Cord, M., & Parikh, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10169 · Cited by 194
Self-Critical Reasoning for Robust Visual Question AnsweringPreprint
Wu, J., & Mooney, R. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09998 · Cited by 192
Counterfactual Vision and Language Learning
Abbasnejad, E., Teney, D., Parvaneh, A., Shi, Q., & van den Hengel, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01006 · Cited by 186
MUTANT: A Training Paradigm for Out-of-Distribution Generalization in Visual Question Answering
Gokhale, T., Banerjee, P., Baral, C. R., & Yang, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.63 · Cited by 181
Causal Attention for Vision-Language Tasks
Yang, X., Zhang, H., Qi, G., & Cai, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00972 · Cited by 175
Self-citationLearning to Contrast the Counterfactual Samples for Robust Visual Question Answering
Liang, Z., Jiang, W., Hu, H., & Zhu, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.265 · Cited by 175
Hendricks, L. A., Hu, R., Darrell, T., & Akata, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_17 · Cited by 171
Deconfounded Image Captioning: A Causal Retrospect
Yang, X., Zhang, H., & Cai, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3121705 · Cited by 169
Self-citationDistilling Causal Effect of Data in Class-Incremental Learning
Hu, X., Tang, K., Miao, C., Hua, X.-S., & Zhang, H. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00395 · Cited by 167
Self-citationCounterfactual Critic Multi-Agent Training for Scene Graph Generation
Chen, L., Zhang, H., Xiao, J., He, X., Pu, S., & Chang, S.-F. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00471 · Cited by 166
Self-citationTwo Causal Principles for Improving Visual Dialog
Qi, J., Niu, Y., Huang, J., & Zhang, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01087 · Cited by 152
Self-citationLearning What Makes a Difference from Counterfactual Examples and Gradient Supervision
Teney, D., Abbasnedjad, E., & van den Hengel, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58607-2_34 · Cited by 150
Overcoming Language Priors with Self-supervised Learning for Visual Question Answering
Zhu, X., Mao, Z., Liu, C., Zhang, P., Wang, B., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/151 · Cited by 128
Overcoming Language Priors in VQA via Decomposed Linguistic Representations
Jing, C., Wu, Y., Zhang, X., Jia, Y., & Wu, Q. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6776 · Cited by 122
SCOUT: Self-Aware Discriminant Counterfactual Explanations
Wang, P., & Vasconcelos, N. M. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00900 · Cited by 117
DeVLBert: Learning Deconfounded Visio-Linguistic Representations
Zhang, S., Jiang, T., Wang, T., Kuang, K., Zhao, Z., Zhu, J., Yu, J., Yang, H., & Wu, F. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413518 · Cited by 107
On the Value of Out-of-Distribution Testing: An Example of Goodhart's LawPreprint
Teney, D., Kafle, K., Shrestha, R., Abbasnejad, E., Kanan, C., & van den Hengel, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09241 · Cited by 105
Reducing Language Biases in Visual Question Answering with Visually-Grounded Question Encoder
Gouthaman, K. V., & Mittal, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_2 · Cited by 103
SSCR: Iterative Language-Based Image Editing via Self-Supervised Counterfactual Reasoning
Fu, T.-J., Wang, X. E., Grafton, S. T., Eckstein, M. P., & Wang, W. Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.357 · Cited by 49
Deconfounded Image Captioning: A Causal RetrospectPreprint
Yang, X., Zhang, H., & Cai, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.03923 · Cited by 45
Self-citationUnshuffling Data for Improved GeneralizationPreprint
Teney, D., Abbasnejad, E., & van den Hengel, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11894 · Cited by 38
Modularized Textual Grounding for Counterfactual Resilience
Fang, Z., Kong, S., Fowlkes, C. C., & Yang, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00654 · Cited by 36
Multimodal Explanations by Predicting Counterfactuality in Videos
Kanehira, A., Takemoto, K., Inayoshi, S., & Harada, T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00879 · Cited by 36
Cited by
Unbiased Scene Graph Generation From Biased Training
Tang, K., Niu, Y., Huang, J., Shi, J., & Zhang, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00377 · Cited by 755
Self-citationCausal Intervention for Weakly-Supervised Semantic Segmentation
Zhang, D., Zhang, H., Tang, J., Hua, X., & Sun, Q. (2020) · arXiv · doi:10.48550/arxiv.2009.12547 · Cited by 353
“Recently, there is a growing number of computer vision tasks that benefit from causality [34, 40, 51, 52, 56, 63, 65].”
BackgroundSelf-citationCross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering
Liu, Y., Li, G., & Lin, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3284038 · Cited by 348
Multi-Modal Knowledge Graph Construction and Application: A Survey
Zhu, X., Li, Z., Wang, X., Jiang, X., Sun, P., Wang, X., Xiao, Y., & Yuan, N. J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3224228 · Cited by 291
Towards artificial general intelligence via a multimodal foundation model
Fei, N., Lu, Z., Gao, Y., Yang, G., Huo, Y., Wen, J., Lu, H., Song, R., Gao, X., Xiang, T., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-30761-2 · Cited by 280
Self-citation
Metadata
- Authors
- Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X.-S., & Wen, J.-R.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 12695-12705
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3177934633
- Cited by
- 418
- References
- 57
- FWCI
- 26.68
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.