Faithful Multimodal Explanation for Visual Question Answering
Open accessAbstract
AI systems' ability to explain their reasoning is critical to their utility and trustworthiness.Deep neural networks have enabled significant progress on many challenging problems such as visual question answering (VQA).However, most of them are opaque black boxes with limited explanatory capability.This paper presents a novel approach to developing a high-performing VQA system that can elucidate its answers with integrated textual and visual explanations that faithfully reflect important aspects of its underlying reasoning process while capturing the style of comprehensible human explanations.Extensive experimental evaluation demonstrates the advantages of this approach compared to competing methods using both automated metrics and human evaluation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Faithful Multimodal (FM) model aims at grounding parts of generated explanations in the input image to provide explanations that are faithful to the input image.
“The Faithful Multimodal (FM) model [53] aims at grounding parts of generated explanations in the input image to provide explanations that are faithful to the input image.”
From CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations (2022) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
(2004) · Cited by 9,718 · see citing works
Efron, B., Hastie, T., Johnstone, I., & Tibshirani, R. (2004) · The Annals of Statistics · doi:10.1214/009053604000000067 · Cited by 9,357 · see citing works
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
(2005) · Cited by 4,152 · see citing works
Jain, S., & Wallace, B. C. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1357 · Cited by 2,287 · see citing works
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.10 · Cited by 2,036 · see citing works
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
Network Dissection: Quantifying Interpretability of Deep Visual Representations
Bau, D., Zhou, B., Khosla, A., Oliva, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.354 · Cited by 1,389 · see citing works
Hierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,303 · see citing works
Top-Down Neural Attention by Excitation Backprop
Zhang, J., Bargal, S. A., Lin, Z., Brandt, J., Shen, X., & Sclaroff, S. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-1059-x · Cited by 769 · see citing works
VQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · see citing works
Generating Visual Explanations
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_1 · Cited by 534 · see citing works
Attention is not ExplanationPreprint
Jain, S., & Wallace, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.10186 · Cited by 468 · see citing works
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence
Park, D. H., Hendricks, L. A., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., & Rohrbach, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00915 · Cited by 418 · see citing works
A Linear Time Histogram Metric for Improved SIFT Matching
Pele, O., & Werman, M. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88690-7_37 · Cited by 309 · see citing works
Learning to Segment Every Thing
Hu, R., Dollár, P., He, K., Darrell, T., & Girshick, R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00445 · Cited by 292 · see citing works
Explaining Recommendations: Satisfaction vs. Promotion
Bilgic, M., & Mooney, R. J. (2005) · Cited by 252 · see citing works
Self-citationHendricks, L. A., Hu, R., Darrell, T., & Akata, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_17 · Cited by 171 · see citing works
Pythia v0.1: the Winning Entry to the VQA Challenge 2018Preprint
Jiang, Y. L., Natarajan, V., Chen, X., Rohrbach, M., Batra, D., & Parikh, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.09956 · Cited by 150 · see citing works
Self-Critical Reasoning for Robust Visual Question AnsweringPreprint
Wu, J., & Mooney, R. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09998 · Cited by 123 · see citing works
Self-citationDynamic Filtering with Large Sampling Field for ConvNets
Wu, J., Dai, L., Yu, Y., Bajaj, C., & Ji, X. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01249-6_12 · Cited by 47 · see citing works
Self-citationJoint Image Captioning and Question AnsweringPreprint
Wu, J., Hu, Z., & Mooney, R. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08389 · Cited by 12 · see citing works
Self-citationHendricks, L. A., Hu, R., Darrell, T., & Akata, Z. (2018) · Max Planck Digital Library · doi:10.48550/arxiv.1807.09685
Flagged first, then by citations
Research works printed but not linked: 2
- Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question an- swering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425-2433.
- Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. 2013. Rectifier Nonlinearities Improve Neural Net- work Acoustic Models. In International Conference on Machine Learning, volume 30, page 3.
Cited by
Sorted by citations
Survey of Explainable AI Techniques in Healthcare
Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023) · Sensors · doi:10.3390/s23020634 · Cited by 591 · see citing works
Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?
Jacovi, A., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.386 · Cited by 503 · see citing works
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
Measuring Association Between Labels and Free-Text Rationales
Wiegreffe, S., Marasović, A., & Smith, N. A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.804 · Cited by 119 · see citing works
Dynamic Context-guided Capsule Network for Multimodal Machine Translation
Lin, H., Meng, F., Su, J., Yin, Y., Yang, Z., Ge, Y., Zhou, J., & Luo, J. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413715 · Cited by 83 · see citing works
Rodis, N., Sardianos, C., Grammatikis, P. I. R., Sarigiannidis, P. G., Varlamis, I., & Papadopoulos, G. T. (2024) · IEEE Access · doi:10.1109/access.2024.3467062 · Cited by 84 · see citing works
Explainable Activity Recognition for Smart Home Systems
Das, D., Nishimura, Y., Vivek, R. P., Takeda, N., Fish, S. T., Plötz, T., & Chernova, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3561533 · Cited by 70 · see citing works
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks
Kayser, M., Camburu, O.-M., Salewski, L., Emde, C., Do, V., Akata, Z., & Lukasiewicz, T. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00128 · Cited by 68 · see citing works
NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Tasks
Sammani, F., Mukherjee, T., & Deligiannis, N. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00814 · Cited by 59 · see citing works
Few-Shot Self-Rationalization with Natural Language Prompts
Marasović, A., Beltagy, I., Downey, D., & Peters, M. E. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.31 · Cited by 64 · see citing works
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54 · see citing works
Marasović, A., Bhagavatula, C., Park, J. S., Bras, R. J. L., Smith, N. A., & Choi, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.253 · Cited by 52 · see citing works
Local Interpretations for Explainable Natural Language Processing: A Survey
Luo, S., Ivison, H., Han, S. C., & Poon, J. (2024) · ACM Computing Surveys · doi:10.1145/3649450 · Cited by 60 · see citing works
Generating Question Relevant Captions to Aid Visual Question Answering
Wu, J., Hu, Z., & Mooney, R. J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1348 · Cited by 49 · see citing works
Self-citationAdvisable Learning for Self-Driving Vehicles by Internalizing Observation-to-Action Rules
Kim, J., Moon, S., Rohrbach, A., Darrell, T., & Canny, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00968 · Cited by 45 · see citing works
Effects of AI and Logic-Style Explanations on Users’ Decisions Under Different Levels of Uncertainty
Cau, F. M., Hauptmann, H., Spano, L. D., & Tintarev, N. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3588320 · Cited by 41 · see citing works
Diagnosing Vision-and-Language Navigation: What Really Matters
Zhu, W., Qi, Y., Narayana, P., Sone, K., Basu, S., Wang, X., Wu, Q., Eckstein, M. P., & Wang, W. Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.438 · Cited by 35 · see citing works
Xue, D., Qian, S., & Xu, C. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3398012 · Cited by 32 · see citing works
Wang, Z., Zhou, Y., Guo, Q., & Sun, H. (2021) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2021.3091710 · Cited by 31 · see citing works
One-class learning for fake news detection through multimodal variational autoencoders
Gôlo, M. P. S., de Souza, M. C., Rossi, R. G., Rezende, S. O., Nogueira, B. M., & Marcacini, R. M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106088 · Cited by 32 · see citing works
Explainable Image Classification: The Journey So Far and the Road Ahead
Kamakshi, V., & Krishnan, N. C. (2023) · AI · doi:10.3390/ai4030033 · Cited by 31 · see citing works
Faithfully Explainable Recommendation via Neural Logic Reasoning
Zhu, Y., Xian, Y., Fu, Z., de Melo, G., & Zhang, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.245 · Cited by 31 · see citing works
Joint Answering and Explanation for Visual Commonsense Reasoning
Li, Z., Guo, Y., Wang, K., Wei, Y., Nie, L., & Kankanhalli, M. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3286259 · Cited by 27 · see citing works
REX: Reasoning-aware and Grounded Explanation
Chen, S., & Zhao, Q. G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01514 · Cited by 25 · see citing works
Variational Causal Inference Network for Explanatory Visual Question Answering
Xue, D., Qian, S., & Xu, C. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00238 · Cited by 25 · see citing works
Toward explainable and advisable model for self‐driving cars
Kim, J., Rohrbach, A., Akata, Z., Moon, S., Misu, T., Chen, Y., Darrell, T., & Canny, J. (2021) · Applied AI Letters · doi:10.1002/ail2.56 · Cited by 24 · see citing works
Knowledge-Augmented Visual Question Answering With Natural Language Explanation
Xie, J., Cai, Y., Chen, J., Xu, R., Wang, J., & Li, Q. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3379900 · Cited by 22 · see citing works
Grounding Visual Representations with Texts for Domain Generalization
Min, S., Park, N., Kim, S., Park, S., & Kim, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_3 · Cited by 22 · see citing works
Why do you think that? Exploring Faithful Sentence-Level Rationales Without Supervision
Glockner, M., Habernal, I., & Gurevych, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.97 · Cited by 24 · see citing works
CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
Salewski, L., Koepke, A. S., Lensch, H. P. A., & Akata, Z. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-04083-2_5 · Cited by 20 · see citing works
Reasoning Models Don't Always Say What They Think
Chen, Y., Benton, J., Radhakrishnan, A., Uesato, J., Denison, C., Schulman, J., Somani, A., Hase, P., Wagner, M., Roger, F., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.05410 · Cited by 18 · see citing works
“Different from plausibility, faithfulness evaluates whether the CoT accurately reveals the model’s internal reasoning process behind responding to an input (Ribeiro et al., 2016; Gilpin et al., 2018; Wu and Mooney, 2019; Jacovi and Goldberg, 2020; Lanham et al., 2023; Turpin et al., 2023).”
Employing Co-Learning to Evaluate the Explainability of Multimodal Sentiment Analysis
Jain, D. K., Rahate, A., Joshi, G., Walambe, R. A., & Kotecha, K. V. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3176403 · Cited by 17 · see citing works
A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations
Rao, V. N., Zhen, X., Hovsepian, K., & Shen, M. (2021) · doi:10.18653/v1/2021.maiworkshop-1.4 · Cited by 17 · see citing works
Rationale-Inspired Natural Language Explanations with Commonsense.Preprint
Majumder, B. P., Camburu, O.-M., Lukasiewicz, T., & McAuley, J. (2021) · arXiv (Cornell University) · Cited by 16 · see citing works
Connecting Attributions and QA Model Behavior on Realistic Counterfactuals
Ye, X., Nair, R., & Durrett, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.447 · Cited by 13 · see citing works
Explaining Deep Convolutional Neural Networks via Latent Visual-Semantic Filter Attention
Yang, Y., Kim, S., & Joo, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00815 · Cited by 13 · see citing works
Yang, Q., Li, Y., Hu, B., Ma, L., Ding, Y., & Zhang, M. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548284 · Cited by 12 · see citing works
Interpretable visual reasoning: A survey
He, F., Wang, Y., Miao, X., & Sun, X. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2021.104194 · Cited by 11 · see citing works
Segmentation-Guided Attention for Visual Question Answering from Remote Sensing Images
Tosato, L., Boussaid, H., Weissgerber, F., Kurtz, C., Wendling, L., & Lobry, S. (2024) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss53475.2024.10642082 · Cited by 10 · see citing works
LININ: Logic Integrated Neural Inference Network for Explanatory Visual Question Answering
Xue, D., Qian, S., Fang, Q., & Xu, C. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3521709 · Cited by 10 · see citing works
Shekarpour, S., Nadgeri, A., & Singh, K. (2020) · doi:10.18653/v1/2020.intexsempar-1.1 · Cited by 9 · see citing works
Towards More Faithful Natural Language Explanation Using Multi-Level Contrastive Learning in VQA
Lai, C., Song, S., Meng, S., Li, J., Yan, S., & Hu, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i3.28065 · Cited by 9 · see citing works
Mixture of causal experts: A causal perspective to build dual-level mixture-of-experts models
Gao, Y., Xu, N., Tian, H., Zheng, B., Cao, J., & Liu, A.-A. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127422 · Cited by 8 · see citing works
GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths
Chen, X., Jiang, M. X., & Zhao, Q. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73242-3_18 · Cited by 7 · see citing works
Zhu, H., Togo, R., Ogawa, T., & Haseyama, M. (2023) · Electronics · doi:10.3390/electronics12102183 · Cited by 8 · see citing works
Evaluating Explanations for Reading Comprehension with Realistic Counterfactuals.Preprint
Ye, X., Nair, R., & Durrett, G. (2021) · arXiv (Cornell University) · Cited by 6 · see citing works
A Study of the Plausibility of Attention between RNN Encoders in Natural Language Inference
Nguyen, D. H., Gravier, G., & Sébillot, P. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00259 · Cited by 6 · see citing works
S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning
Suo, W., Sun, M., Liu, W., Gao, Y., Wang, P., Zhang, Y., & Wu, Q. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00260 · Cited by 6 · see citing works
Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations
Chen, Y., Zhong, R., Ri, N., Zhao, C., He, H., Steinhardt, J., Yu, Z., & McKeown, K. (2023) · arXiv · doi:10.48550/arxiv.2307.08678 · Cited by 5 · see citing works
“It is different from faithfulness, which measures whether an explanation is consistent with the model’s own de- cision process (Harrington et al., 1985; Ribeiro et al., 2016; Gilpin et al., 2018; Wu and Mooney, 2019; Lakkaraju et al., 2019; Jacovi and Goldberg, 2020).”
ResultFew-Shot Multimodal Explanation for Visual Question Answering
Xue, D., Qian, S., & Xu, C. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681597 · Cited by 5 · see citing works
Uni-NLX: Unifying Textual Explanations for Vision and Vision-Language Tasks
Sammani, F., & Deligiannis, N. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00498 · Cited by 5 · see citing works
Ferguson, W., Batra, D., Mooney, R. J., Parikh, D., Torralba, A. B., Bau, D., Diller, D., Fasching, J., Fiotto-Kaufman, J., Goyal, Y., et al. (2021) · Applied AI Letters · doi:10.1002/ail2.60 · Cited by 5 · see citing works
Self-citationMultimodal Rationales for Explainable Visual Question Answering
Li, K., George Vosselman, A., & Yang, M. Y. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00024 · Cited by 4 · see citing works
Synchronization-Inspired Interpretable Neural Networks
Han, W., Qin, Z., Liu, J., Böhm, C., & Shao, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3297672 · Cited by 4 · see citing works
From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation
Ge, J., Subramanian, S., Darrell, T., & Li, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.75 · Cited by 4 · see citing works
Formal Definition of Interpretability and Explainability in XAI
Dib, L. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-66431-1_9 · Cited by 3 · see citing works
Towards a Unified Model for Generating Answers and Explanations in Visual Question Answering
Whitehouse, C., Weyde, T., & Madhyastha, P. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.126 · Cited by 3 · see citing works
SOrT-ing VQA Models : Contrastive Gradient Learning for Improved Consistency
Dharur, S., Tendulkar, P., Batra, D., Parikh, D., & Selvaraju, R. R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.248 · Cited by 3 · see citing works
Faithfully Explainable Recommendation via Neural Logic ReasoningPreprint
Zhu, Y., Xian, Y., Fu, Z., de Melo, G., & Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07869 · Cited by 3 · see citing works
Black-Box Attacks on Image Activity Prediction and its Natural Language Explanations
Baia, A. E., Poggioni, V., & Cavallaro, A. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00396 · Cited by 2 · see citing works
An Explainable Vision Question Answer Model via Diffusion Chain-of-Thought
Lu, C., Lü, Q., & Luo, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72855-6_9 · Cited by 2 · see citing works
Объяснительный искусственный интеллект в моделях поддержки принятия решений для Здравоохранения 5.0
Averkin, A. N., & Yarushev, S. (2023) · Computer Tools in Education · doi:10.32603/2071-2340-2023-2-41-61 · Cited by 3 · see citing works
Interpretable Visual Question Answering Referring to Outside Knowledge
Zhu, H., Togo, R., Ogawa, T., & Haseyama, M. (2023) · Proceedings - International Conference on Image Processing · doi:10.1109/icip49359.2023.10222423 · Cited by 3 · see citing works
Shekarpour, S., Nadgeri, A., & Singh, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08323 · Cited by 2 · see citing works
Towards explainable visual question answering via cross-modal causal reasoning
Li, W., Deng, F., & Li, Z. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.113765 · Cited by 1 · see citing works
Retrieval-Augmented Natural Language Reasoning for Explainable Visual Question Answering
Lim, S. H., Kim, M., Kim, H. B., & Kim, S. (2024) · IEEE International Conference on Image Processing Workshops (ICIPW) · doi:10.1109/icipcw64161.2024.10769112 · Cited by 1 · see citing works
Designing Multimodal Datasets for NLP ChallengesPreprint
Pustejovsky, J. D., Holderness, E., Tu, J., Glenn, P., Rim, K., Lynch, K., & Brutti, R. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05999 · Cited by 4 · see citing works
Connecting Attributions and QA Model Behavior on Realistic CounterfactualsPreprint
Ye, X., Nair, R., & Durrett, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04515 · Cited by 4 · see citing works
Rio-Torto, I., Cardoso, J. S., & Teixeira, L. F. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-04881-4_5
Explaining Multimodal Image Retrieval Using A Vision and Language Task Model
Sarker, M. I., Milanova, M. G., & Talburt, J. R. (2023) · Advances in intelligent systems and computing · doi:10.1007/978-3-031-28332-1_40
Opportunities and challenges of explainable artificial intelligence in medicine
Yen, A.-Z., Wu, C.-K., & Chen, H. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-99136-0.00009-x · Cited by 1 · see citing works
MERGE: Multi-Entity Relational Reasoning Based Explanation in Visual Question Answering
Pan, M., Li, Q., & Tian, Q. (2023) · IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech) · doi:10.1109/dasc/picom/cbdcom/cy59711.2023.10361383
MSGeN: Multimodal Selective Generation Network for Grounded Explanations
Li, D., Chen, W., & Lin, X. A. (2023) · Electronics · doi:10.3390/electronics13010152
On the Risk of Misleading Reports: Diagnosing Textual Biases in Multimodal Clinical AI
Restrepo, D. S., Ktena, S. I., Vakalopoulou, M., Christodoulidis, S., & Ferrante, E. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06004-4_32
Visual Question Answering Challenges: A Review
Nuseir, A., & Ebner, M. (2025) · International Conference on Embedded Systems and Artificial Intelligence (ESAI) · doi:10.1109/esai67033.2025.11438832
- Authors
- Wu, J., & Mooney, R.
- Affiliations
- The University of Texas at Austin (US)
- Venue
- Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
- Published
- 2019
- Pages
- 103-112
- OpenAlex
- W2973009097
- Cited by
- 73
- References
- 32
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 12 · 2023: 20 · 2022: 11 · 2021: 17 · 2020: 7 · 2019: 1
- FWCI
- 3.44
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Defense Advanced Research Projects Agency
- Air Force Research Laboratory
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- visual question answering, multimodal explanations, explainable AI, model interpretability, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
