CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
Open accessThe DOI matches its Crossref record; 63 of the 67 references it cites were found in a registry or an index; 6 of the 26 sources its claims cite were checked for support.
Abstract
Providing explanations in the context of Visual Question Answering (VQA) presents a fundamental problem in machine learning. To obtain detailed insights into the process of generating natural language explanations for VQA, we introduce the large-scale CLEVR-X dataset that extends the CLEVR dataset with natural language explanations. For each image-question pair in the CLEVR dataset, CLEVR-X contains multiple structured textual explanations which are derived from the original scene graphs. By construction, the CLEVR-X explanations are correct and describe the reasoning and visual information that is necessary to answer a given question. We conducted a user study to confirm that the ground-truth explanations in our proposed dataset are indeed complete and relevant. We present baseline results for generating natural language explanations in the context of VQA using two state-of-the-art frameworks on the CLEVR-X dataset. Furthermore, we provide a detailed analysis of the explanation generation quality for different question and answer types. Additionally, we study the influence of using different numbers of ground-truth explanations on the convergence of natural language generation (NLG) metrics. The CLEVR-X dataset is publicly available at https://github.com/ExplainableML/CLEVR-X.
Overview
CoChat Summary
TL;DR
CLEVR-X extends CLEVR with 3.6 million structured natural language explanations for 850k question-image pairs, generated from scene graphs and validated by a user study. Baselines with PJ-X and FM show strong explanation scores on CLEVR-X, while counting and binary questions remain harder to explain.
Key findings
- CLEVR-X contains 85,000 images, 849,948 questions, and 3,645,137 explanations, with an average of 4.29 explanations per question and 21.53 words per explanation; vocabulary is only 96 words.
- In the MTurk user study, participants selected complete explanations with 92.19% average accuracy and relevant explanations with 92.52% average accuracy; performance was weaker for complex types such as compare-integer (77.00% completeness, 83.67% relevance) and comparison (83.67% completeness, 87.33% relevance).
- PJ-X achieved METEOR 58.9, CIDEr 639.8, and answer accuracy 63.0% on CLEVR-X; FM achieved METEOR 52.5, CIDEr 566.8, and answer accuracy 80.3%.
- Explanation quality was lower for counting answers (METEOR 57.3) than attribute answers (63.3), and lower for binary questions (METEOR 55.6); BLEU-4 needed close to 3 ground-truth explanations to reach 95% of its final value, while CIDEr reached 95.7% with only one.
Methods
- The dataset was built by extending CLEVR: for each question-image pair, the functional program was traced on the scene graph, relevant object attributes were filtered, and natural language templates with synonym sampling generated up to 10 explanations per question.
- A user study on Amazon Mechanical Turk used two HIT types (completeness and relevance), sampling 100 explanations from each of 9 question types for a total of 1800 samples; each sample was rated by 3 workers, with 78 workers in completeness and 101 in relevance (134 total participants).
- Baselines used PJ-X and FM with ResNet-101 image grid features; evaluation used BLEU, METEOR, ROUGE-L, and CIDEr, reported for correctly answered questions.
Limitations
- The paper states that only two natural language generation methods were considered as baselines, and that benchmarking additional recent frameworks is a next step.
- The paper notes the synthetic nature and limited domain of CLEVR, resulting in a very small vocabulary of 96 words; this is evident in the dataset construction and may limit transfer to natural images or human-written explanations.
- The user study showed weaker and variable performance on complex question types and across individual workers, which the paper reports; this suggests the validation is strongest for simpler explanations and question types.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
3 Valid · 2 Not valid · 16 Not checked · 95 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Explanation datasets commonly used in the context of VQA, such as the VQA-X dataset [26] or the e-SNLI-VE dataset [13,29] for visual entailment, contain explanations of widely varying quality since they are generated by humans.”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 1 Introduction
What the sources say
- Partly supportedcites [26] 26. Park, D.H., et al.: Multimodal explanations: justifying decisions and pointing to the evidence. In: CVPR, pp. 8779-8788 (2018) sourceExperimental — accuracy not yet measured
The Introduction identifies VQA-X as a dataset of human-annotated explanations, supporting the part attributed to this citation about VQA-X and its human origin. The VQA-X dataset section describes collecting textual explanations, but neither it nor the Experimental Setup establishes that their quality varies widely because humans generated them. This citation does not address the separately cited e-SNLI-VE dataset. (LLM, conf=0.86)
- Not checked yetcites [13] 13. Do, V., Camburu, O.M., Akata, Z., Lukasiewicz, T.: e-SNLI-VE-2.0: corrected visual-textual entailment with natural language explanations. arXiv preprint arXiv:2004.03744 (2020)
- Partly supportedcites [29] 29. Kayser, M., Camburu, O.M., Salewski, L., Emde, C., Do, V., Akata, Z., Lukasiewicz, T.: e-VIL: a dataset and benchmark for natural language explanations in vision-language tasks. In: ICCV, pp. 1244-1254 (2021) sourceExperimental — accuracy not yet measured
Section 4.2 (p. 4) reports varying human-evaluated quality for ground-truth explanations in both VQA-X and e-SNLI-VE, and Sections 3.2 and 4.2 identify their human-written explanations. The paper does not attribute that variation simply to human authorship; for e-SNLI-VE, it points to explanations adapted from a text-based task, imperfect filtering, and task ambiguity. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The groundtruth explanations in VQA-X and e-SNLI-VE can range from statements that merely describe an image to explaining the reasoning about the question and image involving prior information, such as common knowledge.”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 1 Introduction
What the sources say
- Partly supportedcites [26] 26. Park, D.H., et al.: Multimodal explanations: justifying decisions and pointing to the evidence. In: CVPR, pp. 8779-8788 (2018) sourceExperimental — accuracy not yet measured
Our copy of Park et al. (2018) supports only the VQA-X half of the claim, and even there at a weaker level: the 'VQA-X.' qualitative section states that textual justifications 'are able to both capture common sense and discuss specific image parts important for answering a question' (e.g. the concept of a zoo as 'animals in an enclosure' vs. 'foam on the waves'), and the 'VQA Explanation Dataset (VQA-X)' section notes VQA requires 'understanding vision, language, and commonsense knowledge'. However, that passage concerns the model's generated justifications rather than a stated range of the human ground-truth annotations, and the dataset section in fact describes deliberately excluding trivially descriptive questions ('What is the color of the banana?', age-9+ filter) — so 'statements that merely describe an image' is not what this paper attributes to its ground truth. Nothing in our copy addresses e-SNLI-VE at all (searched: e-SNLI, visual entailment, explanation dataset ranges); that dataset is from later work and this paper only introduces VQA-X and ACT-X. The e-SNLI-VE portion, and the descriptive-vs-reasoning spectrum framed as a property of ground-truth annotations across both datasets, are therefore not supported by this source. (LLM, conf=0.75)
- Not checked yetcites [13] 13. Do, V., Camburu, O.M., Akata, Z., Lukasiewicz, T.: e-SNLI-VE-2.0: corrected visual-textual entailment with natural language explanations. arXiv preprint arXiv:2004.03744 (2020)
- Partly supportedcites [29] 29. Kayser, M., Camburu, O.M., Salewski, L., Emde, C., Do, V., Akata, Z., Lukasiewicz, T.: e-VIL: a dataset and benchmark for natural language explanations in vision-language tasks. In: ICCV, pp. 1244-1254 (2021) sourceExperimental — accuracy not yet measured
Appendix A, Qualitative Examples (p. 12) does characterize the ground-truth explanations of VQA-X and e-SNLI-VE along the image-dependence axis: VQA-X examples where 'a generic explanation, that is not necessarily grounded in the image, will often suffice', versus e-SNLI-VE examples that 'require the explanations to describe image-specific characteristics'. But the upper end of the claimed range — reasoning involving prior information such as common knowledge — is attributed by this paper to VCR, not to VQA-X or e-SNLI-VE ('VCR has many questions that require substantial commonsense reasoning and general knowledge', p. 12; Figure 1 / Related Work, p. 2). The paper in fact states the opposite for these two datasets: e-SNLI-VE 'contains over 430k instances for which the explanations rely on the image content' (p. 2) and many VQA-X NLEs 'are trivial and could be guessed without looking at the image' (p. 2), reinforced by the human evaluation finding that VQA-X explanations are 'easier and less elaborate' (5.3.1, p. 7). So the part of the claim extending to prior/common-knowledge reasoning in VQA-X and e-SNLI-VE is not supported. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The e-SNLI-VE dataset contains numerous explanation samples which consist of repeated statements (”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 1 Introduction
What the sources say
- Not checked yetcites [13] 13. Do, V., Camburu, O.M., Akata, Z., Lukasiewicz, T.: e-SNLI-VE-2.0: corrected visual-textual entailment with natural language explanations. arXiv preprint arXiv:2004.03744 (2020)
- Partly supportedcites [29] 29. Kayser, M., Camburu, O.M., Salewski, L., Emde, C., Do, V., Akata, Z., Lukasiewicz, T.: e-VIL: a dataset and benchmark for natural language explanations in vision-language tasks. In: ICCV, pp. 1244-1254 (2021) sourceExperimental — accuracy not yet measured
Section 3.2 (p. 3) does document this failure mode: the similarity filter was introduced because 'the share of low-quality explanations is highest for entailment examples... when the textual premise and hypothesis are almost identical, as then the explanation often just repeats both statements', and Appendix B.4 (pp. 16-17, Fig. 13) gives a concrete tautological example. But what our copy attributes to the released e-SNLI-VE is removal of these cases (all pairs with premise-hypothesis ROUGE-1 > 0.57 deleted, cutting low-quality entailment explanations by 4.2%), plus a general caveat that the filters were 'effective, but imperfect' leaving residual noise (Table 2, p. 4: 10.3% 'No', 6.9% 'Weak No'; e-SNLI's 9.6% explanation error rate as an upper bound of cleaning, p. 16). The paper nowhere states that the final dataset still contains 'numerous' repeated-statement samples, so the quantity/target-of-claim part is unsupported. The '(x because x)' phrasing also loosely echoes p. 7 (Sec. 5.3.1), where nonsensical explanations like 'a woman is a woman' (~10% prevalence) are reported for model-generated, not dataset, explanations. (LLM, conf=0.75)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 4.2 Evaluating Explanations Generated by State-of-the-Art Methods
What the sources say
- Could not verifycites [16] 16. Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multimodal com- pact bilinear pooling for visual question answering and visual grounding. In: EMNLP, pp. 457-468 (2016) sourceExperimental — accuracy not yet measured
Our copy of Fukui et al. (2016) never mentions CLEVR or its publication timing: the Datasets sections (4.1 VQA real-image/Visual Genome/Visual7W; 5.1 Flickr30k Entities/ReferItGame) and the Related Work (Sec. 2) and Conclusion (Sec. 6) contain no such statement, and the reference list carries no Johnson/CLEVR entry (searched: 'CLEVR', 'synthetic rendered scenes Cactus Johnson', 'Johnson clevr', 'PJ-X backbone parallel joint crossmodal'). The term 'PJ-X' does not appear anywhere in our copy either — the paper only proposes MCB pooling with attention for VQA and grounding. A chronological claim that MCB preceded CLEVR's publication is therefore outside what the source itself says; nothing in our copy contradicts it, but nothing supports it. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately, VQA-X and e-SNLI-VE contain spelling errors, resulting in multiple versions of the same words.”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 3.3 Dataset Analysis
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
58 more claims
Create a free account to see all 116 claims and to verify the 20 references still open.
- Probably not a substantial claim. 32
- It may state two separate claims as one. 29
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Natural language explanations [27,67], quantitative measures of the quality of explanations [68,69], and models that support their learning process with formal symbolic basis such as language, rules, compositional relationships or knowledge graphs (neural-symbolic learning and reasoning [66]) are key for explanations to be understood by non-expert audience.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [67]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Approaches without human feedback, for example, because they use an artificial benchmark with defined ground truth for the desired explanations [18,32,36], suffer from a direct proof of being transferable to real-world use cases.”
From Study on the Helpfulness of Explainable Artificial Intelligence (2024) · cites this paper as [36]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Total 28,180 32,886 41,817 10,315 1.48 10.64 7.49”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 3.3 Dataset Analysis
Reported 1.48 · recomputed 1.271575
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We extracted 14 × 14 × 1024 grid features for the images in the CLEVR-X dataset using a ResNet-101 [19], pre-trained on Ima-geNet [12].”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“For PJ-X, our best model was trained for 52 epochs, using the Adam optimizer [33] with a learning rate of 0.0002 and a batch size of 256.”
MethodInfluentialImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“We extracted 14 × 14 × 1024 grid features for the images in the CLEVR-X dataset using a ResNet-101 [19], pre-trained on Ima-geNet [12].”
MethodGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1×
“To evaluate the quality of the generated explanations, we use the standard natural language generation metrics BLEU [39], METEOR [9], ROUGE-L [36] and CIDEr [50].”
MethodVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
(2004) · Cited by 9,718
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“Decisions made by neural networks can be visually explained with visual attribution that is determined by introspecting trained networks and their features [46,57,43,8,58], by using input perturbations [42,14,15], or by training a probabilistic feature attribution model along with a task-specific CNN [30].”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948 · cited 1×
“It is based on the Up-Down VQA model [4].”
MethodCIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 1×
“To evaluate the quality of the generated explanations, we use the standard natural language generation metrics BLEU [39], METEOR [9], ROUGE-L [36] and CIDEr [50].”
MethodOn Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · cited 1×
“Decisions made by neural networks can be visually explained with visual attribution that is determined by introspecting trained networks and their features [46,57,43,8,58], by using input perturbations [42,14,15], or by training a probabilistic feature attribution model along with a task-specific CNN [30].”
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
(2005) · Cited by 4,152
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“For a comprehensive overview of research on explainability and interpretability, we refer to recent surveys [7,10,17].”
ResultFilm: Visual reasoning with a general conditioning layer
Perez, E., Strub, F., De Vries, H., Dumoulin, V., & Courville, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11671 · Cited by 2,490
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.10 · Cited by 1,998 · cited 1×
“The VQA [5] task has been addressed by several works that apply attention mechanisms to text and image features [16,45,55,56,60].”
MethodCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · cited 3×
“It combines a modified MCB [16] framework, pre-trained on the VQA v2 dataset, with a visual pointing and textual justification module.”
MethodResultGQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00686 · Cited by 1,451 · cited 1×
“This has been further investigated in more controlled dataset settings, such as the CLEVR [27], VQA-CP [2], and GQA [25] datasets.”
CAPTCHA: Using Hard AI Problems for Security
von Ahn, L., Blum, M., Hopper, N. J., & Langford, J. (2003) · Lecture Notes in Computer Science · doi:10.1007/3-540-39200-9_18 · Cited by 1,447 · cited 2×
“Each HIT was made up of (1) An explanation of the task; (2) A non-trivial example, where the correct answers are already selected; (3) A CAPTCHA [3] to verify that the user is human; (4) The problem definition consisting of a question and an image; (5) A user qualification step, for which the user has to correctly answe…”
MethodInterpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097 · cited 1×
“…made by neural networks can be visually explained with visual attribution that is determined by introspecting trained networks and their features [8,43,46,57,58], by using input perturbations [14,15,42], or by training a probabilistic feature attribution model along with a task-specific CNN [30].”
MethodVisual7W: Grounded Question Answering in Images
Zhu, Y., Groth, O., Bernstein, M., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.540 · Cited by 794 · cited 1×
“The VQA [5] task has been addressed by several works that apply attention mechanisms to text and image features [16,45,55,56,60].”
MethodAsk, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Xu, H., & Saenko, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_28 · Cited by 727 · cited 1×
“The VQA [5] task has been addressed by several works that apply attention mechanisms to text and image features [16,45,55,56,60].”
MethodTop-Down Neural Attention by Excitation Backprop
Zhang, J., Bargal, S. A., Lin, Z., Brandt, J., Shen, X., & Sclaroff, S. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-017-1059-x · Cited by 680 · cited 1×
“…made by neural networks can be visually explained with visual attribution that is determined by introspecting trained networks and their features [8,43,46,57,58], by using input perturbations [14,15,42], or by training a probabilistic feature attribution model along with a task-specific CNN [30].”
MethodVQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · cited 1×
“The VQA [5] task has been addressed by several works that apply attention mechanisms to text and image features [16,45,55,56,60].”
MethodDon't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00522 · Cited by 566
Understanding Deep Networks via Extremal Perturbations and Smooth Masks
Fong, R., Patrick, M., & Vedaldi, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00304 · Cited by 538 · cited 1×
“…made by neural networks can be visually explained with visual attribution that is determined by introspecting trained networks and their features [8,43,46,57,58], by using input perturbations [14,15,42], or by training a probabilistic feature attribution model along with a task-specific CNN [30].”
MethodWhere to Look: Focus Regions for Visual Question Answering
Shih, K. J., Singh, S., & Hoiem, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.499 · Cited by 493
Inferring and Executing Programs for Visual Reasoning
Johnson, J., Hariharan, B., Van Der Maaten, L., Hoffman, J., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.325 · Cited by 467 · cited 1×
“Multiple frameworks have been proposed to address the CLEVR task [23,24,28,41,44,47].”
BackgroundMultimodal Explanations: Justifying Decisions and Pointing to the Evidence
Park, D. H., Hendricks, L. A., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., & Rohrbach, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00915 · Cited by 418
Self-citatione-SNLI: Natural language inference with natural language explanations
Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., & Blunsom, P. (2018) · arXiv · doi:10.48550/arxiv.1812.01193 · Cited by 367
Textual Explanations for Self-Driving Vehicles
Kim, J., Rohrbach, A., Darrell, T., Canny, J., & Akata, Z. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_35 · Cited by 337
Self-citationYin and Yang: Balancing and Answering Binary Visual Questions
Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., & Parikh, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.542 · Cited by 312 · cited 1×
“However, recent works observed that the question-answer bias in common VQA datasets can be exploited in order to answer questions without leveraging any visual information [1,2,27,59].”
BackgroundToward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306 · cited 1×
“For a comprehensive overview of research on explainability and interpretability, we refer to recent surveys [7,10,17].”
ResultAnalyzing the Behavior of Visual Question Answering Models
Agrawal, A., Batra, D., & Parikh, D. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1203 · Cited by 297 · cited 1×
“However, recent works observed that the question-answer bias in common VQA datasets can be exploited in order to answer questions without leveraging any visual information [1,2,27,59].”
BackgroundLearning by Abstraction: The Neural State MachinePreprint
Hudson, D. A., & Manning, C. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.03950 · Cited by 262 · cited 1×
“Multiple frameworks have been proposed to address the CLEVR task [23,24,28,41,44,47].”
BackgroundAdvances in Cryptology — EUROCRYPT 2003
(2003) · Lecture Notes in Computer Science · doi:10.1007/3-540-39200-9 · Cited by 241
Explainable and Explicit Visual Reasoning Over Scene Graphs
Shi, J., Zhang, H., & Li, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00857 · Cited by 223 · cited 2×
“Multiple frameworks have been proposed to address the CLEVR task [23,24,28,41,44,47].”
BackgroundMethodRISE: Randomized Input Sampling for Explanation of Black-box Models
Petsiuk, V., Das, A., & Saenko, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07421 · Cited by 196 · cited 1×
“Decisions made by neural networks can be visually explained with visual attribution that is determined by introspecting trained networks and their features [46,57,43,8,58], by using input perturbations [42,14,15], or by training a probabilistic feature attribution model along with a task-specific CNN [30].”
CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations
Arras, L., Osman, A., & Samek, W. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.008 · Cited by 179 · cited 1×
“Additionally, [6] proposed the CLEVR-XAI-simple and CLEVR-XAI-complex datasets which provide ground-truth segmentation information for heatmap-based visual explanations.”
MethodFleet, D. J., Tuytelaars, T., Schiele, B., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10590-1 · Cited by 176
Hendricks, L. A., Hu, R., Darrell, T., & Akata, Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_17 · Cited by 171
Self-citationVisual Entailment: A Novel Task for Fine-Grained Image Understanding
Xie, N., Lai, F., Doran, D., & Kadav, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06706 · Cited by 149 · cited 1×
“The e-SNLI-VE [29,13] dataset combines human explanations from e-SNLI [11] and the image-sentence pairs for the Visual Entailment task from SNLI-VE [54].”
MethodVQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
Li, Q., Tao, Q., Joty, S., Cai, J., & Luo, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_34 · Cited by 123
Compositional Attention Networks for Machine Reasoning
Hudson, D. A., & Manning, C. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.03067 · Cited by 117
CLEVR-Ref+: Diagnosing Visual Reasoning With Referring Expressions
Liu, R., Liu, C., Bai, Y., & Yuille, A. L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00431 · Cited by 109 · cited 1×
“There have been numerous dataset extensions for the CLEVR dataset, for instance to measure the generalization capabilities of models pre-trained on CLEVR (CLOSURE [51]), to evaluate object detection and segmentation (CLEVR-Ref+ [37]), or to benchmark visual dialog models (CLEVR dialog [34]).”
MethodCLOSURE: Assessing Systematic Generalization of CLEVR Models
Bahdanau, D., de Vries, H., O'Donnell, T. J., Murty, S., Beaudoin, P., Bengio, Y., & Courville, A. (2019) · arXiv · doi:10.48550/arxiv.1912.05783 · Cited by 96 · cited 1×
“There have been numerous dataset extensions for the CLEVR dataset, for instance to measure the generalization capabilities of models pre-trained on CLEVR (CLOSURE [51]), to evaluate object detection and segmentation (CLEVR-Ref+ [37]), or to benchmark visual dialog models (CLEVR dialog [34]).”
MethodFaithful Multimodal Explanation for Visual Question Answering
Wu, J., & Mooney, R. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4812 · Cited by 73 · cited 5×
“All other hyperparameters were taken from [26,53].”
BackgroundMethodInfluentialHIVE: Evaluating the Human Interpretability of Visual Explanations
Kim, S. S. Y., Meister, N., Ramaswamy, V. V., Fong, R., & Russakovsky, O. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19775-8_17 · Cited by 72
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks
Kayser, M., Camburu, O.-M., Salewski, L., Emde, C., Do, V., Akata, Z., & Lukasiewicz, T. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00128 · Cited by 68 · cited 7×
“Explanation datasets commonly used in the context of VQA, such as the VQA-X dataset [26] or the e-SNLI-VE dataset [13,29] for visual entailment, contain explanations of widely varying quality since they are generated by humans.”
BackgroundMethodInfluentialSelf-citationCLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
Kottur, S., Moura, J. M. F., Parikh, D., Batra, D., & Rohrbach, M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1058 · Cited by 58
Marasović, A., Bhagavatula, C., Park, J. S., Le Bras, R., Smith, N. A., & Choi, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.253 · Cited by 51
Generating Counterfactual Explanations with Natural Language
Hendricks, L. A., Hu, R., Darrell, T. J., & Akata, Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09809 · Cited by 49
Self-citationCLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
Kottur, S., Moura, J. M. F., Parikh, D., Batra, D., & Rohrbach, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.03166 · Cited by 46
Interpretable Counting for Visual Question Answering
Trott, A., Xiong, C., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.08697 · Cited by 36 · cited 2×
“In particular, the challenging counting problem [48], which is not well-represented in the VQA-X dataset, can be studied in detail on CLEVR-X.”
BackgroundResultRobust Explanations for Visual Question Answering
Patro, B. N., Patel, S., & Namboodiri, V. P. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093295 · Cited by 33 · cited 1×
“Recently, several works have aimed at generating natural language explanations for vision-language tasks [26,29,38,40,52,53].”
BackgroundDDRprog: A CLEVR Differentiable Dynamic Reasoning ProgrammerPreprint
Suárez, J., Johnson, J., & Fei-Fei, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.11361 · Cited by 32 · cited 1×
“Multiple frameworks have been proposed to address the CLEVR task [23,24,28,41,44,47].”
BackgroundKeep CALM and Improve Visual Feature Attribution
Kim, J. M., Choe, J., Akata, Z., & Oh, S. J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00824 · Cited by 28 · cited 1×
“…made by neural networks can be visually explained with visual attribution that is determined by introspecting trained networks and their features [8,43,46,57,58], by using input perturbations [14,15,42], or by training a probabilistic feature attribution model along with a task-specific CNN [30].”
MethodSelf-citatione-SNLI-VE: Corrected Visual-Textual Entailment with Natural Language Explanations
Do, V., Camburu, O.-M., Akata, Z., & Lukasiewicz, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03744 · Cited by 23 · cited 3×
“Explanation datasets commonly used in the context of VQA, such as the VQA-X dataset [26] or the e-SNLI-VE dataset [13,29] for visual entailment, contain explanations of widely varying quality since they are generated by humans.”
MethodSelf-citationCLEVR-Ref+: Diagnosing Visual Reasoning with Referring ExpressionsPreprint
Liu, R., Liu, C., Bai, Y., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00850 · Cited by 20 · cited 1×
“There have been numerous dataset extensions for the CLEVR dataset, for instance to measure the generalization capabilities of models pre-trained on CLEVR (CLOSURE [51]), to evaluate object detection and segmentation (CLEVR-Ref+ [37]), or to benchmark visual dialog models (CLEVR dialog [34]).”
MethodCURI: A Benchmark for Productive Concept Learning Under UncertaintyPreprint
Vedantam, R., Szlam, A. D., Nickel, M., Morcos, A. S., & Lake, B. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02855 · Cited by 16 · cited 1×
“The Compositional Reasoning Under Uncertainty (CURI) benchmark uses the CLEVR renderer to construct a test bed for compositional and relational learning under uncertainty [49].”
MethodHolzinger, A., Saranti, A., & Mueller, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00519 · Cited by 11 · cited 1×
“[22] provide an extensive survey of further experimental diagnostic benchmarks for analyzing explainable machine learning frameworks along with proposing the KandinskyPATTERNS benchmark that contains synthetic images with simple 2-dimensional objects.”
BackgroundImproving VQA and its Explanations \\ by Comparing Competing Explanations
Wu, J., Chen, L., & Mooney, R. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15631 · Cited by 8 · cited 1×
“Recently, several works have aimed at generating natural language explanations for vision-language tasks [26,29,38,40,52,53].”
Background
Cited by
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Díaz-Rodríguez, N., Del Ser, J., Coeckelbergh, M., López de Prado, M., Herrera-Viedma, E., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101896 · Cited by 800
Ishmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 129
“The CLEVR dataset has been extended to various other tasks including referring expressions [136], visual dialog [137], explainable AI [138], natural language explanations [139], domain robustness [140], and more.”
BackgroundRoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics
Song, C. H., Blukis, V., Tremblay, J., Tyree, S., Su, Y., & Birchfield, S. T. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01470 · Cited by 241
Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning
Li, Z., Wong, X., Stengel-Eskin, E., Kortylewski, A., Ma, W., Durme, B. V., & Yuille, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01437 · Cited by 149
“To assess a model’s true reasoning ability, the CLEVR dataset [27] proposes to generate complex multi-step questions on synthetic images, which is then extended to various vision-and-language tasks [6,23,34,41,53,58,61].”
Background
Metadata
- Authors
- Salewski, L., Koepke, A. S., Lensch, H. P. A., & Akata, Z.
- Affiliations
- University of Tübingen (DE)
- Max Planck Institute for Informatics (DE)
- Corresponding author
- Salewski, Leonard
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2022
- Pages
- 69-88
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W4226136952
- Semantic Scholar
- 22ef74d60e8dabf3167b8288ffb0a6faa8797736
- Cited by
- 20
- References
- 67
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 5 · 2023: 10
- FWCI
- 1.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 106 d)
- Locations
- Lecture notes in computer science (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Deutsche Forschungsgemeinschaft (206/4-1, EXC number 2064/1, 390727645, SFB1233)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- visual reasoning, natural language explanations, visual question answering, scene graph, explanation generation, user study, explainable AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.