The DOI matches its Crossref record; 65 of the 67 references it cites were found in a registry or an index; 6 of the 26 sources its claims cite were checked for support.
CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
Open accessAbstract
Providing explanations in the context of Visual Question Answering (VQA) presents a fundamental problem in machine learning. To obtain detailed insights into the process of generating natural language explanations for VQA, we introduce the large-scale CLEVR-X dataset that extends the CLEVR dataset with natural language explanations. For each image-question pair in the CLEVR dataset, CLEVR-X contains multiple structured textual explanations which are derived from the original scene graphs. By construction, the CLEVR-X explanations are correct and describe the reasoning and visual information that is necessary to answer a given question. We conducted a user study to confirm that the ground-truth explanations in our proposed dataset are indeed complete and relevant. We present baseline results for generating natural language explanations in the context of VQA using two state-of-the-art frameworks on the CLEVR-X dataset. Furthermore, we provide a detailed analysis of the explanation generation quality for different question and answer types. Additionally, we study the influence of using different numbers of ground-truth explanations on the convergence of natural language generation (NLG) metrics. The CLEVR-X dataset is publicly available at https://github.com/ExplainableML/CLEVR-X.
Overview
CoChat Summary
TL;DR
CLEVR-X extends CLEVR with 3.6 million structured natural language explanations for 850k question-image pairs, generated from scene graphs and validated by a user study. Baselines with PJ-X and FM show strong explanation scores on CLEVR-X, while counting and binary questions remain harder to explain.
Key findings
- CLEVR-X contains 85,000 images, 849,948 questions, and 3,645,137 explanations, with an average of 4.29 explanations per question and 21.53 words per explanation; vocabulary is only 96 words.
- In the MTurk user study, participants selected complete explanations with 92.19% average accuracy and relevant explanations with 92.52% average accuracy; performance was weaker for complex types such as compare-integer (77.00% completeness, 83.67% relevance) and comparison (83.67% completeness, 87.33% relevance).
- PJ-X achieved METEOR 58.9, CIDEr 639.8, and answer accuracy 63.0% on CLEVR-X; FM achieved METEOR 52.5, CIDEr 566.8, and answer accuracy 80.3%.
- Explanation quality was lower for counting answers (METEOR 57.3) than attribute answers (63.3), and lower for binary questions (METEOR 55.6); BLEU-4 needed close to 3 ground-truth explanations to reach 95% of its final value, while CIDEr reached 95.7% with only one.
Methods
- The dataset was built by extending CLEVR: for each question-image pair, the functional program was traced on the scene graph, relevant object attributes were filtered, and natural language templates with synonym sampling generated up to 10 explanations per question.
- A user study on Amazon Mechanical Turk used two HIT types (completeness and relevance), sampling 100 explanations from each of 9 question types for a total of 1800 samples; each sample was rated by 3 workers, with 78 workers in completeness and 101 in relevance (134 total participants).
- Baselines used PJ-X and FM with ResNet-101 image grid features; evaluation used BLEU, METEOR, ROUGE-L, and CIDEr, reported for correctly answered questions.
Limitations
- The paper states that only two natural language generation methods were considered as baselines, and that benchmarking additional recent frameworks is a next step.
- The paper notes the synthetic nature and limited domain of CLEVR, resulting in a very small vocabulary of 96 words; this is evident in the dataset construction and may limit transfer to natural images or human-written explanations.
- The user study showed weaker and variable performance on complex question types and across individual workers, which the paper reports; this suggests the validation is strongest for simpler explanations and question types.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
3 Valid · 2 Not valid · 16 Not checked · 95 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Explanation datasets commonly used in the context of VQA, such as the VQA-X dataset [26] or the e-SNLI-VE dataset [13,29] for visual entailment, contain explanations of widely varying quality since they are generated by humans.”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 1 Introduction
What the sources say
- Partly supportedcites [26] Park, D.H., et al.: Multimodal explanations: justifying decisions and pointing to the evidence. In: CVPR, pp. 8779–8788 (2018) sourceExperimental — accuracy not yet measured
The Introduction identifies VQA-X as a dataset of human-annotated explanations, supporting the part attributed to this citation about VQA-X and its human origin. The VQA-X dataset section describes collecting textual explanations, but neither it nor the Experimental Setup establishes that their quality varies widely because humans generated them. This citation does not address the separately cited e-SNLI-VE dataset. (LLM, conf=0.86)
- Not checked yetcites [13] Do, V., Camburu, O.M., Akata, Z., Lukasiewicz, T.: e-SNLI-VE-2.0: corrected visual-textual entailment with natural language explanations. arXiv preprint arXiv:2004.03744 (2020)
- Partly supportedcites [29] Kayser, M., Camburu, O.M., Salewski, L., Emde, C., Do, V., Akata, Z., Lukasiewicz, T.: e-VIL: a dataset and benchmark for natural language explanations in vision-language tasks. In: ICCV, pp. 1244–1254 (2021) sourceExperimental — accuracy not yet measured
Section 4.2 (p. 4) reports varying human-evaluated quality for ground-truth explanations in both VQA-X and e-SNLI-VE, and Sections 3.2 and 4.2 identify their human-written explanations. The paper does not attribute that variation simply to human authorship; for e-SNLI-VE, it points to explanations adapted from a text-based task, imperfect filtering, and task ambiguity. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The groundtruth explanations in VQA-X and e-SNLI-VE can range from statements that merely describe an image to explaining the reasoning about the question and image involving prior information, such as common knowledge.”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 1 Introduction
What the sources say
- Partly supportedcites [26] Park, D.H., et al.: Multimodal explanations: justifying decisions and pointing to the evidence. In: CVPR, pp. 8779–8788 (2018) sourceExperimental — accuracy not yet measured
Our copy of Park et al. (2018) supports only the VQA-X half of the claim, and even there at a weaker level: the 'VQA-X.' qualitative section states that textual justifications 'are able to both capture common sense and discuss specific image parts important for answering a question' (e.g. the concept of a zoo as 'animals in an enclosure' vs. 'foam on the waves'), and the 'VQA Explanation Dataset (VQA-X)' section notes VQA requires 'understanding vision, language, and commonsense knowledge'. However, that passage concerns the model's generated justifications rather than a stated range of the human ground-truth annotations, and the dataset section in fact describes deliberately excluding trivially descriptive questions ('What is the color of the banana?', age-9+ filter) — so 'statements that merely describe an image' is not what this paper attributes to its ground truth. Nothing in our copy addresses e-SNLI-VE at all (searched: e-SNLI, visual entailment, explanation dataset ranges); that dataset is from later work and this paper only introduces VQA-X and ACT-X. The e-SNLI-VE portion, and the descriptive-vs-reasoning spectrum framed as a property of ground-truth annotations across both datasets, are therefore not supported by this source. (LLM, conf=0.75)
- Not checked yetcites [13] Do, V., Camburu, O.M., Akata, Z., Lukasiewicz, T.: e-SNLI-VE-2.0: corrected visual-textual entailment with natural language explanations. arXiv preprint arXiv:2004.03744 (2020)
- Partly supportedcites [29] Kayser, M., Camburu, O.M., Salewski, L., Emde, C., Do, V., Akata, Z., Lukasiewicz, T.: e-VIL: a dataset and benchmark for natural language explanations in vision-language tasks. In: ICCV, pp. 1244–1254 (2021) sourceExperimental — accuracy not yet measured
Appendix A, Qualitative Examples (p. 12) does characterize the ground-truth explanations of VQA-X and e-SNLI-VE along the image-dependence axis: VQA-X examples where 'a generic explanation, that is not necessarily grounded in the image, will often suffice', versus e-SNLI-VE examples that 'require the explanations to describe image-specific characteristics'. But the upper end of the claimed range — reasoning involving prior information such as common knowledge — is attributed by this paper to VCR, not to VQA-X or e-SNLI-VE ('VCR has many questions that require substantial commonsense reasoning and general knowledge', p. 12; Figure 1 / Related Work, p. 2). The paper in fact states the opposite for these two datasets: e-SNLI-VE 'contains over 430k instances for which the explanations rely on the image content' (p. 2) and many VQA-X NLEs 'are trivial and could be guessed without looking at the image' (p. 2), reinforced by the human evaluation finding that VQA-X explanations are 'easier and less elaborate' (5.3.1, p. 7). So the part of the claim extending to prior/common-knowledge reasoning in VQA-X and e-SNLI-VE is not supported. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The e-SNLI-VE dataset contains numerous explanation samples which consist of repeated statements (”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 1 Introduction
What the sources say
- Not checked yetcites [13] Do, V., Camburu, O.M., Akata, Z., Lukasiewicz, T.: e-SNLI-VE-2.0: corrected visual-textual entailment with natural language explanations. arXiv preprint arXiv:2004.03744 (2020)
- Partly supportedcites [29] Kayser, M., Camburu, O.M., Salewski, L., Emde, C., Do, V., Akata, Z., Lukasiewicz, T.: e-VIL: a dataset and benchmark for natural language explanations in vision-language tasks. In: ICCV, pp. 1244–1254 (2021) sourceExperimental — accuracy not yet measured
Section 3.2 (p. 3) does document this failure mode: the similarity filter was introduced because 'the share of low-quality explanations is highest for entailment examples... when the textual premise and hypothesis are almost identical, as then the explanation often just repeats both statements', and Appendix B.4 (pp. 16-17, Fig. 13) gives a concrete tautological example. But what our copy attributes to the released e-SNLI-VE is removal of these cases (all pairs with premise-hypothesis ROUGE-1 > 0.57 deleted, cutting low-quality entailment explanations by 4.2%), plus a general caveat that the filters were 'effective, but imperfect' leaving residual noise (Table 2, p. 4: 10.3% 'No', 6.9% 'Weak No'; e-SNLI's 9.6% explanation error rate as an upper bound of cleaning, p. 16). The paper nowhere states that the final dataset still contains 'numerous' repeated-statement samples, so the quantity/target-of-claim part is unsupported. The '(x because x)' phrasing also loosely echoes p. 7 (Sec. 5.3.1), where nonsensical explanations like 'a woman is a woman' (~10% prevalence) are reported for model-generated, not dataset, explanations. (LLM, conf=0.75)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, the PJ-X backbone MCB [16] (which is crucial for the answering performance) preceded the publication of the CLEVR dataset.”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 4.2 Evaluating Explanations Generated by State-of-the-Art Methods
What the sources say
- Could not verifycites [16] Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multimodal compact bilinear pooling for visual question answering and visual grounding. In: EMNLP, pp. 457–468 (2016) sourceExperimental — accuracy not yet measured
Our copy of Fukui et al. (2016) never mentions CLEVR or its publication timing: the Datasets sections (4.1 VQA real-image/Visual Genome/Visual7W; 5.1 Flickr30k Entities/ReferItGame) and the Related Work (Sec. 2) and Conclusion (Sec. 6) contain no such statement, and the reference list carries no Johnson/CLEVR entry (searched: 'CLEVR', 'synthetic rendered scenes Cactus Johnson', 'Johnson clevr', 'PJ-X backbone parallel joint crossmodal'). The term 'PJ-X' does not appear anywhere in our copy either — the paper only proposes MCB pooling with attention for VQA and grounding. A chronological claim that MCB preceded CLEVR's publication is therefore outside what the source itself says; nothing in our copy contradicts it, but nothing supports it. (LLM, conf=0.72)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unfortunately, VQA-X and e-SNLI-VE contain spelling errors, resulting in multiple versions of the same words.”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 3.3 Dataset Analysis
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
58 more claims
Create a free account to see all 116 claims and to verify the 20 references still open.
- Probably not a substantial claim. 32
- It may state two separate claims as one. 29
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Natural language explanations [27,67], quantitative measures of the quality of explanations [68,69], and models that support their learning process with formal symbolic basis such as language, rules, compositional relationships or knowledge graphs (neural-symbolic learning and reasoning [66]) are key for explanations to be understood by non-expert audience.”
From Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation (2023) · cites this paper as [67]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Approaches without human feedback, for example, because they use an artificial benchmark with defined ground truth for the desired explanations [18,32,36], suffer from a direct proof of being transferable to real-world use cases.”
From Study on the Helpfulness of Explainable Artificial Intelligence (2024) · cites this paper as [36]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“Total 28,180 32,886 41,817 10,315 1.48 10.64 7.49”
§ CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations > 3.3 Dataset Analysis
Reported 1.48 · recomputed 1.271575
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We extracted 14 × 14 × 1024 grid features for the images in the CLEVR-X dataset using a ResNet-101 [19], pre-trained on Ima-geNet [12].”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“For PJ-X, our best model was trained for 52 epochs, using the Adam optimizer [33] with a learning rate of 0.0002 and a batch size of 256.”
MethodInfluentialImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“We extracted 14 × 14 × 1024 grid features for the images in the CLEVR-X dataset using a ResNet-101 [19], pre-trained on Ima-geNet [12].”
MethodGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1×
“To evaluate the quality of the generated explanations, we use the standard natural language generation metrics BLEU [39], METEOR [9], ROUGE-L [36] and CIDEr [50].”
MethodVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
(2004) · Cited by 9,718
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“Decisions made by neural networks can be visually explained with visual attribution that is determined by introspecting trained networks and their features [46,57,43,8,58], by using input perturbations [42,14,15], or by training a probabilistic feature attribution model along with a task-specific CNN [30].”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948 · cited 1×
“It is based on the Up-Down VQA model [4].”
Method
Cited by
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
Díaz-Rodríguez, N., Del Ser, J., Coeckelbergh, M., López de Prado, M., Herrera-Viedma, E., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101896 · Cited by 800
Ishmam, M. F., Shovon, M. S. H., Mridha, M. F., & Dey, N. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102270 · Cited by 129
“The CLEVR dataset has been extended to various other tasks including referring expressions [136], visual dialog [137], explainable AI [138], natural language explanations [139], domain robustness [140], and more.”
BackgroundRoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics
Song, C. H., Blukis, V., Tremblay, J., Tyree, S., Su, Y., & Birchfield, S. T. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01470 · Cited by 240
Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoning
Li, Z., Wong, X., Stengel-Eskin, E., Kortylewski, A., Ma, W., Durme, B. V., & Yuille, A. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01437 · Cited by 149
“To assess a model’s true reasoning ability, the CLEVR dataset [27] proposes to generate complex multi-step questions on synthetic images, which is then extended to various vision-and-language tasks [6,23,34,41,53,58,61].”
Background
Metadata
- Authors
- Salewski, L., Koepke, A. S., Lensch, H. P. A., & Akata, Z.
- Affiliations
- University of Tübingen (DE)
- Max Planck Institute for Informatics (DE)
- Corresponding author
- Salewski, Leonard
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2022
- Pages
- 69-88
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W4226136952
- Semantic Scholar
- 22ef74d60e8dabf3167b8288ffb0a6faa8797736
- Cited by
- 20
- References
- 67
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 5 · 2023: 10
- FWCI
- 1.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 106 d)
- Locations
- Lecture notes in computer science (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Deutsche Forschungsgemeinschaft (206/4-1, EXC number 2064/1, 390727645, SFB1233)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- visual reasoning, natural language explanations, visual question answering, scene graph, explanation generation, user study, explainable AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations. https://app.cochat.ai/papers/10.1007/978-3-031-04083-2_5
- The paper
- Salewski, L., Koepke, A. S., Lensch, H. P. A., & Akata, Z. (2022). CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations. Lecture Notes in Computer Science. https://doi.org/10.1007/978-3-031-04083-2_5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.