Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 65 of the 67 references it cites were found in a registry or an index; 6 of the 26 sources its claims cite were checked for support.

CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations

Open access
Salewski, Leonard, Koepke, A. Sophia, Lensch, Hendrik P. A., Akata, Zeynep · 2022 · Lecture Notes in Computer Science

Abstract

Providing explanations in the context of Visual Question Answering (VQA) presents a fundamental problem in machine learning. To obtain detailed insights into the process of generating natural language explanations for VQA, we introduce the large-scale CLEVR-X dataset that extends the CLEVR dataset with natural language explanations. For each image-question pair in the CLEVR dataset, CLEVR-X contains multiple structured textual explanations which are derived from the original scene graphs. By construction, the CLEVR-X explanations are correct and describe the reasoning and visual information that is necessary to answer a given question. We conducted a user study to confirm that the ground-truth explanations in our proposed dataset are indeed complete and relevant. We present baseline results for generating natural language explanations in the context of VQA using two state-of-the-art frameworks on the CLEVR-X dataset. Furthermore, we provide a detailed analysis of the explanation generation quality for different question and answer types. Additionally, we study the influence of using different numbers of ground-truth explanations on the convergence of natural language generation (NLG) metrics. The CLEVR-X dataset is publicly available at https://github.com/ExplainableML/CLEVR-X.

Overview

CoChat Summary

TL;DR

CLEVR-X extends CLEVR with 3.6 million structured natural language explanations for 850k question-image pairs, generated from scene graphs and validated by a user study. Baselines with PJ-X and FM show strong explanation scores on CLEVR-X, while counting and binary questions remain harder to explain.

Key findings

  • CLEVR-X contains 85,000 images, 849,948 questions, and 3,645,137 explanations, with an average of 4.29 explanations per question and 21.53 words per explanation; vocabulary is only 96 words.
  • In the MTurk user study, participants selected complete explanations with 92.19% average accuracy and relevant explanations with 92.52% average accuracy; performance was weaker for complex types such as compare-integer (77.00% completeness, 83.67% relevance) and comparison (83.67% completeness, 87.33% relevance).
  • PJ-X achieved METEOR 58.9, CIDEr 639.8, and answer accuracy 63.0% on CLEVR-X; FM achieved METEOR 52.5, CIDEr 566.8, and answer accuracy 80.3%.
  • Explanation quality was lower for counting answers (METEOR 57.3) than attribute answers (63.3), and lower for binary questions (METEOR 55.6); BLEU-4 needed close to 3 ground-truth explanations to reach 95% of its final value, while CIDEr reached 95.7% with only one.

Methods

  • The dataset was built by extending CLEVR: for each question-image pair, the functional program was traced on the scene graph, relevant object attributes were filtered, and natural language templates with synonym sampling generated up to 10 explanations per question.
  • A user study on Amazon Mechanical Turk used two HIT types (completeness and relevance), sampling 100 explanations from each of 9 question types for a total of 1800 samples; each sample was rated by 3 workers, with 78 workers in completeness and 101 in relevance (134 total participants).
  • Baselines used PJ-X and FM with ResNet-101 image grid features; evaluation used BLEU, METEOR, ROUGE-L, and CIDEr, reported for correctly answered questions.

Limitations

  • The paper states that only two natural language generation methods were considered as baselines, and that benchmarking additional recent frameworks is a next step.
  • The paper notes the synthetic nature and limited domain of CLEVR, resulting in a very small vocabulary of 96 words; this is evident in the dataset construction and may limit transfer to natural images or human-written explanations.
  • The user study showed weaker and variable performance on complex question types and across individual workers, which the paper reports; this suggests the validation is strongest for simpler explanations and question types.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/978-3-031-04083-2_5

Cites

65 of 65 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Answer | Explanation: Yes | The photo is in black and white and the cars are all classic designs from the 1950s
Answer | Explanation: Yes | The photo is in black and white and the cars are all classic designs from the 1950sCC-BY · From the PDF
Answer | Explanation: 1 | There is a cyan cylinder which is on the right side of
Answer | Explanation: 1 | There is a cyan cylinder which is on the right side ofCC-BY · From the PDF
Answer | Explanation: Entailment | If a woman holds a child she is holding a child.
Answer | Explanation: Entailment | If a woman holds a child she is holding a child.CC-BY · From the PDF
Fig. 2 CLEVR-X dataset generation: Generating a natural language explanation for a sample from the CLEVR dataset. Based on the question, the functional program for answering the question is executed on the scene graph and traced. A language template is used to cast the gathered information into a natural language explanation.
Fig. 2 CLEVR-X dataset generation: Generating a natural language explanation for a sample from the CLEVR dataset. Based on the question, the functional program for answering the question is executed on the scene graph and traced. A language template is used to cast the gathered information into a natural language explanation.CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Fig. 4 Two examples from our user study to evaluate the completeness (left) and relevance (right) of natural language explanations in the CLEVR-X dataset.
Fig. 4 Two examples from our user study to evaluate the completeness (left) and relevance (right) of natural language explanations in the CLEVR-X dataset.CC-BY · From the PDF
Fig. 5 Average answering accuracies for each worker (top) and average work time (bottom) for the user study (left: completeness, right: relevance). The boxes indicate the mean as well as lower and upper quartiles, the lines extend 1.5 interquartile ranges of the lower and upper quartile. All other values are plotted as diamonds.
Fig. 5 Average answering accuracies for each worker (top) and average work time (bottom) for the user study (left: completeness, right: relevance). The boxes indicate the mean as well as lower and upper quartiles, the lines extend 1.5 interquartile ranges of the lower and upper quartile. All other values are plotted as diamonds.CC-BY · From the PDF
Fig. 6 Explanation generation results for PJ-X on the CLEVR-X test set according to question (left) and answer (middle) types compared to the overall explanation quality. Easier types yield higher METEOR scores. NLG metrics using different numbers of ground-truth explanations on the CLEVR-X test set (right). CIDEr converges faster than the other NLG metrics.
Fig. 6 Explanation generation results for PJ-X on the CLEVR-X test set according to question (left) and answer (middle) types compared to the overall explanation quality. Easier types yield higher METEOR scores. NLG metrics using different numbers of ground-truth explanations on the CLEVR-X test set (right). CIDEr converges faster than the other NLG metrics.CC-BY · From the PDF
GT Answer | Explanation: 1 | The tiny red metal block has the same material as a big sphere.
GT Answer | Explanation: 1 | The tiny red metal block has the same material as a big sphere.CC-BY · From the PDF
GT Answer | Explanation: Small | The cylinder is small.
GT Answer | Explanation: Small | The cylinder is small.CC-BY · From the PDF
GT Answer | Explanation: No | There are no small matte cubes.
GT Answer | Explanation: No | There are no small matte cubes.CC-BY · From the PDF

Verification

Claims the paper makes

3 Valid · 2 Not valid · 16 Not checked · 95 Own results and interpretations

58 more claims

Create a free account to see all 116 claims and to verify the 20 references still open.

See all claims — free account
  • Probably not a substantial claim. 32
  • It may state two separate claims as one. 29

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

1 of 1

Citations

Cites

Show all 67 references

Cited by

See all 21

Metadata

Authors
Salewski, L., Koepke, A. S., Lensch, H. P. A., & Akata, Z.
Affiliations
  • University of Tübingen (DE)
  • Max Planck Institute for Informatics (DE)
Corresponding author
Salewski, Leonard
Venue
Lecture Notes in Computer Science
Published
2022
Pages
69-88
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
OpenAlex
W4226136952
Cited by
20
References
67
Citations by year
2026: 2 · 2025: 3 · 2024: 5 · 2023: 10
FWCI
1.63
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Deutsche Forschungsgemeinschaft (206/4-1, EXC number 2064/1, 390727645, SFB1233)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
visual reasoning, natural language explanations, visual question answering, scene graph, explanation generation, user study, explainable AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations. https://app.cochat.ai/papers/10.1007/978-3-031-04083-2_5
The paper
Salewski, L., Koepke, A. S., Lensch, H. P. A., & Akata, Z. (2022). CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations. Lecture Notes in Computer Science. https://doi.org/10.1007/978-3-031-04083-2_5

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.