Discovering Causal Signals in Images
Open accessAbstract
This paper establishes the existence of observable footprints that reveal the causal dispositions of the object categories appearing in collections of images. We achieve this goal in two steps. First, we take a learning approach to observational causal discovery, and build a classifier that achieves state-of-the-art performance on finding the causal direction between pairs of random variables, given samples from their joint distribution. Second, we use our causal direction classifier to effectively distinguish between features of objects and features of their contexts in collections of static images. Our experiments demonstrate the existence of a relation between the direction of causality and the difference between objects and their contexts, and by the same token, the existence of observable signals that reveal the causal dispositions of objects.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Lopez et al. first identified causal dispositions among semantic categories in hand-crafted datasets and proposed observational causal discovery.
“Lopez et al. [36] first reveal the causal dispositions of the semantic categories in the hand-crafted dataset, which could affect each other's existence, and propose the observational causal discovery technique.”
From Unbiased Embodied Visual Representation Learning with Causal Inference and Cross-Modality Alignment (2025) · cites this paper as Lopez et al. [36]
Section 1.1 describes causal dispositions between object categories that affect whether other objects are present, and Section 2 describes the authors’ observational causal-discovery classifier. But Section 4.1 says the image analysis used MSCOCO images and Pascal VOC categories, not hand-crafted image datasets. The paper proposes a classifier for observational causal discovery; it does not claim to have originated observational causal discovery itself. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe use of counterfactual explanations can help the user to understand the decision of a model.
“It was also shown that the use of counterfactual explanations can help the user to understand the decision of a model [40, 42, 351].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [42]
Section 2, “Observational causal discovery,” discusses inferring cause and effect from data, and Section 5 describes a classifier for causal signals in images; no page numbers are supplied in our copy. Searches for counterfactual explanations and users understanding model decisions found no passage addressing that claim. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · see citing works
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653 · see citing works
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
Flagged first, then by citations
Research works printed but not linked: 22
- K. Chalupka, F. Eberhardt, and P. Perona. Estimating causal direction and confounding of two discrete variables. arXiv, 2016. 3
- K. Chalupka, P. Perona, and F. Eberhardt. Visual causal feature learning. In UAI, 2015. 3
- P. Daniušis, D. Janzing, J. Mooij, J. Zscheischler, B. Steudel, K. Zhang, and B. Schölkopf. Inferring deterministic causal relations. In UAI, 2010. 4
- P. O. Hoyer, D. Janzing, J. M. Mooij, J. Peters, and B. Schölkopf. Nonlinear causal discovery with additive noise models. In NIPS, 2009. 3, 4
- S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015. 6
- D. Janzing and B. Schölkopf. Causal inference using the algo- rithmic Markov condition. IEEE Transactions on Information Theory, 2010. 4
- K. Lebeda, S. Hadfield, and R. Bowden. Exploring causal relationships in visual object tracking. In ICCV, 2015. 3
- T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ra- manan, P. Dollár, and C. L. Zitnick. Microsoft COCO: com- mon objects in context. In Proceedings of 13th European Conference on Computer Vision (ECCV 2014), Part V, pages 740-755, 2014. 6
- D. Lopez-Paz, K. Muandet, and B. Recht. The randomized causation coefficient. JMLR, 2015. 3
- D. Lopez-Paz, K. Muandet, B. Schölkopf, and I. O. Tolstikhin. Towards a learning theory of cause-effect inference. In Pro- ceedings of the 32nd International Conference on Machine Learning (ICML 2015), pages 1452-1461, 2015. 3, 4, 5, 6
- J. Mooij, J. Peters, D. Janzing, J. Zscheischler, and B. Schölkopf. Distinguishing cause from effect using ob- servational data: methods and benchmarks. JMLR, 2016. 3, 5, 6
- S. Mumford and R. L. Anjum. Getting Causes from Powers. Oxford University Press, 2011. 2
- M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Is object localiza- tion for free? -weakly-supervised learning with convolutional neural networks. In CVPR, 2015. 2, 3
- J. Peters, J. Mooij, D. Janzing, and B. Schölkopf. Causal discovery with continuous additive noise models. JMLR, 2014. 3, 4
- L. C. Pickup, Z. Pan, D. Wei, Y. Shih, C. Zhang, A. Zisserman, B. Schölkopf, and W. T. Freeman. Seeing the arrow of time. In CVPR, 2014. 3, 8
- D. B. Rubin. Which ifs have causal answers. Discussion of Holland's "Statistics and Causal Inference". Journal of the Americal Statistical Association, 1986. 2
- O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, April 2015. 6
- B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. M. Mooij. On causal and anticausal learning. In Proceedings of the 29th International Conference on Machine Learning (ICML 2012), 2012. 2, 3, 4
- A. Smola, A. Gretton, L. Song, and B. Schölkopf. A Hilbert space embedding for distributions. In Proceedings ALT, 2007. 4, 5
- N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. JMLR, 2014. 6
- M. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. arXiv, 2013. 2, 3
- B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene CNNs. ICLR, 2015. 2, 3
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
Deep Learning: A Critical Appraisal
Marcus, G. (2018) · arXiv · doi:10.48550/arxiv.1801.00631 · Cited by 853 · see citing works
Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
Lee, J., Lee, Y., Kim, J., Kosiorek, A. R., Choi, S., & Teh, Y. W. (2018) · arXiv · doi:10.48550/arxiv.1810.00825 · Cited by 691 · see citing works
A Survey of Learning Causality with Data: Problems and Methods
Guo, R., Cheng, L., Li, J., Hahn, P. R., & Liu, H. (2021) · ACM Computing Surveys · doi:10.1145/3397269 · Cited by 362 · see citing works
MUREL: Multimodal Relational Reasoning for Visual Question Answering
Cadene, R., Ben-younes, H., Cord, M., & Thome, N. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00209 · Cited by 330 · see citing works
Wang, T., Huang, J., Zhang, H., & Sun, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01077 · Cited by 291 · see citing works
Learning Functional Causal Models with Generative Neural Networks
Goudet, O., Kalainathan, D., Caillou, P., Guyon, I., Lopez-Paz, D., & Sebag, M. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_3 · Cited by 123 · see citing works
Unbiased Embodied Visual Representation Learning with Causal Inference and Cross-Modality Alignment
Kang, J., Chen, B., Zhong, P., Wang, Y., Yang, H., & Sheng, Y. (2025) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3760261
- Authors
- Lopez-Paz, D., Nishihara, R., Chintala, S., Scholkopf, B., & Bottou, L.
- Affiliations
- University of California, Berkeley (US)
- Meta (United States) (US)
- ORCID
- Chintala, Soumith
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 58-66
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2404287312
- Cited by
- 209
- References
- 4
- Citations by year
- 2026: 12 · 2025: 16 · 2024: 31 · 2023: 27 · 2022: 34 · 2021: 32 · 2020: 30 · 2019: 16 · 2018: 8 · 2017: 2 · 2016: 1
- FWCI
- 6.99
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Keywords
- causal discovery, image collections
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
