CoChat
Sign Up
PreprintCited by 48Explainable Artificial Intelligence (XAI)Bayesian Modeling and Causal InferenceLogic, Reasoning, and Knowledge

Causal abstractions of neural networks

Open access
Geiger, Atticus, Lu, Hanson, Icard, Thomas, Potts, Christopher·2021·arXiv
DOI 10.48550/arxiv.2106.02997arXiv 2106.02997
Open this paper in CoChat — free accountOpen PDF

Abstract

Structural analysis methods (e.g., probing and feature attribution) are increasingly important tools for neural network analysis. We propose a new structural analysis method grounded in a formal theory of causal abstraction that provides rich characterizations of model-internal representations and their roles in input/output behavior. In this method, neural representations are aligned with variables in interpretable causal models, and then interchange interventions are used to experimentally verify that the neural representations have the causal properties of their aligned variables. We apply this method in a case study to analyze neural models trained on Multiply Quantified Natural Language Inference (MQNLI) corpus, a highly complex NLI dataset that was constructed with a tree-structured natural logic causal model. We discover that a BERT-based model with state-of-the-art performance successfully realizes parts of the natural logic model's causal structure, whereas a simpler baseline model fails to show any such structure, demonstrating that BERT representations encode the compositional structure of MQNLI.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2106.02997

Cites

30 of 31 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A procedure for aligning individual activations of a neural network to an explicit causal graph has been used to show that language models' inferences obey abstract syntactic and semantic rules.

    “More recently, [54] developed a procedure for aligning individual activations of a neural network to an explicit causal graph, and used it to show that language models’ inferences obey abstract syntactic and semantic rules.”

    From Symbols and grounding in large language models (2023) · cites this paper as [54]

    The Introduction (p. 1) describes aligning neural representations with variables in a causal model, and the Conclusion (p. 10) says a BERT-based model partially realizes the natural-logic model’s causal structure. The claim overstates the result: the evidence concerns particular subsets of examples from the MQNLI task, not language models’ inferences generally obeying abstract syntactic and semantic rules. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Multiple studies suggest that apparently symbolic representations play a causal role in models' subsequent predictions.

    “In the past few years, multiple studies have suggested that these apparently symbolic representations do indeed play a causal role in models’ subsequent predictions, each study using a different method to arrive at this conclusion.”

    From Symbols and grounding in large language models (2023) · cites this paper as [54]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 32 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. [4][4] S. Bongers, P. Forré, J. Peters, B. Schölkopf, and J. M. Mooij. Foundations of structural causal models with cycles and latent variables. arXiv.org preprint , arXiv:1611.06221v4 [stat.ME], Oct. 2020. URL https://arxiv.org/abs/1611.06221v4 . - · arXiv
  2. [6][6] A. Chattopadhyay, P. Manupriya, A. Sarkar, and V. N. Balasubramanian. Neural network attributions: A causal perspective. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 981-990, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http://proceedings.mlr.press/v97/chattopadhyay19a.html . -
  3. [15][15] G. W. Imbens and D. B. Rubin. Causal inference in statistics, social, and biomedical sciences . Cambridge University Press, 2015. -

Cited by

Sorted by citations

Related

Authors
Geiger, A., Lu, H., Icard, T., & Potts, C.
Affiliations
Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
34
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2106.02997
arXiv
2106.02997
OpenAlex
W3168829745
Cited by
45
References
32
Citations by year
2026: 1 · 2025: 5 · 2024: 7 · 2023: 27 · 2022: 5
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
Keywords
causal abstraction, neural network interpretability, natural logic, natural language inference, compositional structure, causal models, intervention, probing, feature attribution

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review