CoChat

From attribution maps to human-understandable explanations through Concept Relevance Propagation

Open access

The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

The field of explainable artificial intelligence (XAI) aims to bring transparency to today’s powerful but opaque deep learning models. While local XAI methods explain individual predictions in the form of attribution maps, thereby identifying ‘where’ important features occur (but not providing information about ‘what’ they represent), global explanation techniques visualize what concepts a model has generally learned to encode. Both types of method thus provide only partial insights and leave the burden of interpreting the model’s reasoning to the user. Here we introduce the Concept Relevance Propagation (CRP) approach, which combines the local and global perspectives and thus allows answering both the ‘where’ and ‘what’ questions for individual predictions. We demonstrate the capability of our method in various settings, showcasing that CRP leads to more human interpretable explanations and provides deep insights into the model’s representation and reasoning through concept atlases, concept-composition analyses, and quantitative investigations of concept subspaces and their role in fine-grained decision-making.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42256-023-00711-8

Cites

62 of 63 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Glocal XAI can tell which features exist and how they are used for predictions by unifying local and global XAI. Left: local explanations visualize which input pixels are relevant for the prediction. Here, the model focuses on the eye region for all three predictions. However, what features in particular the model has recognized in those regions remains open for interpretation by the user. Right: by finding reference images that maximally represent particular (groups of) neurons, global XAI methods give insight into the concepts generally encoded by the model. However, global methods alone do not inform which concepts are recognized, used and combined by the model in per-sample inference. Centre: glocal XAI can identify the relevant neurons for a particular prediction (property of local XAI) and then visualize the concepts these neurons
Fig. 1 Glocal XAI can tell which features exist and how they are used for predictions by unifying local and global XAI. Left: local explanations visualize which input pixels are relevant for the prediction. Here, the model focuses on the eye region for all three predictions. However, what features in particular the model has recognized in those regions remains open for interpretation by the user. Right: by finding reference images that maximally represent particular (groups of) neurons, global XAI methods give insight into the concepts generally encoded by the model. However, global methods alone do not inform which concepts are recognized, used and combined by the model in per-sample inference. Centre: glocal XAI can identify the relevant neurons for a particular prediction (property of local XAI) and then visualize the concepts these neuronsCC-BY · From the PDF
Fig. 2 Brief overview over the methodological contributions of this work.
Fig. 2 Brief overview over the methodological contributions of this work.CC-BY · From the PDF
Fig. 3 Understanding concepts and concept composition with CRP.
Fig. 3 Understanding concepts and concept composition with CRP.CC-BY · From the PDF
Fig. 4 From concept-level explanations to model and data debugging.
Fig. 4 From concept-level explanations to model and data debugging.CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Fig. 5 Similarity of concepts and analysis of fine-grained decision-making. a , Left: channels from layer features.40 of a VGG-16 with BatchNorm, clustered and embedded according to ρ similarity with t-SNE (Methods). Markers are coloured according to their ρ similarity to filter 446. Centre and right: one particular cluster around channel 446 is shown in more detail with five similarly activating channels and their reference images 𝒳𝒳 ⋆ Ũ rel sum obtained via RelMax. As per the reference images, the overall concept of the cluster seems to be related to keyboard keys, round buttons and rectangular roofing shingles. b , Relevancebased investigation of the previously identified similarly activating channels. Centre: reference examples for the identified filters with a similar underlying
Fig. 5 Similarity of concepts and analysis of fine-grained decision-making. a , Left: channels from layer features.40 of a VGG-16 with BatchNorm, clustered and embedded according to ρ similarity with t-SNE (Methods). Markers are coloured according to their ρ similarity to filter 446. Centre and right: one particular cluster around channel 446 is shown in more detail with five similarly activating channels and their reference images 𝒳𝒳 ⋆ Ũ rel sum obtained via RelMax. As per the reference images, the overall concept of the cluster seems to be related to keyboard keys, round buttons and rectangular roofing shingles. b , Relevancebased investigation of the previously identified similarly activating channels. Centre: reference examples for the identified filters with a similar underlyingCC-BY · From the PDF
Fig. 6 User study evaluating the informativeness of different explanation methods with respect to the model's reliance on a border artefact. a , An example input image with an added black border artefact. b , Attribution maps computed via the methods LRP, SHAP, Grad-CAM and IG. c , An explanation derived with CRP and RelMax. In both b and c , the model has been trained to
Fig. 6 User study evaluating the informativeness of different explanation methods with respect to the model's reliance on a border artefact. a , An example input image with an added black border artefact. b , Attribution maps computed via the methods LRP, SHAP, Grad-CAM and IG. c , An explanation derived with CRP and RelMax. In both b and c , the model has been trained toCC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 63 references

Research works printed but not linked: 6

  1. Goodman, B. & Flaxman, S. European Union regulations on algorithmic decision-making and a ‘right to explanation’. AI Mag. 38, 50–57 (2017).
  2. Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks.In Proc. 34th International Conference on Machine Learning (eds Precup, D. & Teh, Y. W.) 3319–3328 (PMLR, 2017).
  3. Murdoch, W. J., Liu, P. J. & Yu, B. Beyond word importance: contextual decomposition to extract interactions from LSTMs. In 6th International Conference on Learning Representations (ICLR, 2018).
  4. Erhan, D., Bengio, Y., Courville, A. & Vincent, P. Visualizing higher-layer features of a deep network. Univ. Montreal 1341, 1 (2009).
  5. Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T. & Clune, J. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Adv. Neural Inf. Process. Syst. 29, 3387–3395 (2016).
  6. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (eds Bengio, Y. & LeCun, Y.) (ICLR, 2015).

Cited by

See all 238

Metadata

Authors
Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., & Lapuschkin, S.
Affiliations
Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute (DE)
Corresponding author
  • Samek, Wojciech
  • Lapuschkin, Sebastian
Venue
Nature Machine Intelligence
Published
2023-09-20
Volume
5
Issue
9
Pages
1006-1019
ISSN
2522-5839 (ISSN-L)
OpenAlex
W4386897466
Cited by
181
References
45
Citations by year
2026: 43 · 2025: 71 · 2024: 57 · 2023: 10
FWCI
16.02
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Fraunhofer-Gesellschaft
  • Karl-Franzens-Universität Graz
  • Medizinische Universität Graz
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
explainable AI, attribution maps, local and global explanations, concept composition, deep learning model interpretability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: From attribution maps to human-understandable explanations through Concept Relevance Propagation. https://app.cochat.ai/papers/10.1038/s42256-023-00711-8
The paper
Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., & Lapuschkin, S. (2023). From attribution maps to human-understandable explanations through Concept Relevance Propagation. Nature Machine Intelligence. https://doi.org/10.1038/s42256-023-00711-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.