CoChat
Sign Up
Cited by 375Explainable Artificial Intelligence (XAI)Advanced Neural Network ApplicationsGenerative Adversarial Networks and Image Synthesis

Understanding the role of individual units in a deep neural network

Open access
Bau, David, Zhu, Jun-Yan, Strobelt, Hendrik, Lapedriza, Agata, Zhou, Bolei, Torralba, Antonio·2020·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.1907375117PMC PMC7720226
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep neural networks excel at finding hierarchical representations that solve complex tasks over large datasets. How can we humans understand these learned representations? In this work, we present network dissection, an analytic framework to systematically identify the semantics of individual hidden units within image classification and image generation networks. First, we analyze a convolutional neural network (CNN) trained on scene classification and discover units that match a diverse set of object concepts. We find evidence that the network has learned many object classes that play crucial roles in classifying scene classes. Second, we use a similar analytic method to analyze a generative adversarial network (GAN) model trained to generate scenes. By analyzing changes made when small sets of units are activated or deactivated, we find that objects can be added and removed from the output scenes while adapting to the context. Finally, we apply our analytic framework to understanding adversarial attacks and to semantic image editing.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.1907375117

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A 'cat neuron' phenomenon has been found and is remarkably replicable across newer models and analyses.

    “[41] famously found evidence of a ‘cat neuron’, a phenomenon that has been remarkably replicable across newer models and analyses [14,42,43]”

    From Symbols and grounding in large language models (2023) · cites this paper as [42]

    The Results (pp. 2–3) identify individual object-detector units, including an airplane detector, supporting the general phenomenon of neurons responsive to recognizable objects. The passages found do not identify a cat neuron specifically. The sentence attributes replication across newer models to other citations, not to this paper. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. [1]B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, Object detectors emerge in deep scene CNNs. arXiv:1412.6856 (22 December 2014). · arXiv
  2. [2]M. D. Zeiler, R. Fergus, 'Visualizing and understanding convolutional networks' in European Conference on Computer Vision (Springer, Berlin, Germany, 2014), pp. 818833.
  3. [3]A. Mahendran, A. Vedaldi, 'Understanding deep image representations by inverting them' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, New York, NY, 2015), pp. 5188-5196.
  4. [5]A. Bau et al. , Identifying and controlling important neurons in neural machine translation. https://openreview.net/pdf?id=H1z-PsR5KX. Accessed 24 August 2020.
  5. [6]A. Karpathy, J. Johnson, L. Fei-Fei, Visualizing and understanding recurrent networks. arXiv:1506.02078 (5 June 2015). · arXiv
  6. [8]Y. Bengio, A. Courville, P. Vincent, Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intelligence 35 , 1798-1828 (2013).
  7. [9]B. Zhou, D. Bau, A. Oliva, A. Torralba, Interpreting deep visual representations via network dissection. arXiv:1711.05611 (26 June 2018). · arXiv
  8. [10]D. Bau et al. , Gan dissection: Visualizing and understanding generative adversarial networks. https://openreview.net/pdf?id=Hyg X2C5FX. Accessed 24 August 2020.
  9. [11]Y. LeCun, Y. Bengio, 'Convolutional networks for images, speech, and time series' in The Handbook of Brain Theory and Neural Networks , M. A. Arbib, Ed. (MIT Press, Cambridge, MA, 1995), vol. 3361, pp. 255-258.
  10. [13]K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (4 September 2014). · arXiv
  11. [14]I. Goodfellow et al. , 'Generative adversarial nets' in Advances in Neural Information Processing Systems (Curran Associates, Red Hook, NY, 2014), pp. 2672-2680.
  12. [17]P. Isola, J. Y . Zhu, T. Zhou, A. A. Efros, 'Image-to-image translation with conditional adversarial networks' in Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (IEEE, New York, NY, 2017), pp. 1125-1134.
  13. [19]T. Karras, T. Aila, S. Laine, J. Lehtinen, Progressive growing of GANs for improved quality, stability, and variation. https://openreview.net/pdf?id=Hk99zCeAb. Accessed 24 August 2020.
  14. [21]T. Zhou, P. Krahenbuhl, M. Aubry, Q. Huang, A. A. Efros, 'Learning dense correspondence via 3D-guided cycle consistency' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, New York, NY, 2016), pp. 117-126.
  15. [22]R. C. Fong, A. Vedaldi, 'Interpretable explanations of black boxes by meaningful perturbation' in International Conference on Computer Vision (IEEE, New York, NY, 2017), pp. 3429-3437.
  16. [24]R. R. Selvaraju et al. , 'Grad-cam: Visual explanations from deep networks via gradient-based localization' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, New York, NY, 2017), pp. 618-626.
  17. [25]M. Sundararajan, A. Taly, Q. Yan, 'Axiomatic attribution for deep networks' in Proceedings of the 34th International Conference on Machine Learning (JMLR, 2017), vol. 70 , pp. 3319-3328.
  18. [26]D. Smilkov, N. Thorat, B. Kim, F. Vi´ egas, M. Wattenberg, Smoothgrad: Removing noise by adding noise. arXiv:1706.03825 (12 June 2017). · arXiv
  19. [27]V. Petsiuk, A. Das, K. Saenko, 'Rise: Randomized input sampling for explanation of black-box models' in British Machine Vision Conference (BMVA Press, Malvern, UK, 2018).
  20. [28]M. T. Ribeiro, S. Singh, C. Guestrin, 'Why should I trust you?: Explaining the predictions of any classifier' in Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (ACM, New York, NY, 2016), pp. 1135-1144.
  21. [29]B. Kim, J. Gilmer, F. Viegas, U. Erlingsson, M. Wattenberg, Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). arXiv:1711.11279 (7 June 2018). · arXiv
  22. [30]A. Koul, A. Fern, S. Greydanus, Learning finite state representations of recurrent policy networks. https://openreview.net/pdf?id=S1gOpsCctm. Accessed 24 August 2020.
  23. [31]L. A. Hendricks et al. , 'Generating visual explanations' in European Conference on Computer Vision (Springer, Berlin, Germany, 2016), pp. 3-19.
  24. [32]B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, A. Oliva, 'Learning deep features for scene recognition using places database' in Advances in Neural Information Processing Systems (Curran Associates, Red Hook, NY, 2014), pp. 487-495.
  25. [35]J. Deng et al. , 'Imagenet: A large-scale hierarchical image database' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, New York, NY, 2009), pp. 248-255.
  26. [36]W. Wen, C. Wu, Y. Wang, Y. Chen, H. Li, 'Learning structured sparsity in deep neural networks' in Advances in Neural Information Processing Systems (Curran Associates, Red Hook, NY, 2016), pp. 2074-2082.
  27. [37]H. Li, A. Kadav, I. Durdanovic, H. Samet, H. P. Graf, Pruning filters for efficient convnets. https://openreview.net/pdf?id=rJqFGTslg. Accessed 24 August 2020.
  28. [38]A. S. Morcos, D. G. Barrett, N. C. Rabinowitz, M. Botvinick, On the importance of single directions for generalization. arXiv:1803.06959 (22 May 2018). · arXiv
  29. [39]B. Zhou, Y. Sun, D. Bau, A. Torralba, Revisiting the importance of individual units in CNNs via ablation. arXiv:1806.02891 (7 June 2018). · arXiv
  30. [40]A. Radford, L. Metz, S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434 (19 November 2015). · arXiv
  31. [41]J. Y. Zhu, P. Kr¨ ahenb ¨ uhl, E. Shechtman, A. A. Efros, 'Generative visual manipulation on the natural image manifold' in European Conference on Computer Vision (Springer, Berlin, Germany, 2016), pp. 597-613.
  32. [42]F. Yu et al. , LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv:1506.03365 (4 June 2016). · arXiv
  33. [43]C. Szegedy et al. , Intriguing properties of neural networks. arXiv:1312.6199 (21 December 2013). · arXiv
  34. [44]I. J. Goodfellow, J. Shlens, C. Szegedy, Explaining and harnessing adversarial examples. arXiv:1412.6572 (20 December 2014). · arXiv
  35. [46]A. Madry, A. Makelov, L. Schmidt, D. Tsipras, A. Vladu, Towards deep learning models resistant to adversarial attacks. https://openreview.net/pdf?id=rJzIBfZAb. Accessed 24 August 2020.
  36. [47]J. Rauber, W. Brendel, M. Bethge, Foolbox: A python toolbox to benchmark the robustness of machine learning models. arXiv:1707.04131 (20 March 2018). · arXiv
  37. [48]A. Ilyas et al. , Adversarial examples are not bugs, they are features. arXiv:1905.02175 (12 August 2019). · arXiv
  38. [49]X. Chen et al. , 'Infogan: Interpretable representation learning by information maximizing generative adversarial nets' in Advances in Neural Information Processing Systems (Curran Associates, Red Hook, NY, 2016), pp. 2172-2180.
  39. [50]I. Higgins et al. , β -vae: Learning basic visual concepts with a constrained variational framework. https://openreview.net/pdf?id=Sy2fzU9gl. Accessed 24 August 2020.
  40. [51]Q. Zhang, Y. Nian Wu, S. C. Zhu, Interpretable convolutional neural arXiv:1710.00935 (2 October 2017). · arXiv
  41. [52]A. Achille, S. Soatto, Emergence of invariance and disentanglement in deep representations. JMLR 19 , 1947-1980 (2018).
  42. [53]B. Zhou et al. , 'Scene parsing through ade20k dataset' in Computer Vision and Pattern Recognition (IEEE, 2017).
  43. [61]J. Van De Weijer, C. Schmid, J. Verbeek, D. Larlus, Learning color names for real-world applications. IEEE Trans. Image Process. 18 , 1512-1523 (2009).

Cited by

Sorted by citations

Authors
Bau, D., Zhu, J.-Y., Strobelt, H., Lapedriza, A., Zhou, B., & Torralba, A.
Affiliations
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • Adobe Systems (United States) (US)
  • IBM (United States) (US)
  • Media Lab, Massachusetts Institute of Technology, Cambridge, MA 02139;
  • Chinese University of Hong Kong (HK)
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2020-09
Volume
117
Issue
48
Pages
30071-30078
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.1907375117
PMC
PMC7720226
OpenAlex
W3082665562
Cited by
375
References
9
Citations by year
2026: 22 · 2025: 51 · 2024: 66 · 2023: 87 · 2022: 76 · 2021: 62 · 2020: 11
FWCI
22.98
Type
journal article
Language
en
Open access
Licence
www.pnas.org/site/aboutpnas/licenses.xhtml (vor, 181 d)
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • arXiv (Cornell University)
Funders
  • Defense Sciences Office, DARPA (FA8750-18- 413 C000)
  • National Science Foundation (1524817, 1447476)
  • University Grants Committee (24206219)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
deep neural network interpretability, convolutional neural networks, generative adversarial networks, scene classification, image generation, object concepts, adversarial attacks, semantic image editing, hierarchical representation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review