CoChat
Sign Up
PreprintCited by 581Explainable Artificial Intelligence (XAI)Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

Understanding intermediate layers using linear classifier probes

Open access
Alain, Guillaume, Bengio, Yoshua·2016·arXiv
DOI 10.48550/arxiv.1610.01644arXiv 1610.01644
Open this paper in CoChat — free accountOpen PDF

Abstract

Neural network models have a reputation for being black boxes. We propose to monitor the features at every layer of a model and measure how suitable they are for classification. We use linear classifiers, which we refer to as "probes", trained entirely independently of the model itself. This helps us better understand the roles and dynamics of the intermediate layers. We demonstrate how this can be used to develop a better intuition about models and to diagnose potential problems. We apply this technique to the popular models Inception v3 and Resnet-50. Among other things, we observe experimentally that the linear separability of features increase monotonically along the depth of the model.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1610.01644

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Our framework significantly outperforms previous adaptation methods across all datasets.

    “The experiment reveals our approach significantly outperforms previous adaptation methods across all datasets.”

    From Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model (2024) · cites this paper as [1]

    The paper’s Abstract (p. 1) describes linear classifier probes, and §4.2 (p. 7) reports probe prediction errors and feature separability. Searches for the claimed outperformance and for adaptation-method comparisons found no passage reporting a comparison with previous adaptation methods across datasets. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 27

  1. Alain, G. and Bengio, Y. (2016). Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 .
  2. Arras, L., Montavon, G., M¨ uller, K.-R., and Samek, W. (2017). Explaining recurrent neural network predictions in sentiment analysis. arXiv preprint arXiv:1706.07206 . · arXiv
  3. Bach, S., Binder, A., Montavon, G., Klauschen, F., M¨ uller, K.-R., and Samek, W. (2015). On pixelwise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one , 10 (7), e0130140.
  4. Biggio, B., Corona, I., Maiorca, D., Nelson, B., ˇ Srndi´ c, N., Laskov, P., Giacinto, G., and Roli, F. (2013). Evasion attacks against machine learning at test time. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 387-402. Springer.
  5. Binder, A., Montavon, G., Lapuschkin, S., M¨ uller, K.-R., and Samek, W. (2016). Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks , pages 63-71. Springer.
  6. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., and Darrell, T. (2014). Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning , pages 647-655.
  7. Dosovitskiy, A. and Brox, T. (2016). Inverting visual representations with convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4829-4837.
  8. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems , pages 2672-2680.
  9. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770778.
  10. Jarrett, K., Kavukcuoglu, K., Lecun, Y., et al. (2009). What is the best multi-stage architecture for object recognition? In 2009 IEEE 12th International Conference on Computer Vision , pages 2146-2153. IEEE.
  11. Jastrzebski, S., Arpit, D., Ballas, N., Verma, V ., Che, T., and Bengio, Y . (2017). Residual connections encourage iterative inference. arXiv preprint arXiv:1710.04773 . · arXiv
  12. Lapuschkin, S., Binder, A., Montavon, G., M¨ uller, K.-R., and Samek, W. (2016). Analyzing classifiers: Fisher vectors and deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2912-2920.
  13. Larsson, G., Maire, M., and Shakhnarovich, G. (2016). Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648 . · arXiv
  14. Mahendran, A. and Vedaldi, A. (2015). Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5188-5196.
  15. Mahendran, A. and Vedaldi, A. (2016). Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision , 120 (3), 233-255.
  16. Montavon, G., Braun, M. L., and M¨ uller, K.-R. (2011). Kernel analysis of deep networks. Journal of Machine Learning Research , 12 (Sep), 2563-2581.
  17. Raghu, M., Yosinski, J., and Sohl-Dickstein, J. (2017a). Bottom up or top down? dynamics of deep representations via canonical correlation analysis. arxiv .
  18. Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. (2017b). Svcca: Singular vector canonical correlation analysis for deep understanding and improvement. arXiv preprint arXiv:1706.05806 . · arXiv
  19. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115 (3), 211-252.
  20. Singh, S., Hoiem, D., and Forsyth, D. (2016). Swapout: Learning an ensemble of deep architectures. In Advances In Neural Information Processing Systems , pages 28-36.
  21. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2013). Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 . · arXiv
  22. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1-9.
  23. Veit, A., Wilber, M. J., and Belongie, S. (2016). Residual networks behave like ensembles of relatively shallow networks. In Advances in Neural Information Processing Systems , pages 550558.
  24. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., and Bengio, Y. (2015). Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning , pages 2048-2057.
  25. Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. (2014). How transferable are features in deep neural networks? In Advances in neural information processing systems , pages 3320-3328.
  26. Zeiler, M. D. and Fergus, R. (2014). Visualizing and understanding convolutional networks. In European conference on computer vision , pages 818-833. Springer.
  27. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 . · arXiv

Cited by

Sorted by citations

Authors
Alain, G., & Bengio, Y.
Venue
arXiv
Publisher
arXiv
Published
2016
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1610.01644
arXiv
1610.01644
OpenAlex
W2529194139
Cited by
581
References
0
Citations by year
2026: 125 · 2025: 68 · 2024: 63 · 2023: 80 · 2022: 70 · 2021: 66 · 2020: 47 · 2019: 33 · 2018: 21 · 2017: 8
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Learning Representations
  • Zenodo (CERN European Organization for Nuclear Research) (cc-by)
  • SSRN Electronic Journal (acceptedVersion, cc-by)
Funders
  • Canadian Institute for Advanced Research
  • Compute Canada
  • Canada Research Chairs
  • Fonds Québécois de la Recherche sur la Nature et les Technologies
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
neural network interpretability, intermediate layer features, model depth, InceptionV3, ResNet50, black-box models, model diagnosis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review