Unit Testing for Concepts in Neural Networks
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Many complex problems are naturally understood in terms of symbolic concepts. For example, our concept of “cat” is related to our concepts of “ears” and “whiskers” in a non-arbitrary way. Fodor (1998) proposes one theory of concepts, which emphasizes symbolic representations related via constituency structures. Whether neural networks are consistent with such a theory is open for debate. We propose unit tests for evaluating whether a system’s behavior is consistent with several key aspects of Fodor’s criteria. Using a simple visual concept learning task, we evaluate several modern neural architectures against this specification. We find that models succeed on tests of groundedness, modularity, and reusability of concepts, but that important questions about causality remain open. Resolving these will require new methods for analyzing models’ internal states.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Models trained end-to-end on the higher-level concept labelling task (i.e. with no training to explicitly encourage encoding information about shape, stroke or layout) nonetheless learn representations of the constituent concepts. These representations can be localized to linear combinations of activations at a given layer”
From Symbols and grounding in large language models (2023) · cites this paper as [40]
In the citing paper
“First, the representations of the constituents are reused across different high-level concepts. That is, the representation of circle in the context of dax is the same (encoded by the same linear combination of activations) as the representation of circle in the context of wix.”
From Symbols and grounding in large language models (2023) · cites this paper as [40]
In the citing paper
“Second, the representations are modular. That is, it is possible to ablate the representation of the constituent shape at a given layer without impacting the model’s knowledge of the layout or the stroke.”
From Symbols and grounding in large language models (2023) · cites this paper as [40]
Citations
Cites
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Connectionism and cognitive architecture: A critical analysis
Fodor, J. A., & Pylyshyn, Z. W. (1988) · Cognition · doi:10.1016/0010-0277(88)90031-5 · Cited by 4,022
Carey, S. (2009) · doi:10.1093/acprof:oso/9780195367638.001.0001 · Cited by 2,322
The Origin of Concepts: (513702014-002)
Carey, S. (2013) · PsycEXTRA Dataset · doi:10.1037/e513702014-002 · Cited by 2,312
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Building high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695 · cited 1×
“First, work on identifying concepts in neural networks seeks interpretable patterns in the activations and gradients of neural networks, e.g., that unsupervised CNNs encode concepts such as edges (Ser-manet et al., 2013; Le, 2013).”
BackgroundInternational Conference on Learning Representations (ICLR)
Abbas, M. Z. A., Kivinen, J., & Raiko, T. (2016) · Cited by 1,694
Concepts: Where Cognitive Science Went Wrong
Fodor, J. A. (1998) · doi:10.1093/0198236360.001.0001 · Cited by 1,274
Mitzias, P., Riga, M., Waddington, S. N., Kontopoulos, E., Μεδίτσκος, Γ., Laurenson, P., & Kompatsiaris, I. Y. (2015) · Cited by 1,240
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240
Margolis, E., & Laurence, S. (1999) · The MIT Press eBooks · Cited by 1,181
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097
Spelke, E. S., & Kinzler, K. D. (2007) · Developmental Science · doi:10.1111/j.1467-7687.2007.00569.x · Cited by 1,089
Attention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · cited 1×
“Many techniques have been proposed in order to determine which input features are “important” to model decisions (Ribeiro et al., 2016; Sundararajan et al., 2017; Kim et al., 2018; Wiegreffe and Pinter, 2019).”
BackgroundBeyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890
Causal Models: How People Think about the World and Its Alternatives
Sloman, S. (2005) · doi:10.1093/acprof:oso/9780195183115.001.0001 · Cited by 677 · cited 1×
“The subset of Fodor’s criteria on which we focus (§3) are fairly uncontroversial, and arguably would transfer to alternative theories of conceptual structure–for example, Bayesian causal models (Sloman, 2005).”
BackgroundBlock, N. (1981) · The Philosophical Review · doi:10.2307/2184371 · Cited by 562
Distributional Models of Word Meaning
Lenci, A. (2018) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-030514-125254 · Cited by 447
Semantically Equivalent Adversarial Rules for Debugging NLP models
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1079 · Cited by 432
Vector Space Models of Word Meaning and Phrase Meaning: A Survey
Erk, K. (2012) · Language and Linguistics Compass · doi:10.1002/lnco.362 · Cited by 321
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.647 · Cited by 287
Distributional Semantics and Linguistic Theory
Boleda, G. (2020) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-011619-030303 · Cited by 213
Syntactic Structure from Deep Learning
Linzen, T., & Baroni, M. (2021) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-032020-051035 · Cited by 202
Measuring and Reducing Gendered Correlations in Pre-trained Models
Webster, K., Wang, X., Tenney, I., Beutel, A., Pitler, E., Pavlick, E., Chen, J., Chi, E., & Petrov, S. (2020) · arXiv · doi:10.48550/arxiv.2010.06032 · Cited by 195
Self-citationSinha, K., Jia, R., Hupkes, D., Pineau, J., Williams, A., & Kiela, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.230 · Cited by 181
COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
Kim, N., & Linzen, T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.731 · Cited by 153
(2020) · neural information processing systems · Cited by 142
Probing for semantic evidence of composition by means of simple classification tasks
Ettinger, A., Elgohary, A., & Resnik, P. (2016) · Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP · doi:10.18653/v1/w16-2524 · Cited by 129
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00359 · Cited by 126
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation
Huang, P.-S., Zhang, H., Jiang, R., Stanforth, R., Welbl, J., Rae, J., Maini, V., Yogatama, D., & Kohli, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.7 · Cited by 124
Can Neural Networks Understand Monotonicity Reasoning?
Yanaka, H., Mineshima, K., Bekki, D., Inui, K., Sekine, S., Abzianidze, L., & Bos, J. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4804 · Cited by 99
Warstadt, A., Zhang, Y., Li, X., Liu, H., & Bowman, S. R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.16 · Cited by 98
Correlating Neural and Symbolic Representations of Language
Chrupała, G., & Alishahi, A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1283 · Cited by 96
Debugging Tests for Model Explanations
Adebayo, J., Muelly, M., Liccardi, I., & Kim, B. (2020) · arXiv · doi:10.48550/arxiv.2011.05429 · Cited by 88
Probing Linguistic Systematicity
Goodwin, E., Sinha, K., & O’Donnell, T. J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.177 · Cited by 68
Semantic Structure in Deep Learning
Pavlick, E. (2022) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-122924 · Cited by 66
Self-citationMerrill, W., Goldberg, Y., Schwartz, R., & Smith, N. A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00412 · Cited by 44 · cited 1×
“…mappings between humans’ and neural networks’ conceptual spaces, e.g., by defining measures of compositionality or groundedness based on how well similarity in vector space reflects similarity according to a symbolic representation (Andreas, 2019; Chrupała and Alishahi, 2019; Merrill et al., 2021).”
BackgroundWhen Bert Forgets How To POS: Amnesic Probing of Linguistic Properties and MLM PredictionsPreprint
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2020) · arXiv (Cornell University) · Cited by 22
(2022) · doi:10.18653/v1/2022.findings-acl.106 · Cited by 17
What if This Modified That? Syntactic Interventions with Counterfactual Embeddings
Tucker, M., Qian, P., & Levy, R. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.76 · Cited by 16
Research works printed but not linked: 19
- Adi, 2017, Fine- grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
- Andreas, 2019, Measuring compositionality in representation learning
- Andreas, 2016, Neural module networks, 39
- Chang, 2018, Explaining image classifiers by counterfactual generation, arXiv preprint arXiv:1807.08024 · arXiv
- Dosovitskiy, 2020, An image is worth 16 × 16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929 · arXiv
- Goyal, 2019, Explaining classifiers with causal concept effect, arXiv preprint arXiv:1907. 07165
- Goyal, 2019, Counterfactual visual explanations, 2376
- He, 2016, Deep residual learning for image recognition, 770
- Hupkes, 2018, Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure, Journal of Artificial Intelligence Research, 61, 907, 10.1613/jair.1.11196 · DOI · Paper page
- Ioffe, 2015, Batch normalization: Accelerating deep network training by reducing internal covariate shift, 448
- Jackman, 2020, Meaning Holism
- Lake, 2018, Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
- Lovering, 2020, Predicting inductive biases of pre-trained models
- Meng, 2022, Locating and editing factual knowledge in GPT, arXiv preprint arXiv:2202.05262 · arXiv
- Radford, 2021, Learning transferable visual models from natural language supervision, 8748, 10.48550/arXiv.2103.00020 · DOI · Paper page
- Sermanet, 2013, Overfeat: Integrated recognition, localization and detection using convolutional networks, arXiv preprint arXiv:1312.6229 · arXiv
- Sundararajan, 2017, Axiomatic attribution for deep networks, 3319
- Tenney, 2019, What do you learn from context? Probing for sentence structure in contextualized word representations
- Veldhoen, 2016, Diagnostic classifiers: Revealing how neural networks process hierarchical structure
Cited by
Eight Things to Know about Large Language Models
Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00612 · Cited by 137
Symbols and grounding in large language models
Pavlick, E. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0041 · Cited by 95
“These properties are highly consistent with what we would expect from an explicitly symbolic model designed to solve the same task (figure 1 b ). in [40].”
BackgroundResultSelf-citation
Metadata
- Authors
- Lovering, C., & Pavlick, E.
- Affiliations
- Brown University (US)
- Corresponding author
- Lovering, Charles
- Pavlick, Ellie
- ORCID
- Pavlick, Ellie
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2022
- Volume
- 10
- Pages
- 1193-1208
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4313119036
- Cited by
- 12
- References
- 42
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 4 · 2022: 2
- FWCI
- 2.76
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Brown University
- Defense Advanced Research Projects Agency
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Social Sciences › Arts and Humanities › Philosophy › Philosophy and Theoretical Science
- Keywords
- unit testing, artificial neural networks, symbolic concepts, visual concept learning, groundedness, modularity, causality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.