The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Unit Testing for Concepts in Neural Networks
Open accessAbstract
Many complex problems are naturally understood in terms of symbolic concepts. For example, our concept of “cat” is related to our concepts of “ears” and “whiskers” in a non-arbitrary way. Fodor (1998) proposes one theory of concepts, which emphasizes symbolic representations related via constituency structures. Whether neural networks are consistent with such a theory is open for debate. We propose unit tests for evaluating whether a system’s behavior is consistent with several key aspects of Fodor’s criteria. Using a simple visual concept learning task, we evaluate several modern neural architectures against this specification. We find that models succeed on tests of groundedness, modularity, and reusability of concepts, but that important questions about causality remain open. Resolving these will require new methods for analyzing models’ internal states.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Models trained end-to-end on the higher-level concept labelling task (i.e. with no training to explicitly encourage encoding information about shape, stroke or layout) nonetheless learn representations of the constituent concepts. These representations can be localized to linear combinations of activations at a given layer”
From Symbols and grounding in large language models (2023) · cites this paper as [40]
In the citing paper
“First, the representations of the constituents are reused across different high-level concepts. That is, the representation of circle in the context of dax is the same (encoded by the same linear combination of activations) as the representation of circle in the context of wix.”
From Symbols and grounding in large language models (2023) · cites this paper as [40]
In the citing paper
“Second, the representations are modular. That is, it is possible to ablate the representation of the constituent shape at a given layer without impacting the model’s knowledge of the layout or the stroke.”
From Symbols and grounding in large language models (2023) · cites this paper as [40]
Citations
Cites
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,667
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Connectionism and cognitive architecture: A critical analysis
Fodor, J. A., & Pylyshyn, Z. W. (1988) · Cognition · doi:10.1016/0010-0277(88)90031-5 · Cited by 4,022
The Origin of Concepts: (513702014-002)
Carey, S. (2013) · PsycEXTRA Dataset · doi:10.1037/e513702014-002 · Cited by 2,312
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Building high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695
International Conference on Learning Representations (ICLR)
Abbas, M. Z. A., Kivinen, J., & Raiko, T. (2016) · Cited by 1,694
Concepts: Where Cognitive Science Went Wrong
Fodor, J. A. (1998) · doi:10.1093/0198236360.001.0001 · Cited by 1,274
Mitzias, P., Riga, M., Waddington, S. N., Kontopoulos, E., Μεδίτσκος, Γ., Laurenson, P., & Kompatsiaris, I. Y. (2015) · Cited by 1,240
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240
Research works printed but not linked: 21
- Adebayo, 2020, Debugging tests for model explanations, arXiv preprint arXiv:2011.05429 · arXiv
- Adi, 2017, Fine- grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
- Andreas, 2019, Measuring compositionality in representation learning
- Andreas, 2016, Neural module networks, 39
- Carey, 2009, The Origin of Concepts, 10.1093/acprof:oso/9780195367638.001.0001 · DOI · Paper page
- Chang, 2018, Explaining image classifiers by counterfactual generation, arXiv preprint arXiv:1807.08024 · arXiv
- Dosovitskiy, 2020, An image is worth 16 × 16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929 · arXiv
- Goyal, 2019, Explaining classifiers with causal concept effect, arXiv preprint arXiv:1907. 07165
- Goyal, 2019, Counterfactual visual explanations, 2376
- He, 2016, Deep residual learning for image recognition, 770
- Hupkes, 2018, Visualisation and ‘diagnostic classifiers’ reveal how recurrent and recursive neural networks process hierarchical structure, Journal of Artificial Intelligence Research, 61, 907, 10.1613/jair.1.11196 · DOI · Paper page
- Ioffe, 2015, Batch normalization: Accelerating deep network training by reducing internal covariate shift, 448
- Jackman, 2020, Meaning Holism
- Lake, 2018, Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
- Lovering, 2020, Predicting inductive biases of pre-trained models
- Meng, 2022, Locating and editing factual knowledge in GPT, arXiv preprint arXiv:2202.05262 · arXiv
- Radford, 2021, Learning transferable visual models from natural language supervision, 8748, 10.48550/arXiv.2103.00020 · DOI · Paper page
- Sermanet, 2013, Overfeat: Integrated recognition, localization and detection using convolutional networks, arXiv preprint arXiv:1312.6229 · arXiv
- Sundararajan, 2017, Axiomatic attribution for deep networks, 3319
- Tenney, 2019, What do you learn from context? Probing for sentence structure in contextualized word representations
- Veldhoen, 2016, Diagnostic classifiers: Revealing how neural networks process hierarchical structure
Cited by
Eight Things to Know about Large Language Models
Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00612 · Cited by 137
Symbols and grounding in large language models
Pavlick, E. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0041 · Cited by 95
“These properties are highly consistent with what we would expect from an explicitly symbolic model designed to solve the same task (figure 1 b ). in [40].”
BackgroundResultSelf-citation
Metadata
- Authors
- Lovering, C., & Pavlick, E.
- Affiliations
- Brown University (US)
- Corresponding author
- Lovering, Charles
- Pavlick, Ellie
- ORCID
- Pavlick, Ellie
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2022
- Volume
- 10
- Pages
- 1193-1208
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4313119036
- Cited by
- 12
- References
- 40
- Citations by year
- 2025: 2 · 2024: 4 · 2023: 4 · 2022: 2
- FWCI
- 2.76
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Brown University
- Defense Advanced Research Projects Agency
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Social Sciences › Arts and Humanities › Philosophy › Philosophy and Theoretical Science
- Keywords
- unit testing, artificial neural networks, symbolic concepts, visual concept learning, groundedness, modularity, causality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Unit Testing for Concepts in Neural Networks. https://app.cochat.ai/papers/10.1162/tacl_a_00514
- The paper
- Lovering, C., & Pavlick, E. (2022). Unit Testing for Concepts in Neural Networks. Transactions of the Association for Computational Linguistics. https://doi.org/10.1162/tacl_a_00514
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.