Linguistic generalization and compositionality in modern artificial neural networks
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
In the last decade, deep artificial neural networks have achieved astounding performance in many natural language-processing tasks. Given the high productivity of language, these models must possess effective generalization abilities. It is widely assumed that humans handle linguistic productivity by means of algebraic compositional rules: are deep networks similarly compositional? After reviewing the main innovations characterizing current deep language-processing networks, I discuss a set of studies suggesting that deep networks are capable of subtle grammar-dependent generalizations, but also that they do not rely on systematic compositional rules. I argue that the intriguing behaviour of these devices (still awaiting a full understanding) should be of interest to linguists and cognitive scientists, as it offers a new perspective on possible computational strategies to deal with linguistic productivity beyond rule-based compositionality, and it might lead to new insights into the less systematic generalization patterns that also appear in natural language. This article is part of the theme issue ‘Towards mechanistic models of meaning composition’.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“Gated recurrent networks, such as long short-term memory networks [36] and gated recurrent units [37], possess mechanisms regulating the dynamics of information processing across time, whose parameters are jointly induced with the rest of the network in the training phase.”
BackgroundLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686
ASPECTS OF THE THEORY OF SYNTAX
Chomsky, N. (1964) · doi:10.21236/ad0616323 · Cited by 13,587
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Chomsky, N. (1957) · doi:10.1515/9783112316009 · Cited by 7,885
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Landauer, T. K., & Dumais, S. T. (1997) · Psychological Review · doi:10.1037/0033-295x.104.2.211 · Cited by 6,282
Connectionism and cognitive architecture: A critical analysis
Fodor, J. A., & Pylyshyn, Z. W. (1988) · Cognition · doi:10.1016/0010-0277(88)90031-5 · Cited by 4,022
Supervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Cited by
Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks
Hasson, U., Nastase, S. A., & Goldstein, A. (2020) · Neuron · doi:10.1016/j.neuron.2019.12.002 · Cited by 387
van Rooij, I., & Baggio, G. (2021) · Perspectives on Psychological Science · doi:10.1177/1745691620970604 · Cited by 354
Explainability in deep reinforcement learning
Heuillet, A., Couthouis, F., & Díaz-Rodríguez, N. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106685 · Cited by 319
Evidence of a predictive coding hierarchy in the human brain listening to speech
Caucheteux, C., Gramfort, A., & King, J.-R. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01516-2 · Cited by 308
Flexible multitask computation in recurrent networks utilizes shared dynamical motifs
Driscoll, L., Shenoy, K., & Sussillo, D. (2022) · doi:10.1101/2022.08.15.503870 · Cited by 217
Deep language algorithms predict semantic comprehension from brain activity
Caucheteux, C., Gramfort, A., & King, J.-R. (2022) · Scientific Reports · doi:10.1038/s41598-022-20460-9 · Cited by 91
Does ChatGPT have semantic understanding? A problem with the statistics-of-occurrence strategy
Titus, L. M. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101174 · Cited by 82
How deep is the brain? The shallow brain hypothesis
Suzuki, M., Pennartz, C. M. A., & Aru, J. (2023) · Nature Reviews Neuroscience · doi:10.1038/s41583-023-00756-z · Cited by 79
Beguš, G. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.03.017 · Cited by 42
Automated scholarly paper review: Concepts, technologies, and challenges
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101830 · Cited by 37
This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models
García-Ferrero, I., Altuna, B., Álvez, J., Gonzalez-Dios, I., & Rigau, G. (2023) · arXiv · doi:10.48550/arxiv.2310.15941 · Cited by 18
Metadata
- Authors
- Baroni, M.
- Venue
- Philosophical Transactions of the Royal Society B: Biological Sciences
- Publisher
- The Royal Society
- Published
- 2019-12-16
- Volume
- 375
- Issue
- 1791
- Pages
- 20190307
- ISSN
- 0962-8436 (ISSN-L)
- 1471-2970
- OpenAlex
- W2996556191
- Cited by
- 102
- References
- 26
- FWCI
- 51.71
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.