Large language models demonstrate the potential of statistical learning in language
Open accessAbstract
To what degree can language be acquired from linguistic input alone? This question has vexed scholars for millennia and is still a major focus of debate in the cognitive science of language. The complexity of human language has hampered progress because studies of language–especially those involving computational modeling–have only been able to deal with small fragments of our linguistic skills. We suggest that the most recent generation of Large Language Models (LLMs) might finally provide the computational tools to determine empirically how much of the human language ability can be acquired from linguistic experience. LLMs are sophisticated deep learning architectures trained on vast amounts of natural language data, enabling them to perform an impressive range of linguistic tasks. We argue that, despite their clear semantic and pragmatic limitations, LLMs have already demonstrated that human‐like grammatical language can be acquired without the need for a built‐in grammar. Thus, while there is still much to learn about how humans acquire and use language, LLMs provide full‐fledged computational models for cognitive scientists to empirically evaluate just how far statistical learning might take us in explaining the full complexity of human language.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The output of large language models is, almost without exception, grammatical.
“The key point that we want to make is that the output of [large language models] is, almost without exception, grammatical.”
From Natural syntax, artificial intelligence and language acquisition (2023) · cites this paper as [1]
Large language models have remarkable grammatical accuracy that seems to surpass even the most articulate humans.
“A striking feature of contemporary large language models (LLMs)-ChatGPT, GPT4, BERT and others-is their remarkable grammatical accuracy, which seems to surpass the abilities of even the most articulate of humans.”
From Natural syntax, artificial intelligence and language acquisition (2023) · cites this paper as [1]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686 · see citing works
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Pinker, S. A. (1994) · PsycEXTRA Dataset · doi:10.1037/e412952005-009 · Cited by 5,879 · see citing works
Skinner, B. F. (1957) · doi:10.1037/11256-000 · Cited by 5,759 · see citing works
Chomsky (1959) · doi:10.2307/411334 · Cited by 3,425 · see citing works
Chomsky, N. A. D. T. (1980) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00001515 · Cited by 2,935 · see citing works
Rethinking Innateness: A Connectionist Perspective on Development
Harris, M. J., Karmiloff‐Smith, A., Parisi, D., & Plunkett, K. (1996) · Medical Entomology and Zoology · Cited by 1,990 · see citing works
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552 · see citing works
Rethinking Innateness: A Connectionist Perspective on Development
Elman, J., Bates, E., Johnson, M. H., Karmiloff-Smith, A., Parisi, D., & Plunkett, K. (1996) · doi:10.7551/mitpress/5929.001.0001 · Cited by 1,511 · see citing works
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models
Ettinger, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00298 · Cited by 511 · see citing works
Mapping the Early Language Environment Using All-Day Recordings and Automated Analysis
Gilkerson, J., Richards, J. A., Warren, S. F., Montgomery, J. K., Greenwood, C. R., Kimbrough Oller, D., Hansen, J. H. L., & Paul, T. D. (2017) · American Journal of Speech-Language Pathology · doi:10.1044/2016_ajslp-15-0169 · Cited by 472 · see citing works
Shared computational principles for language processing in humans and deep language models
Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., Nastase, S. A., Feder, A., Emanuel, D., Cohen, A., et al. (2022) · Nature Neuroscience · doi:10.1038/s41593-022-01026-4 · Cited by 447 · see citing works
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2208.07339 · Cited by 305 · see citing works
What is the human language faculty? Two views
Jackendoff, R. S. (2011) · Language · doi:10.1353/lan.2011.0063 · Cited by 250 · see citing works
The growth of language: Universal Grammar, experience, and principles of computation
Yang, C. D., Crain, S., Berwick, R. C., Chomsky, N., & Bolhuis, J. J. (2017) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.12.023 · Cited by 241 · see citing works
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · see citing works
Christiansen, M. H., & Chater, N. (2016) · The MIT Press eBooks · doi:10.7551/mitpress/10406.001.0001 · Cited by 212 · see citing works
Self-citationThe usage-based theory of language acquisition
Tomasello, M. (2012) · Cambridge University Press eBooks · doi:10.1017/cbo9780511576164.005 · Cited by 202 · see citing works
Goldberg, A. E. (2019) · Princeton University Press eBooks · doi:10.2307/j.ctvc772nn · Cited by 199 · see citing works
Creating Language: Integrating Evolution, Acquisition, and Processing
Christiansen, M. H., & Chater, N. (2016) · Project Muse (Johns Hopkins University) · Cited by 186 · see citing works
Self-citationConnectionist models of cognitive development: where next?
Elman, J. L. (2005) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2005.01.005 · Cited by 173 · see citing works
Neural language models as psycholinguistic subjects: Representations of syntactic state
Futrell, R., Wilcox, E., Morita, T., Qian, P., Ballesteros, M., & Lévy, R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1004 · Cited by 154 · see citing works
What exactly is Universal Grammar, and has anyone seen it?
Dąbrowska, E. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00852 · Cited by 153 · see citing works
Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N., & Choi, Y. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.501 · Cited by 118 · see citing works
The language capacity: architecture and evolution
Chomsky, N. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1078-6 · Cited by 108 · see citing works
McClelland, J. L., Hill, F., Rudolph, M., Baldridge, J., & Schütze, H. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1910416117 · Cited by 91 · see citing works
Dou, Y., Forbes, M., Koncel-Kedziorski, R., Smith, N. A., & Choi, Y. (2021) · arXiv · doi:10.48550/arxiv.2107.01294 · Cited by 90 · see citing works
Using Computational Models to Test Syntactic Learnability
Wilcox, E., Futrell, R., & Lévy, R. (2022) · Linguistic Inquiry · doi:10.1162/ling_a_00491 · Cited by 66 · see citing works
First language development: a usage-based perspective on past and current research
Lieven, E. (2014) · Journal of Child Language · doi:10.1017/s0305000914000282 · Cited by 40 · see citing works
Models of Language and Multiword Expressions
Kallens, P. C., & Christiansen, M. H. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.781962 · Cited by 37 · see citing works
Self-citationArehalli, S., Dillon, B., & Linzen, T. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.20 · Cited by 30 · see citing works
Pandia, L., & Ettinger, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.119 · Cited by 25 · see citing works
Goldstein, A., Ham, E., Nastase, S. A., Zada, Z., Grinstein-Dabush, A., Aubrey, B., Schain, M., Gazula, H., Feder, A., Doyle, W., et al. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.07.11.499562 · Cited by 22 · see citing works
Arehalli, S., Dillon, B., & Linzen, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.12187 · Cited by 7 · see citing works
Jackendoff, R. S., & Audring, J. (2019) · De Gruyter eBooks · doi:10.1515/9783110540253-008 · Cited by 5 · see citing works
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
Flagged first, then by citations
Research works printed but not linked: 7
- Bender, E. M., Gebru, T., McMillan-Major, A., & Schmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of FAccT 2021, Canada (pp. 610-623).
- BigScience Workshop. (2022). BLOOM. Hugging Face. Available at: https://huggingface.co/bigscience/bloom P. Contreras Kallens, R. D. Kristensen-McLachlan, M. H. Christiansen / Cognitive Science 47 (2023)5 of 6
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901.
- Chomsky, N. (1995). The minimalist program. Cambridge, MA: The MIT Press.
- Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., & Fiedel, N. (2022). PaLM: Scaling language modeling with pathways. https://arxiv.org/abs/ 2204.02311
- Hosseini, E. A., Schrimpf, M. A., Zhang, Y., Bowman, S., Zaslavsky, N., & Fedorenko, E. (2022). Artificial neural network language models align neurally and behaviorally with humans even after a developmentally realistic amount of training. BioRxiv. https://doi.org/10.1101/2022.07.11.499562 · DOI
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems 30, Long Beach, CA.
Cited by
Sorted by citations
Natural syntax, artificial intelligence and language acquisition
O’Grady, W., & Lee, M. (2023) · Information · doi:10.3390/info14070418 · Cited by 5 · see citing works
- Authors
- Contreras Kallens, P., Kristensen‐McLachlan, R. D., & Christiansen, M. H.
- Affiliations
- Department of Psychology Cornell University
- Aarhus University (DK)
- Haskins Laboratories (US)
- Corresponding author
- Christiansen, Morten H.
- Venue
- Cognitive Science
- Publisher
- Wiley
- Published
- 2023-02-25
- Volume
- 47
- Issue
- 3
- Pages
- e13256-e13256
- ISSN
- 0364-0213 (ISSN-L)
- 1551-6709
- OpenAlex
- W4322006551
- Cited by
- 130
- References
- 36
- Citations by year
- 2026: 20 · 2025: 49 · 2024: 42 · 2023: 18 · 2022: 1
- FWCI
- 12.16
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Cognitive Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Psychology › Developmental and Educational Psychology › Language Development and Disorders
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Keywords
- large language models, statistical learning, language acquisition, language experience, computational models of language, grammar
- MeSH terms
- Cognition, Humans, Language, Language Development, Linguistics, Semantics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
