CoChat
Sign Up
Cited by 111Language Development and DisordersNatural Language Processing TechniquesSyntax, Semantics, Linguistic Variation

What Artificial Neural Networks Can Tell Us about Human Language Acquisition

Open access
Warstadt, Alex, Bowman, Samuel R.·2022·Algebraic Structures in Natural Language
DOI 10.1201/9781003205388-2
Open this paper in CoChat — free accountOpen PDF

Abstract

Rapid progress in machine learning for natural language processing has the potential to transform debates about how humans learn language. However, the learning environments and biases of current artificial learners and humans diverge in ways that weaken the impact of the evidence gleaned from learning simulations. Today's most effective neural language models are trained on roughly one thousand times the amount of linguistic data available to a typical child. To increase the relevance of learnability results from computational models, we need to train model learners without significant advantages over humans. If an appropriate model successfully acquires some target linguistic knowledge, it can provide a proof of concept that the target is learnable in a hypothesised human learning scenario. Plausible model learners will enable us to carry out experimental manipulations to make causal inferences about variables in the learning environment, and to rigorously test poverty-of-the-stimulus-style claims arguing for innate linguistic knowledge in humans on the basis of speculations about learnability. Comparable experiments will never be possible with human subjects due to practical and ethical considerations, making model learners an indispensable resource. So far, attempts to deprive current models of unfair advantages obtain sub-human results for key grammatical behaviours such as acceptability judgements. But before we can justifiably conclude that language learning requires more prior domain-specific knowledge than current models possess, we must first explore non-linguistic inputs in the form of multimodal stimuli and multi-agent interaction as ways to make our learners more efficient at learning from limited linguistic input.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1201/9781003205388-2

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Modern language models are typically trained on 3 or 4 orders-of-magnitude more data than children.

    “Large language models are far less data-efficient: they typically require 3 or 4 orders of magnitude more data and still do not perform as well as humans on many evaluations.”

    From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Warstadt and Bowman (2022)

    Section 4 says widely studied language models encounter hundreds or thousands of times more words than a child, supporting the broad data-efficiency comparison. Section 4.1 gives model training sizes from 1 billion to 200 billion words, versus tens of millions to about 100 million words for a child. Those ranges do not support saying models typically use three or four orders of magnitude more data; the lower end is substantially less. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 141

  1. Omri Abend, Tom Kwiatkowski, Nathaniel J. Smith, Sharon Goldwater, and Mark Steed- man. Bootstrapping language acquisition. Cognition, 164:116-143, July 2017. ISSN 00100277. doi: 10.1016/j.cognition.2017.02.009. URL https://linkinghub. elsevier.com/retrieve/pii/S0010027717300495. · DOI
  2. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceedings of ICLR Conference Track. Toulon, France., 2017.
  3. JL Austin. How to Do Things With Words. Oxford University Press, 1962.
  4. Marco Baroni. On the proper role of linguistically-oriented deep net analysis in linguistic theorizing. arXiv:2106.08694 [cs], June 2021. URL http://arxiv.org/abs/ 2106.08694. arXiv: 2106.08694. · arXiv
  5. Yonatan Belinkov and James R. Glass. Analysis Methods in Neural Language Process- ing: A Survey. Transactions of the Association for Computational Linguistics, 7:49-72, 2019. URL https://www.aclweb.org/anthology/Q19-1004.pdf.
  6. Robert C Berwick, Paul Pietroski, Beracah Yankama, and Noam Chomsky. Poverty of the stimulus revisited. Cognitive Science, 35(7):1207-1242, 2011.
  7. Diane Bouchacourt and Marco Baroni. How agents see things: On visual representa- tions in an emergent language game. In Proceedings of the 2018 conference on em- pirical methods in natural language processing, pages 981-985, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1119. URL https://aclanthology.org/D18-1119. · DOI
  8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Pro- cessing Systems, 2020. URL https://papers.nips.cc/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  9. Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti- efficient encoding in emergent communication. Advances in Neural Information Pro- cessing Systems, 32, 2019.
  10. Rahma Chaabouni, Florian Strub, Florent Altché, Eugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent communication at scale. In International conference on learning repre- sentations, 2022.
  11. Tyler A. Chang and Benjamin K. Bergen. Word Acquisition in Neural Language Models. Transactions of the Association for Computational Linguistics, 10: 1-16, January 2022. ISSN 2307-387X. doi: 10.1162/tacl a 00444. URL https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_ 00444/109271/Word-Acquisition-in-Neural-Language-Models. · DOI
  12. Rui P. Chaves. What Don't RNN Language Models Learn About Filler-Gap Dependencies? In Proceedings of the third meeting of the Society for Computation in Linguistics (SCiL), 2020.
  13. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dol- lar, and C. Lawrence Zitnick. Microsoft COCO Captions: Data Collection and Evalua- tion Server. In European conference on computer vision, pages 740-755. Springer, April 2015. URL http://arxiv.org/abs/1504.00325. arXiv: 1504.00325. · arXiv
  14. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: UNiversal image-text representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Com- puter vision -ECCV 2020, pages 104-120, Cham, 2020. Springer International Publish- ing. ISBN 978-3-030-58577-8.
  15. Noam Chomsky. Aspects of the Theory of Syntax. MIT Press, 1965.
  16. Noam Chomsky. Biolinguistic Explorations: Design, Development, Evolution. In- ternational Journal of Philosophical Studies, 15(1):1-21, January 2007. ISSN 0967-2559, 1466-4542. doi: 10.1080/09672550601143078. URL http://www. tandfonline.com/doi/abs/10.1080/09672550601143078. · DOI
  17. Noam Chomsky and Howard Lasnik. The theory of principles and parameters. In The minimalist program. MIT Press, 1993.
  18. Leshem Choshen, Guy Hacohen, Daphna Weinshall, and Omri Abend. The grammar- learning trajectories of neural language models. arXiv preprint arXiv:2109.06096, 2021. · arXiv
  19. Michelle M Chouinard and Eve V Clark. Adult reformulations of child errors as nega- tive evidence. Journal of child language, 30(3):637-669, 2003. Publisher: Cambridge University Press.
  20. Shammur Absar Chowdhury and Roberto Zamparelli. RNN simulations of grammatical- ity judgments on long-distance dependencies. In Proceedings of the 27th international conference on computational linguistics, pages 133-144, 2018.
  21. Morten H Christiansen and Nick Chater. Creating language: Integrating evolution, acqui- sition, and processing. MIT Press, 2016.
  22. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical Eval- uation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv:1412.3555 [cs], December 2014. URL http://arxiv.org/abs/1412.3555. arXiv: 1412.3555. · arXiv
  23. Alexander Clark and Shalom Lappin. Linguistic Nativism and the Poverty of the Stimulus. John Wiley & Sons, 2011.
  24. Alejandrina Cristia, Emmanuel Dupoux, Michael Gurven, and Jonathan Stieglitz. Child- Directed Speech Is Infrequent in a Forager-Farmer Population: A Time Allocation Study. Child Development, 90(3):759-773, May 2019. ISSN 0009-3920, 1467-8624. doi: 10.1111/cdev.12974. URL https://onlinelibrary.wiley.com/doi/ 10.1111/cdev.12974. · DOI
  25. Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Ad- vances in neural information processing systems, volume 28.
  26. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/file/ 7137debd45ae4d0ab9aa953017286b20-Paper.pdf.
  27. Mark Davies. The 385+ million word Corpus of Contemporary American English (1990-2008+): Design, architecture, and linguistic insights. International journal of corpus linguistics, 14(2):159-190, 2009. Publisher: John Benjamins.
  28. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, 2019. URL https://www.aclweb.org/anthology/N19-1423.
  29. Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. Neu- ral Models for Reasoning over Multiple Mentions Using Coreference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 42-48, 2018.
  30. Emmanuel Dupoux. Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner. Cognition, 173:43-59, April 2018. ISSN 00100277. doi: 10.1016/j.cognition.2017.11.008. URL http://arxiv.org/ abs/1607.08723. arXiv: 1607.08723. · DOI
  31. Gabe Dupre. (What) Can Deep Learning Contribute to Theoretical Linguistics? Minds and Machines, September 2021. ISSN 0924-6495, 1572-8641. doi: 10. 1007/s11023-021-09571-w. URL https://link.springer.com/10.1007/ s11023-021-09571-w. · DOI
  32. Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic Probing: Be- havioral Explanation with Amnesic Counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 2021b.
  33. Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179-211, 1990. Pub- lisher: Wiley Online Library.
  34. Allyson Ettinger, Ahmed Elgohary, and Philip Resnik. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the 1st Work- shop on Evaluating Vector-Space Representations for NLP, pages 134-139, 2016. URL https://www.aclweb.org/anthology/W16-2524.
  35. Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal Analysis of Syntactic Agreement Mechanisms in Neu- ral Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1828-1843, Online, 2021. As- sociation for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.144. URL https://aclanthology.org/2021.acl-long.144. · DOI
  36. Janet Dean Fodor and Carrie Crowther. Understanding stimulus poverty arguments. The Linguistic Review, 18(1-2), January 2002. ISSN 0167-6318, 1613-3676. doi: 10. 1515/tlir.19.1-2.105. URL https://www.degruyter.com/document/doi/ 10.1515/tlir.19.1-2.105/html. · DOI
  37. Michael C. Frank, Mika Braginsky, Daniel Yurovsky, and Virginia A. Marchman. Word- bank: An open repository for developmental vocabulary data. Journal of Child Lan- guage, 44(3):677-694, May 2017. ISSN 0305-0009, 1469-7602. doi: 10.1017/ S0305000916000209. URL https://www.cambridge.org/core/product/ identifier/S0305000916000209/type/journal_article. · DOI
  38. Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven T. Piantadosi, and Evelina Fedorenko. The Natural Stories corpus: a reading- time corpus of English texts containing rare syntactic constructions. Language Re- sources and Evaluation, 55(1):63-77, March 2021. ISSN 1574-020X, 1574-0218. doi: 10.1007/s10579-020-09503-7. URL https://link.springer.com/10.1007/ s10579-020-09503-7. · DOI
  39. Jill Gilkerson, Jeffrey A Richards, Steven F Warren, Judith K Montgomery, Charles R Greenwood, D Kimbrough Oller, John HL Hansen, and Terrance D Paul. Mapping the early language environment using all-day recordings and automated analysis. American journal of speech-language pathology, 26(2):248-265, 2017. Publisher: ASHA.
  40. Lila Gleitman and Eric Wanner. Language Acquisition: The State of the Art. In Lila Gleit- man and Eric Wanner, editors, Language Acquisition: The State of the Art. Cambridge University Press, 1982.
  41. E Mark Gold. Language identification in the limit. Information and control, 10(5):447- 474, 1967. Publisher: Elsevier.
  42. Peter Gordon. Level-ordering in lexical development. Cognition, pages 73-93, 1985.
  43. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. Col- orless green recurrent networks dream hierarchically. Proceedings of the Society for Computation in Linguistics, 2(1):363-364, 2019.
  44. Rebecca L Gómez and LouAnn Gerken. Infant artificial language learning and language acquisition. Trends in cognitive sciences, 4(5):178-186, 2000. Publisher: Elsevier.
  45. John Hale. A Probabilistic Earley Parser as a Psycholinguistic Model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics, 2001. URL https://aclanthology.org/N01-1021.
  46. Chung-hye Han, Julien Musolino, and Jeffrey Lidz. Endogenous sources of variation in language acquisition. Proceedings of the National Academy of Sciences, 113(4):942- 947, January 2016. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1517094113. URL https://pnas.org/doi/full/10.1073/pnas.1517094113. · DOI
  47. Betty Hart and Todd R. Risley. American parenting of language-learning children: Persist- ing differences in family-child interactions observed in natural home environments. De- velopmental Psychology, 28(6):1096, 1992. URL https://psycnet.apa.org/ fulltext/1993-09151-001.pdf. Publisher: American Psychological Associa- tion.
  48. David Haussler. Probably approximately correct learning. In Proceedings of the eighth national conference on artificial intelligence, pages 1101-1108. AAAI Press, 1990.
  49. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding- enhanced BERT with Disentangled Attention. In International conference on learning representations, 2020.
  50. John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, 2019.
  51. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997. Publisher: MIT Press.
  52. Jeremy Howard and Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 328-339, 2018.
  53. Steve R. Howell, Damian Jankowicz, and Suzanna Becker. A model of grounded language acquisition: Sensorimotor features improve lexical and grammatical learning. Journal of Memory and Language, 53(2):258-276, August 2005. ISSN 0749596X. doi: 10.1016/ j.jml.2005.03.002. URL https://linkinghub.elsevier.com/retrieve/ pii/S0749596X05000495. · DOI
  54. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Ma- chine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 2391-2401, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1243. URL https://aclanthology.org/ D19-1243. · DOI
  55. Philip A. Huebner and Jon A. Willits. Using lexical context to discover the noun cat- egory: Younger children have it easier. In Kara D. Federmeier and Lili Sahakyan, editors, The context of cognition: Emerging perspectives, volume 75 of Psychology of learning and motivation, pages 279-331. Academic Press, 2021. doi: https://doi. org/10.1016/bs.plm.2021.08.002. URL https://www.sciencedirect.com/ science/article/pii/S0079742121000256. ISSN: 0079-7421. · DOI
  56. Taichi Iki and Akiko Aizawa. Effect of Visual Extensions on Natural Language Un- derstanding in Vision-and-Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2189-2196, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.167. URL https://aclanthology.org/2021. emnlp-main.167. · DOI
  57. Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nico- las Carion. MDETR-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
  58. Slava Katz. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE transactions on acoustics, speech, and signal processing, 35(3):400-401, 1987.
  59. Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. Measuring Compo- sitional Generalization: A Comprehensive Method on Realistic Data. arXiv:1912.09713 [cs, stat], June 2020. URL http://arxiv.org/abs/1912.09713. arXiv: 1912.09713. · arXiv
  60. Najoung Kim and Tal Linzen. COGS: A Compositional Generalization Challenge Based on Semantic Interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087-9105, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.731. URL https://aclanthology.org/2020.emnlp-main.731. · DOI · Paper page
  61. Simon Kirby. Function, selection, and innateness: The emergence of language universals. OUP Oxford, 1999.
  62. Brenden M Lake and Marco Baroni. Generalization without systematicity: On the compo- sitional skills of sequence-to-sequence recurrent networks. In International Conference on Machine Learning, pages 2879-2888, 2018.
  63. Brenden M Lake, Tal Linzen, and Marco Baroni. Human few-shot learning of compo- sitional instructions. In Proceedings of the 41st Annual Conference of the Cognitive Science Society, May 2019.
  64. Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Ben- jamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, and Emmanuel Dupoux. Generative Spoken Language Modeling from Raw Audio. arXiv:2102.01192 [cs], September 2021. URL http://arxiv.org/abs/2102. 01192. arXiv: 2102.01192. · arXiv
  65. Thomas K Landauer and Susan T Dutnais. A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. Psychological Review, 104(2):211-240, 1997.
  66. Jey Han Lau, Alexander Clark, and Shalom Lappin. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5): 1202-1241, 2017. Publisher: Wiley Online Library.
  67. Marvin Lavechin, Maureen de Seyssel, Marianne Métais, Florian Metze, Abdelrahman Mohamed, Hervé BREDIN, Emmanuel Dupoux, and Alejandrina Cristia. Early phonetic learning from ecological audio: domain-general versus domain-specific mechanisms. 2022. Publisher: PsyArXiv.
  68. Angeliki Lazaridou and Marco Baroni. Emergent Multi-Agent Communication in the Deep Learning Era. arXiv:2006.02419 [cs], July 2020. URL http://arxiv.org/abs/ 2006.02419. arXiv: 2006.02419. · arXiv
  69. Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. Combining Language and Vision with a Multimodal Skip-gram Model. In Proceedings of the 2015 Confer- ence of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, pages 153-163, Denver, Colorado, May 2015. Association for Computational Linguistics. doi: 10.3115/v1/N15-1016. URL https: //aclanthology.org/N15-1016. · DOI
  70. Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-Agent Coopera- tion and the Emergence of (Natural) Language. In International Conference on Learn- ing Representations, March 2017. URL http://arxiv.org/abs/1612.07182. arXiv: 1612.07182. · arXiv
  71. Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International conference on learning representations, 2018.
  72. Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent Communica- tion meets Natural Language: Synergies between Functional and Structural Language Learning. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pages 7663-7674, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.685. URL https://aclanthology. org/2020.acl-main.685. · DOI
  73. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553): 436, 2015.
  74. Julie Anne Legate and Charles D Yang. Empirical re-assessment of stimulus poverty argu- ments. The Linguistic Review, 18(1-2):151-162, 2002. Publisher: Walter de Gruyter.
  75. Fred Lerdahl, Ray S Jackendoff, and Ray Jackendoff. A Generative Theory of Tonal Music. MIT Press, 1983.
  76. Roger Levy. Expectation-based syntactic comprehension. Cognition, 106(3):1126-1177, March 2008. ISSN 00100277. doi: 10.1016/j.cognition.2007.05.006. URL https: //linkinghub.elsevier.com/retrieve/pii/S0010027707001436. · DOI
  77. Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent communication. Advances in neural information processing systems, 32, 2019.
  78. Jeffrey Lidz, Sandra Waxman, and Jennifer Freedman. What infants know about syn- tax but couldn't have learned: Experimental evidence for syntactic structure at 18 months. Cognition, 89(3):295-303, October 2003. ISSN 00100277. doi: 10. 1016/S0010-0277(03)00116-1. URL https://linkinghub.elsevier.com/ retrieve/pii/S0010027703001161. · DOI
  79. Tal Linzen. What can linguistics and deep learning contribute to each other? Response to Pater. Language, 95(1):e99-e108, 2019. Publisher: Linguistic Society of America.
  80. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535, 2016.
  81. Pierre Lison and Jorg Tiedemann. OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016), page 7, 2016.
  82. Charles Lovering, Rohan Jha, Tal Linzen, and Ellie Pavlick. Predicting Inductive Biases of Fine-tuned Models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=mNtmhaDkAr.
  83. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining Task- Agnostic Visiolinguistic Representations for Vision-and-Language Tasks. In Ad- vances in Neural Information Processing Systems, volume 32.
  84. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ c74d97b01eae257e44aa9d5bade97baf-Abstract.html.
  85. Brian MacWhinney. The CHILDES project: Tools for analyzing talk, Volume II: The database. Psychology Press, 2014.
  86. Jessica Mankewitz, Veronica Boyce, Brandon Waldon, Georgia Loukatou, Dhara Yu, Jesse Mu, Noah D Goodman, and Michael C Frank. Multi-party referential communication in complex strategic games. Publisher: PsyArXiv.
  87. Christopher D. Manning. Computational Linguistics and Deep Learning. Computational Linguistics, 41(4):701-707, December 2015. ISSN 0891-2017, 1530-9312. doi: 10. 1162/COLI a 00239. URL https://direct.mit.edu/coli/article/41/ 4/701-707/1512. · DOI
  88. Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48):30046-30054, December 2020. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1907367117. URL http: //www.pnas.org/lookup/doi/10.1073/pnas.1907367117. · DOI
  89. Gary F Marcus. Negative evidence in language acquisition. Cognition, 46(1):53-85, 1993. Publisher: Elsevier.
  90. R Thomas McCoy, Robert Frank, and Tal Linzen. Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks. In Proceedings of the 40th Annual Conference of the Cognitive Science Society., 2018.
  91. R. Thomas McCoy, Robert Frank, and Tal Linzen. Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks. Transac- tions of the Association for Computational Linguistics, 8:125-140, December 2020. ISSN 2307-387X. doi: 10.1162/tacl a 00304. URL https://direct.mit.edu/ tacl/article/43542. · DOI
  92. Richard Montague. The proper treatment of quantification in ordinary English. In Ap- proaches to natural language, pages 221-242. Springer, 1973.
  93. Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling. arXiv:2011.11588 [cs, eess], December 2020. URL http://arxiv.org/abs/ 2011.11588. arXiv: 2011.11588. · arXiv
  94. Ludovica Pannitto and Aurélie Herbelot. Recurrent babbling: evaluating the acquisi- tion of grammar from limited input data. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 165-176, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.conll-1.13. URL https://aclanthology.org/2020.conll-1. · DOI
  95. Isabel Papadimitriou, Ethan A. Chi, Richard Futrell, and Kyle Mahowald. Deep subject- hood: Higher-order grammatical features in multilingual BERT. In Proceedings of the 16th conference of the european chapter of the association for computational linguistics: Main volume, pages 2522-2532, Online, April 2021. Association for Computational Lin- guistics. doi: 10.18653/v1/2021.eacl-main.215. URL https://aclanthology. org/2021.eacl-main.215. · DOI
  96. Joe Pater. Generative linguistics and neural networks at 60: Foundation, friction, and fusion. Language, 95(1):e41-e74, 2019. Publisher: Linguistic Society of America.
  97. Andy Perfors, Joshua B Tenenbaum, and Terry Regier. The learnability of abstract syntactic principles. Cognition, 118(3):306-338, 2011. Publisher: Elsevier.
  98. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yux- iang Wu, and Alexander Miller. Language Models as Knowledge Bases? In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China, November 2019. Associ- ation for Computational Linguistics. doi: 10.18653/v1/D19-1250. URL https: //www.aclweb.org/anthology/D19-1250. · DOI · Paper page
  99. Jackson Petty and Robert Frank. Transformers Generalize Linearly. arXiv:2109.12036 [cs], September 2021. URL http://arxiv.org/abs/2109.12036. arXiv: 2109.12036. · arXiv
  100. Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. Information-Theoretic Probing for Linguistic Structure. In Pro- ceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics, pages 4609-4622, Online, July 2020. Association for Computational Linguis- tics. doi: 10.18653/v1/2020.acl-main.420. URL https://www.aclweb.org/ anthology/2020.acl-main.420. · DOI
  101. Laura Pérez-Mayos, Miguel Ballesteros, and Leo Wanner. How much pretraining data do language models need to learn syntax? September 2021. URL https://arxiv. org/abs/2109.03160v2.
  102. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and others. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021. · arXiv
  103. Ezer Rasin and Athulya Aravind. The nature of the semantic stimulus: the acquisition of every as a case study. Natural Language Semantics, 29(2):339-375, June 2021. ISSN 0925-854X, 1572-865X. doi: 10.1007/s11050-020-09168-6. URL https://link. springer.com/10.1007/s11050-020-09168-6. · DOI
  104. Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages. In Proceedings of NAACL-HLT, pages 3532-3542, 2019.
  105. Florencia Reali and Morten H Christiansen. Uncovering the richness of the stimulus: Struc- ture dependence and indirect statistical evidence. Cognitive Science, 29(6):1007-1028, 2005. Publisher: Wiley Online Library.
  106. Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B Cohen, and Simon Kirby. Compositional languages emerge in a neural iterated learning model. In International conference on learning representations, 2019.
  107. Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew M Dai, and Kyunghyun Cho. Capacity, bandwidth, and compositionality in emergent language learning. In Proceed- ings of the 19th international conference on autonomous agents and MultiAgent systems, pages 1125-1133, 2020.
  108. Eric Reuland. Grammar of binding in the languages of the world: Unity versus diversity. Cognition, 168:370-379, November 2017. ISSN 00100277. doi: 10.1016/j.cognition. 2016.01.020. URL https://linkinghub.elsevier.com/retrieve/pii/ S0010027716300208. · DOI
  109. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A Primer in BERTology: What we know about how BERT works. In Findings of EMNLP, 2020. URL https://www. aclweb.org/anthology/2020.tacl-1.
  110. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34(05), pages 8732-8740, 2020. doi: https://doi.org/10.1609/aaai.v34i05.6399. · DOI · Paper page
  111. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. Masked Lan- guage Model Scoring. In Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, pages 2699-2712, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.240. URL https: //www.aclweb.org/anthology/2020.acl-main.240. · DOI
  112. Carson T. Schütze. The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology. University of Chicago Press, 1996.
  113. John R. Searle. Speech acts: An essay in the philosophy of language, volume 626. Cam- bridge University Press, 1969.
  114. Xing Shi, Inkit Padhi, and Kevin Knight. Does string-based neural MT learn source syn- tax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526-1534, 2016.
  115. Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. arXiv preprint arXiv:2112.04482, 2021. · arXiv
  116. Melanie Soderstrom, Amanda Seidl, Deborah G Kemler Nelson, and Peter W Jusczyk. The prosodic bootstrapping of phrases: Evidence from prelinguistic infants. Journal of Memory and Language, 49(2):249-267, 2003. Publisher: Elsevier.
  117. Jessica Sullivan, Michelle Mei, Andrew Perfors, Erica Wojcik, and Michael C. Frank. SAYCam: A Large, Longitudinal Audiovisual Dataset Recorded From the Infant's Perspective. Open Mind, 5:20-29, May 2021. ISSN 2470-2986. doi: 10.1162/opmi a 00039. URL https://direct. mit.edu/opmi/article/doi/10.1162/opmi_a_00039/97495/ · DOI
  118. SAYCam-A-Large-Longitudinal-Audiovisual-Dataset. · DOI
  119. Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7464-7473, 2019.
  120. Hao Tan and Mohit Bansal. LXMERT: Learning Cross-Modality Encoder Represen- tations from Transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100-5111, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/ D19-1514. URL https://aclanthology.org/D19-1514. · DOI · Paper page
  121. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Na- joung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word represen- tations. In Proceedings of ICLR, 2019.
  122. Annmarie Van Dooren, Anouk Dieuleveut, Ailís Cournane, and Valentine Hacquard. Fig- uring out root and epistemic uses for modals: The role of the input. Journal of Semantics.
  123. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, \Lukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Processing Systems 30, pages 5998- 6008.
  124. Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/ 7181-attention-is-all-you-need.pdf.
  125. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language mod- els using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Had- sell, M. F. Balcan, and H. Lin, editors, Advances in neural information processing systems, volume 33, pages 12388-12401.
  126. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 92650b2e92217715fe312e6fa7b90d82-Paper.pdf.
  127. Elena Voita and Ivan Titov. Information-Theoretic Probing with Minimum Description Length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, November 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/ 2020.emnlp-main.14.pdf.
  128. Alex Wang and Kyunghyun Cho. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the workshop on methods for optimizing and evaluating neural language generation, pages 30-36, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-2304. URL https://aclanthology.org/W19-2304. · DOI
  129. Alex Warstadt and Samuel R. Bowman. Linguistic Analysis of Pretrained Sentence En- coders with Acceptability Judgments. arXiv preprint arXiv:1901.03438, 2019. · arXiv
  130. Alex Warstadt and Samuel R Bowman. Can neural networks acquire a structural bias from raw linguistic data? In Proceedings of the 42nd Annual Conference of the Cognitive Science Society., 2020.
  131. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018. · arXiv
  132. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. BLiMP: The Benchmark of Linguistic Minimal Pairs for En- glish. Transactions of the Association for Computational Linguistics, 8:377-392, 2020a. doi: 10.1162/tacl a 00321. URL https://doi.org/10.1162/tacl_a_00321. · DOI · Paper page
  133. Alex Warstadt, Yian Zhang, Haau-Sing Li, Haokun Liu, and Samuel R Bowman. Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, November 2020b. Association for Computational Linguistics.
  134. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. What do RNN Language Models Learn about Filler-Gap Dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211-221, 2018.
  135. Ethan Wilcox, Pranali Vani, and Roger Levy. A Targeted Assessment of Incremental Pro- cessing in Neural Language Models and Humans. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 939- 952, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.acl-long.76. URL https://aclanthology.org/2021.acl-long. · DOI
  136. Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspec- tive of generalization. Advances in neural information processing systems, 33:4697- 4708, 2020.
  137. Colin Wilson. Learning phonology with substantive bias: An experimental and computa- tional study of velar palatalization. Cognitive science, 30(5):945-982, 2006. Publisher: Wiley Online Library.
  138. Yuan Yang and Steven T. Piantadosi. One model for the learning of language. Proceedings of the National Academy of Sciences, 119(5):e2021865119, February 2022. ISSN 0027- 8424, 1091-6490. doi: 10.1073/pnas.2021865119. URL https://pnas.org/doi/ full/10.1073/pnas.2021865119. · DOI · Paper page
  139. Tian Yun, Chen Sun, and Ellie Pavlick. Does Vision-and-Language Pretraining Improve Lexical Grounding? In Proceedings of EMNLP, September 2021. arXiv: 2109.10246. · arXiv
  140. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adver- sarial dataset for grounded commonsense inference. In Proceedings of the 2018 confer- ence on empirical methods in natural language processing, pages 93-104, Brussels, Bel- gium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1009. URL https://aclanthology.org/D18-1009. · DOI
  141. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. When Do You Need Billions of Words of Pretraining Data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Con- ference on Natural Language Processing (Volume 1: Long Papers), pages 1112-1125, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.acl-long.90. URL https://aclanthology.org/2021.acl-long.90. · DOI · Paper page

Cited by

Sorted by citations

Authors
Warstadt, A., & Bowman, S. R.
ORCID
Warstadt, Alex
Venue
Algebraic Structures in Natural Language
Publisher
CRC Press
Published
2022-11-20
Pages
17-60
DOI
10.1201/9781003205388-2
OpenAlex
W4309419356
Cited by
111
References
18
Citations by year
2026: 8 · 2025: 29 · 2024: 29 · 2023: 45
FWCI
116.23
Type
book chapter
Language
en
Open access
Locations
  • CRC Press eBooks (publishedVersion)
  • arXiv (Cornell University) (publishedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1850208)
  • Johns Hopkins University
  • Eidgenössische Technische Hochschule Zürich
  • Samsung
Topics
  • Social Sciences › Psychology › Developmental and Educational Psychology › Language Development and Disorders
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
Keywords
language acquisition, artificial neural networks, learnability, poverty of the stimulus, neural language models, multimodal stimuli, multi-agent interaction, computational models of language learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review