The DOI matches its DataCite record; its reference list is not available.
Universal linguistic inductive biases via meta-learning
Open accessAbstract
How do learners acquire languages from the limited data available to them? This process must involve some inductive biases - factors that affect how a learner generalizes - but it is unclear which inductive biases can explain observed patterns in language acquisition. To facilitate computational modeling aimed at addressing this question, we introduce a framework for giving particular linguistic inductive biases to a neural network model; such a model can then be used to empirically explore the effects of those inductive biases. This framework disentangles universal inductive biases, which are encoded in the initial values of a neural network's parameters, from non-universal factors, which the neural network must learn from data in a given language. The initial state that encodes the inductive biases is found with meta-learning, a technique through which a model discovers how to acquire new languages more easily via exposure to many possible languages. By controlling the properties of the languages that are used during meta-learning, we can control the inductive biases that meta-learning imparts. We demonstrate this framework with a case study based on syllable structure. First, we specify the inductive biases that we intend to give our model, and then we translate those inductive biases into a space of languages from which a model can meta-learn. Finally, using existing analysis techniques, we verify that our approach has imparted the linguistic inductive biases that it was intended to impart.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
McCoy et al. (2020) showed that their approach identifies inductive biases useful for acquiring a language's syllable structure.
“They showed that this approach identifies inductive biases (e.g. a bias for treating certain phonemes as vowels) that are useful for acquiring a language's syllable structure.”
From Meta-learned models of cognition (2024) · cites this paper as McCoy et al. (2020)
McCoy et al. (2020) meta-learned the initial weights of a neural network such that it can adapt quickly to new languages using standard gradient-based learning.
“In particular, they meta-learned the initial weights of a neural network such that the network can adapt itself quickly to new languages using standard gradient-based learning.”
From Meta-learned models of cognition (2024) · cites this paper as McCoy, Grant, Smolensky, Griffiths, and Linzen (2020)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Cited by 13
Research works printed but not linked: 17
- Finn, C., Abbeel, P., & Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In Proc. ICML.
- Grant, E., Peterson, J. C., & Griffiths, T. L. (2019). Learning deep taxonomic priors for concept learning from few positive examples. In Proceedings of the 41st Annual Conference of the Cognitive Science Society.
- Greenberg, J. H. (1963). Universals of language . Cambridge, MA: MIT Press.
- Gu, J., Wang, Y., Chen, Y., Li, V. O. K., & Cho, K. (2018). Meta-learning for low-resource neural machine translation. In Proc. EMNLP.
- Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., & Baroni, M. (2018). Colorless green recurrent networks dream hierarchically. In Proc. NAACL.
- Hochreiter, S., Younger, A., & Conwell, P. (2001). Learning to learn using gradient descent. Proc. ICANN , 87-94.
- Kann, K., Bowman, S. R., & Cho, K. (2020). Learning to learn morphological inflection for resource-poor lan- guages. Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence .
- Kingma, D., & Ba, J. (2015). Adam: A method for stochastic optimization. In Proc. ICLR.
- Lake, B. M. (2019). Compositional generalization through meta sequence-to-sequence learning. In Proc. NeurIPS.
- McCoy, R. T., Pavlick, E., & Linzen, T. (2019). Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proc. ACL.
- Mitchell, T. M. (1997). Machine learning. 1997. Burr Ridge, IL: McGraw Hill , 45 (37).
- Ponti, E. M., Vuli´ c, I., Cotterell, R., Reichart, R., & Korhonen, A. (2019). Towards zero-shot language modeling. In Proc. EMNLP-IJCNLP.
- Prince, A., & Smolensky, P. (1993/2004). Optimality theory: Constraint interaction in generative grammar . Wiley.
- Ross, J. R. (1967). Constraints on variables in syntax . Unpublished doctoral dissertation, MIT.
- Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. In Proc. NeurIPS.
- Swadesh, M. (1950). Salish internal relationships. International Journal of American Linguistics , 16 (4).
- van Schijndel, M., Mueller, A., & Linzen, T. (2019). Quantity doesn't buy quality syntax with neural language models. In Proc. EMNLP-IJCNLP.
Metadata
- Authors
- McCoy, R. T., Grant, E., Smolensky, P., Griffiths, T. L., & Linzen, T.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.16324
- OpenAlex
- W3134976210
- Cited by
- 13
- References
- 0
- Citations by year
- 2024: 2 · 2023: 1 · 2022: 3 · 2021: 2 · 2020: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- arXiv (Cornell University)
- Cognitive Science
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Language Development and Disorders
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Social Sciences › Arts and Humanities › Language and Linguistics › Syntax, Semantics, Linguistic Variation
- Keywords
- meta-learning, language acquisition, neural network initialization, syllable structure, computational modeling of language learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
