Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways
Open accessAbstract
We present Lil-Bevo, our submission to the BabyLM Challenge.We pretrained our masked language models with three ingredients: an initial pretraining with music data, training on shorter sequences before training on longer ones, and masking specific tokens to target some of the BLiMP subtasks.Overall, our baseline models performed above chance, but far below the performance levels of larger LLMs trained on more data.We found that training on short sequences performed better than training on longer sequences.Pretraining on music may help performance marginally, but, if so, the effect seems small.Our targeted Masked Language Modeling augmentation did not seem to improve model performance in general, but did seem to help on some of the specific BLiMP tasks that we were targeting (e.g., Negative Polarity Items).Training performant LLMs on small amounts of data is a difficult but potentially informative task.While some of our techniques showed some promise, more work is needed to explore whether they can improve performance more than the modest gains here.Our code and models are available online. 1 .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Govindarajan et al. (2023) achieved improvements on specific BLiMP subtasks by modifying the masking procedure to preferentially mask specific words thought to be relevant to a particular phenomenon tested by BLiMP.
“Govindarajan et al. (2023) achieved improvements on specific BLiMP subtasks by modifying the masking procedure to preferentially mask specific words thought to be rel-evant to a particular phenomenon tested by BLiMP.”
From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Govindarajan et al. (2023)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1× · see citing works
“Thus, it is possible that models memorize more than humans do and, relative to humans, over-rely on statistical heuristics and memorized chunks of language (Bender et al., 2021).”
Kudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · see citing works
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844 · see citing works
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · see citing works
Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
Linzen, T., Emmanuel Dupoux, A., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 776 · see citing works
The social bases of language acquisition
Tomasello, M. (1992) · Social Development · doi:10.1111/j.1467-9507.1992.tb00135.x · Cited by 574 · see citing works
Negative evidence in language acquisition
Marcus, G. F. (1993) · Cognition · doi:10.1016/0010-0277(93)90022-n · Cited by 553 · see citing works
The scientist in the crib : minds, brains, and how children learn
Gopnik, A., Meltzoff, A. N., & Kuhl, P. K. (1999) · Medical Entomology and Zoology · Cited by 540 · see citing works
Recasts as Feedback to Language Learners
Nicholas, H., Lightbown, P. M., & Spada, N. (2001) · Language Learning · doi:10.1111/0023-8333.00172 · Cited by 464 · see citing works
Salazar, J., Liang, D., Nguyen, T. Q., & Kirchhoff, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.240 · Cited by 414 · see citing works
Quantifying Sources of Variability in Infancy Research Using the Infant-Directed-Speech Preference
Frank, M. C., Alcock, K., Arias‐Trejo, N., Aschersleben, G., Baldwin, D. A., Barbu, S., Bergelson, E., Bergmann, C., Black, A. K., Blything, R. P., et al. (2020) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245919900809 · Cited by 343 · see citing works
Enabling Factorized Piano Music Modeling and Generation with the MAESTRO DatasetPreprint
Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.-Z. A., Dieleman, S., Elsen, E., Engel, J., & Douglas Eck, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12247 · Cited by 301 · see citing works
Language learning, socioeconomic status, and child‐directed speech
Schwab, J. F., & Lew‐Williams, C. (2016) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1393 · Cited by 295 · see citing works
Syntactic Structure from Deep Learning
Linzen, T., & Baroni, M. (2021) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-032020-051035 · Cited by 202 · see citing works
Lerdahl, F. (1996) · Music Perception An Interdisciplinary Journal · doi:10.2307/40286174 · Cited by 130 · see citing works
What Artificial Neural Networks Can Tell Us about Human Language Acquisition
Warstadt, A., & Bowman, S. R. (2022) · Algebraic Structures in Natural Language · doi:10.1201/9781003205388-2 · Cited by 111 · see citing works
Warstadt, A., Zhang, Y., Li, H.-S., Liu, H., & Bowman, S. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.16 · Cited by 106 · see citing works
When Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85 · see citing works
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · cited 3× · see citing works
“In this paper, we describe our experiments with Lil-Bevo, a small language model trained on human-scale data for the BabyLM competition (Warstadt et al., 2023).”
InfluentialExplanatory models in neuroscience, Part 1: Taking mechanistic abstraction seriously
Cao, R., & Yamins, D. L. K. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2024.101244 · Cited by 69 · see citing works
Using Computational Models to Test Syntactic Learnability
Wilcox, E., Futrell, R., & Lévy, R. (2022) · Linguistic Inquiry · doi:10.1162/ling_a_00491 · Cited by 66 · see citing works
Quantity doesn’t buy quality syntax with neural language models
van Schijndel, M., Mueller, A., & Linzen, T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1592 · Cited by 65 · see citing works
Train No Evil: Selective Masking for Task-Guided Pre-Training
Gu, Y., Zhang, Z., Wang, X., Liu, Z., & Sun, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.566 · Cited by 55 · see citing works
Shortformer: Better language modeling using shorter inputs
Press, O., Smith, N. A., & Lewis, M. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.427 · Cited by 55 · see citing works
Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
Papadimitriou, I., & Jurafsky, D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.554 · Cited by 43 · see citing works
On the proper role of linguistically-oriented deep net analysis in linguistic theorizingPreprint
Baroni, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.08694 · Cited by 30 · see citing works
BLiMP: the benchmark of linguistic minimal pairs for English
Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00321 · Cited by 28 · see citing works
Do RNNs learn human-like abstract word order preferences?Preprint
Futrell, R., & Lévy, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01866 · Cited by 22 · see citing works
Adaptation in Child Directed Speech: Evidence from Corpora
Kunert, R., Fernández, R., & Zuidema, W. H. (2011) · UvA-DARE (University of Amsterdam) · Cited by 20 · see citing works
Bajaj, P., Xiong, C., Ke, G., Liu, X., He, D., Tiwary, S., Liu, T., Bennett, P. N., Song, X., & Gao, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.06644 · Cited by 20 · see citing works
Large language models as models of human cognitionPreprint
Frank, M. C. (2023) · doi:10.31234/osf.io/wxt69 · Cited by 17 · see citing works
Mueller, A., & Linzen, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.629 · Cited by 14 · see citing works
InforMask: Unsupervised Informative Masking for Language Model Pretraining
Sadeq, N., Xu, C., & McAuley, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.395 · Cited by 13 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Steven T Piantadosi. 2023. Modern language models refute chomsky's approach to language. Lingbuzz Preprint, lingbuzz/007180.
- Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A Stick- ier Benchmark for General-Purpose Language Un- derstanding Systems. In Proceedings of the 33rd International Conference on Neural Information Pro- cessing Systems. · DOI
- Adriana Weisleder and Anne Fernald. 2013. Talking to children matters: Early language experience strength- ens processing and builds vocabulary. Psychological science, 24(11):2143-2152.
Cited by
Sorted by citations
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works
“One submission used music (Govindarajan et al., 2023), another used vision and language data (Amariucai and Warstadt, 2023), a third explored text-and-audio (Wolf et al., 2023), and a fourth incorporated text-and-image data and lexical sensorimotor data as part of the embedding process using…”
Influential
- Authors
- Govindarajan, V. S., Rodriguez, J. D., Bostrom, K., & Mahowald, K.
- Affiliations
- The University of Texas at Austin (US)
- ORCID
- Mahowald, Kyle
- Venue
- Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning
- Published
- 2023
- Pages
- 280-288
- OpenAlex
- W4389523896
- Cited by
- 1
- References
- 33
- Citations by year
- 2023: 1
- FWCI
- 0.18
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- dx.doi.org
- Licence
- cc-by
- Locations
- BabyLM Challenge (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- masked language modeling, blimp, negative polarity items, sequence length
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
