CLIMB – Curriculum Learning for Infant-inspired Model Building

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works

Language Models are Few-Shot Learners

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works

Decoupled Weight Decay Regularization

Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works

Transformers: State-of-the-Art Natural Language Processing

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works

fairseq: A Fast, Extensible Toolkit for Sequence Modeling

Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · see citing works

A Survey of Large Language Models

Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · see citing works

Learning and development in neural networks: the importance of starting small

Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · see citing works

Finetuned Language Models Are Zero-Shot Learners

Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works

Training Compute-Optimal Large Language Models

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · see citing works

The Goldilocks Effect: Human Infants Allocate Attention to Visual Sequences That Are Neither Too Simple Nor Too Complex

Kidd, C., Piantadosi, S. T., & Aslin, R. Ν. (2012) · PLoS ONE · doi:10.1371/journal.pone.0036399 · Cited by 788 · see citing works

Lexical output as related to children's vocabulary acquisition: Effects of sophisticated exposure and support for meaning.

Weizman, Z. O., & Snow, C. E. (2001) · Developmental Psychology · doi:10.1037/0012-1649.37.2.265 · Cited by 743 · see citing works

A Universal Part-of-Speech Tagset

Petrov, S., Das, D., & McDonald, R. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3pjyez8kmkhz · Cited by 742 · see citing works

A new algorithm for data compression

Gage, P. (1994) · C/C++ users journal · Cited by 655 · see citing works

The Pile: An 800GB Dataset of Diverse Text for Language Modeling

Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works

Curriculum Learning: A Survey

Soviany, P., Ionescu, R. T., Rota, P., & Sebe, N. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01611-x · Cited by 486 · see citing works

Mapping the Early Language Environment Using All-Day Recordings and Automated Analysis

Gilkerson, J., Richards, J. A., Warren, S. F., Montgomery, J. K., Greenwood, C. R., Kimbrough Oller, D., Hansen, J. H. L., & Paul, T. D. (2017) · American Journal of Speech-Language Pathology · doi:10.1044/2016_ajslp-15-0169 · Cited by 472 · see citing works

Competence-based Curriculum Learning for Neural Machine Translation

Platanios, E. A., Stretcu, O., Neubig, G., Póczos, B., & Mitchell, T. M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1119 · Cited by 316 · see citing works

Evidence of a predictive coding hierarchy in the human brain listening to speech

Caucheteux, C., Gramfort, A., & King, J.-R. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01516-2 · Cited by 308 · see citing works

Early Word Comprehension in Infants: Replication and Extension

Bergelson, E., & Swingley, D. (2014) · Language Learning and Development · doi:10.1080/15475441.2014.979387 · Cited by 186 · see citing works

Cramming: Training a Language Model on a Single GPU in One DayPreprint

Geiping, J., & Goldstein, T. (2022) · International Conference on Machine Learning · doi:10.48550/arxiv.2212.14034 · Cited by 125 · see citing works

Curriculum Learning and Minibatch Bucketing in Neural Machine Translation

Kocmi, T., & Bojar, O. (2017) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-049-6_050 · Cited by 121 · see citing works

Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)

Warstadt, A., Zhang, Y., Li, H.-S., Liu, H., & Bowman, S. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.16 · Cited by 106 · see citing works

When Do You Need Billions of Words of Pretraining Data?

Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85 · see citing works

How to Train BERT with an Academic Budget

Izsak, P., Berchansky, M., & Levy, O. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.831 · Cited by 83 · see citing works

Easy Questions First? A Case Study on Curriculum Learning for Question Answering

Sachan, M., & Xing, E. P. (2016) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p16-1043 · Cited by 76 · see citing works

Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works

BabyBERTa: Learning More Grammar With Small-Scale Child-Directed Language

Huebner, P. A., Sulem, E., Fisher, C. L., & Roth, D. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.49 · Cited by 71 · see citing works

Curriculum Learning for Natural Answer Generation

Liu, C., He, S., Liu, K., & Zhao, J. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/587 · Cited by 69 · see citing works

The Structural Sources of Verb Meanings

Gleitman, L. R. (2020) · doi:10.1093/oso/9780199828098.003.0009 · Cited by 57 · see citing works

Unsupervised Part-Of-Speech Tagging with Anchor Hidden Markov Models

Stratos, K., Collins, M., & Hsu, D. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00096 · Cited by 48 · see citing works

BLiMP: the benchmark of linguistic minimal pairs for English

Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00321 · Cited by 28 · see citing works

LERT: A Linguistically-motivated Pre-trained Language ModelPreprint

Cui, Y., Che, W., Wang, S., & Liu, T. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.05344 · Cited by 26 · see citing works

First Language Acquisition

Guasti, M. T. (2016) · doi:10.1093/oxfordhb/9780199573776.013.12 · Cited by 18 · see citing works

Dynamic Data Selection for Curriculum Learning via Ability Estimation

Lalor, J. P., & Yu, H. Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.48 · Cited by 17 · see citing works

Curriculum learning for language modeling

Campos, D. (2021) · arXiv · doi:10.48550/arxiv.2108.02170 · Cited by 15 · see citing works

Trained on 100 million words and still in shape: BERT meets British National Corpus

Samuel, D., Kutuzov, A., Øvrelid, L., & Velldal, E. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.146 · Cited by 14 · see citing works

Better Language Model with Hypernym Class Prediction

Bai, H., Wang, T., Sordoni, A., & Shi, P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.96 · Cited by 10 · see citing works

When data permutations are pathological: the case of neural natural language inference

Schluter, N., & Varab, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1534 · Cited by 9 · see citing works

Computational modeling of human language acquisition

Alishahi, A. (2009) · doi:10.3115/1667899.1667903 · Cited by 6 · see citing works

Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works

Bigger is not always better: The importance of human-scale language modeling for psycholinguistics

Wilcox, E. G., Hu, M. Y., Mueller, A., Warstadt, A., Choshen, L., Zhuang, C., Williams, A., Cotterell, R., & Linzen, T. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104650 · Cited by 9 · see citing works

BabyLM Challenge: Curriculum learning based on sentence complexity approximating language acquisition

Oba, M., Haga, A., Fukatsu, A., & Oseki, Y. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.25

Findings of the Third BabyLM Challenge: Accelerating Language Modeling Research with Cognitively Plausible Data

Charpentier, L., Choshen, L., Cotterell, R., Gul, M. O., Hu, M. Y., Liu, J., Jumelet, J., Linzen, T., Mueller, A., Ross, C., et al. (2025) · Proceedings of the First BabyLM Workshop · doi:10.18653/v1/2025.babylm-main.28 · Cited by 28 · see citing works

Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations

Ma, Z., Wang, Z., & Chai, J. Y. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.46 · Cited by 4 · see citing works

Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works

The BabyLM Challenge: In search of more efficient learning algorithms, researchers look to infants

Fyshe, A. (2025) · The Transmitter · doi:10.53053/vcow7276