Too Much Information: Keeping Training Simple for BabyLMs

RoBERTa: A Robustly Optimized BERT Pretraining Approach

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

Kudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · see citing works

Longformer: The Long-Document Transformer

Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · see citing works

Big Bird: Transformers for Longer Sequences

Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · see citing works

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

Kudo, T. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1007 · Cited by 1,021 · see citing works

DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding SharingPreprint

He, P., Gao, J., & Chen, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09543 · Cited by 862 · see citing works

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1804.07461 · Cited by 562 · see citing works

Curriculum Learning: A Survey

Soviany, P., Ionescu, R. T., Rota, P., & Sebe, N. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01611-x · Cited by 486 · see citing works

ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models

Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., & Raffel, C. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00461 · Cited by 330 · see citing works

Competence-based Curriculum Learning for Neural Machine Translation

Platanios, E. A., Stretcu, O., Neubig, G., Póczos, B., & Mitchell, T. M. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1119 · Cited by 316 · see citing works

Should You Mask 15% in Masked Language Modeling?

Wettig, A., Gao, T., Zhong, Z., & Chen, D. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.217 · Cited by 161 · see citing works

CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters

Boukkouri, H. E., Ferret, O., Lavergne, T., Noji, H., Zweigenbaum, P., & Tsujii, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.609 · Cited by 152 · see citing works

Curriculum Learning and Minibatch Bucketing in Neural Machine Translation

Kocmi, T., & Bojar, O. (2017) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-049-6_050 · Cited by 121 · see citing works

Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)

Warstadt, A., Zhang, Y., Li, H.-S., Liu, H., & Bowman, S. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.16 · Cited by 106 · see citing works

An Empirical Exploration of Curriculum Learning for Neural Machine TranslationPreprint

Zhang, X., Kumar, G., Khayrallah, H., Murray, K., Gwinnup, J., Martindale, M. J., McNamee, P., Duh, K., & Carpuat, M. J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00739 · Cited by 93 · see citing works

Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works

TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

Eldan, R., & Li, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.07759 · Cited by 52 · see citing works

Visualizing and Understanding Curriculum Learning for Long Short-Term Memory NetworksPreprint

Cirik, V., Hovy, E. H., & Morency, L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06204 · Cited by 30 · see citing works

Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning

Chang, E., Yeh, H.-S., & Demberg, V. (2021) · doi:10.18653/v1/2021.eacl-main.61 · Cited by 30 · see citing works

Baby Steps: How “Less is More” in unsupervised dependency parsing

Spitkovsky, V. I., Alshawi, H., & Jurafsky, D. S. (2009) · Cited by 29 · see citing works

BLiMP: the benchmark of linguistic minimal pairs for English

Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00321 · Cited by 28 · see citing works

Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation

Edman, L., Sarti, G., Toral, A., van Noord, G., & Bisazza, A. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00651 · Cited by 10 · see citing works

Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora

Ingólfsdóttir, S. L., Ragnarsson, P. O., Jónsson, H. P., Símonarson, H. B., Thorsteinsson, V., & Snæbjarnarson, V. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.402 · Cited by 4 · see citing works

The Importance of Context in Very Low Resource Language Modeling

Edman, L., Toral, A., & van Noord, G. (2022) · University of Groningen research database (University of Groningen / Centre for Information Technology) · doi:10.48550/arxiv.2205.04810 · Cited by 1 · see citing works

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805

Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works

Findings of the Third BabyLM Challenge: Accelerating Language Modeling Research with Cognitively Plausible Data

Charpentier, L., Choshen, L., Cotterell, R., Gul, M. O., Hu, M. Y., Liu, J., Jumelet, J., Linzen, T., Mueller, A., Ross, C., et al. (2025) · Proceedings of the First BabyLM Workshop · doi:10.18653/v1/2025.babylm-main.28 · Cited by 28 · see citing works

BabyLM Challenge: Curriculum learning based on sentence complexity approximating language acquisition

Oba, M., Haga, A., Fukatsu, A., & Oseki, Y. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.25

ChapGTP, ILLC’s Attempt at Raising a BabyLM: Improving Data Efficiency by Automatic Task Formation

Jumelet, J., Hanna, M., Kloots, M. D. H., Langedijk, A., Pouw, C., & van der Wal, O. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.6 · Cited by 1 · see citing works

Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works

Baby’s CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models

Zhang, Z., Yang, H., Ma, B., Rügamer, D., & Nie, E. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.13 · Cited by 2 · see citing works