The DOI matches its DataCite record.
Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
Open accessAbstract
We present the call for papers for the BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus. This shared task is intended for participants with an interest in small scale language modeling, human language acquisition, low-resource NLP, and cognitive modeling. In partnership with CoNLL and CMCL, we provide a platform for approaches to pretraining with a limited-size corpus sourced from data inspired by the input to children. The task has three tracks, two of which restrict the training data to pre-released datasets of 10M and 100M words and are dedicated to explorations of approaches such as architectural variations, self-supervised objectives, or curriculum learning. The final track only restricts the amount of text used, allowing innovation in the choice of the data, its domain, and even its modality (i.e., data from sources other than text is welcome). We will release a shared evaluation pipeline which scores models on a variety of benchmarks and tasks, including targeted syntactic evaluations and natural language understanding.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The agent achieves only 0-25% success rate on Kaggle challenges and BabyLM.
“However, the agent struggles with Kaggle challenges and BabyLM (Warstadt et al., 2023), with only a 0-25% success rate.”
From MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation (2023) · cites this paper as Warstadt et al., 2023
The Abstract identifies this paper as the call for the BabyLM Challenge, so it supports identifying BabyLM as a challenge. The Evaluation section describes a planned evaluation pipeline, not results for the cited agent; our copy reports neither its performance nor a 0–25% success rate on BabyLM or Kaggle challenges. (LLM, conf=0.86)
Experimental — accuracy not yet measuredMLAgentBench includes CLRS and BabyLM as datasets that are actively being researched and lack consensus on best approaches.
“We include CLRS (Velivckovi'c et al., 2022) and BabyLM (Warstadt et al., 2023) as two datasets that are actively being researched and do not yet have a consensus on the best approaches.”
From MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation (2023) · cites this paper as Warstadt et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 18
- Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014. The AMARA corpus: Building parallel language resources for the educa- tional domain. In Proceedings of the Ninth Inter- national Conference on Language Resources and Evaluation (LREC'14), Reykjavik, Iceland. Euro- pean Language Resources Association (ELRA).
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
- Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghe- mawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fe- dus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankara- narayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling language modeling with pathways. Computing Research Repository, arXiv:2204.02311. · DOI · Paper page
- Emmanuel Dupoux. 2018. Cognitive science in the era of artificial intelligence: A roadmap for reverse- engineering the infant language-learner. Cognition, 173:43-59. · DOI
- Martin Gerlach and Francesc Font-Clos. 2018. A standardized Project Gutenberg corpus for statis- tical analysis of natural language and quantita- tive linguistics. Computing Research Repository, arXiv:1812.08092. · DOI
- Jill Gilkerson, Jeffrey A Richards, Steven F Warren, Ju- dith K Montgomery, Charles R Greenwood, D Kim- brough Oller, John HL Hansen, and Terrance D Paul. 2017. Mapping the early language environ- ment using all-day recordings and automated anal- ysis. American journal of speech-language pathol- ogy, 26(2):248-265.
- Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203.15556. · arXiv
- Frank Keller. 2010. Cognitively plausible models of human language processing. In Proceedings of the ACL 2010 Conference Short Papers, pages 60-67, Uppsala, Sweden. Association for Computational Linguistics.
- Pierre Lison and Jörg Tiedemann. 2016. OpenSub- titles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 923-929, Por- torož, Slovenia. European Language Resources As- sociation (ELRA).
- Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pre- training approach. Computing Research Repository, arXiv:1907.11692. · DOI · Paper page
- Brian MacWhinney. 2000. The CHILDES project: The database, volume 2. Psychology Press.
- Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):1-67.
- Julian Salazar, Davis Liang, Toan Q. Nguyen, and Ka- trin Kirchhoff. 2020. Masked language model scor- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Compu- tational Linguistics. · DOI
- Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Tay- lor, Rachel Martin, Marie Meteer, and Carol Van Ess-Dykema. 2000. Dialogue act modeling for au- tomatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339-371.
- Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Eval- uating Neural Language Generation, pages 30-36, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI
- Alex Warstadt and Samuel R Bowman. 2022. What ar- tificial neural networks can tell us about human lan- guage acquisition. In Algebraic Structures in Natu- ral Language, pages 17-60. CRC Press.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language pro- cessing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Asso- ciation for Computational Linguistics.
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open pre-trained transformer language models. Comput- ing Research Repository, arXiv:2205.01068. · DOI · Paper page
Cited by
Sorted by citations
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74
“In this section, we describe each competition track; for specific details about wording, see the original Call for Papers (Warstadt et al., 2023).”
Self-citationBigger is not always better: The importance of human-scale language modeling for psycholinguistics
Wilcox, E. G., Hu, M. Y., Mueller, A., Warstadt, A., Choshen, L., Zhuang, C., Williams, A., Cotterell, R., & Linzen, T. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104650 · Cited by 9
Self-citationMLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
Huang, Q., Vora, J., Liang, P., & Leskovec, J. (2023) · arXiv · doi:10.48550/arxiv.2310.03302 · Cited by 9
Metadata
- Authors
- Warstadt, A., Choshen, L., Mueller, A., Williams, A., Wilcox, E., & Zhuang, C.
- Affiliations
- ETH Zurich (CH)
- IBM (United States) (US)
- Johns Hopkins University (US)
- Massachusetts Institute of Technology (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2301.11796
- OpenAlex
- W4318621130
- Cited by
- 14
- References
- 0
- Citations by year
- 2025: 3 · 2024: 3 · 2023: 8
- Type
- paratext
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Psychology › Developmental and Educational Psychology › Language Development and Disorders
- Keywords
- language acquisition, low-resource NLP, cognitive modeling, self-supervised learning, curriculum learning, shared task, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
