Trained on 100 million words and still in shape: BERT meets British National Corpus
Open accessAbstract
While modern masked language models (LMs) are trained on ever larger corpora, we here explore the effects of down-scaling training to a modestly-sized but representative, wellbalanced, and publicly available English text source -the British National Corpus.We show that pre-training on this carefully curated corpus can reach better performance than the original BERT model.We argue that this type of corpora has great potential as a language modeling benchmark.To showcase this potential, we present fair, reproducible and data-efficient comparative studies of LMs, in which we evaluate several training objectives and model architectures and replicate previous empirical results in a systematic way.We propose an optimized LM architecture called LTG-BERT.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Winning submissions using the LTG-BERT architecture outperformed models trained on trillions of words.
“The winning submissions using the LTG-BERT architecture (Samuel et al., 2023) outperformed models trained on trillions of words.”
From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Samuel et al., 2023
The Introduction says the LTG-BERT model outperformed the original BERT, which was trained on a corpus 30 times larger. It does not report winning submissions or a comparison with models trained on trillions of words; Section 7 says the authors deliberately did not compare against more recent models. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe best-performing models overall use the LTG-BERT architecture.
“The best-performing models overall (Charpentier and Samuel, 2023) use the LTG-BERT architecture (Samuel et al., 2023)”
From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Samuel et al., 2023
ELC-BERT and the runner-up Boot-BERT used as their starting point the LTG-BERT architecture from Samuel et al. (2023).
“This model, as well as the runner-up submission Boot-BERT (Samuel, 2023), used as their starting point the LTG-BERT architecture from Samuel et al. (2023).”
From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Samuel et al. (2023)
ELC-BERT and Boot-BERT are both based on LTG-BERT.
“ELC-BERT and Boot-BERT are both based on LTG-BERT (Samuel et al., 2023)”
From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Samuel et al., 2023
The Conclusion (page unavailable) says the authors release their model under the name LTG-BERT. Searches for “ELC-BERT Boot-BERT” and for models based on LTG-BERT found no passage identifying either model or stating that it is based on LTG-BERT. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · see citing works
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · see citing works
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · see citing works
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L. (2017) · Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) · doi:10.18653/v1/s17-2001 · Cited by 1,562 · see citing works
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552 · see citing works
DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION
He, P., Liu, X., Gao, J., & Chen, W. (2021) · International Conference on Learning Representations · Cited by 1,116 · see citing works
Automatically Constructing a Corpus of Sentential Paraphrases.
Dolan, W. B., & Brockett, C. (2005) · Cited by 1,101 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 984 · see citing works
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929 · see citing works
Continuous speech recognition by statistical methods
Jelinek, F. (1976) · Proceedings of the IEEE · doi:10.1109/proc.1976.10159 · Cited by 922 · see citing works
CamemBERT: a Tasty French Language Model
Martin, L., Müller, B., Suárez, P. O., Dupont, Y., Romary, L., de la Clergerie, É. V., Seddah, D., & Sagot, B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.645 · Cited by 911 · see citing works
The Third PASCAL Recognizing Textual Entailment Challenge
Giampiccolo, D., Magnini, B., Dagan, I., & Dolan, B. (2007) · ACL-PASCAL@ACL · doi:10.3115/1654536.1654538 · Cited by 801 · see citing works
One billion word benchmark for measuring progress in statistical language modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-564 · Cited by 627 · see citing works
Large Batch Optimization for Deep Learning: Training BERT in 76 minutesPreprint
You, Y., Li, J., Reddi, S. J., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J. W., Keutzer, K., & Hsieh, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.00962 · Cited by 597 · see citing works
Tenney, I., Xia, P., Chen, B., Wang, A., Poliak, A., McCoy, R. T., Kim, N., Durme, B. V., Bowman, S. R., Das, D., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.06316 · Cited by 480 · see citing works
The CommitmentBank: Investigating projection in naturally occurring discourse
de Marneffe, M.-C., Simons, M., & Tonhauser, J. (2019) · Digital Access to Libraries · doi:10.18148/sub/2019.v23i2.601 · Cited by 465 · see citing works
Salazar, J., Liang, D., Nguyen, T. Q., & Kirchhoff, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.240 · Cited by 414 · see citing works
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 391 · see citing works
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., & Roth, D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1023 · Cited by 374 · see citing works
Jiang, H., He, P., Chen, W., Liu, X., Gao, J., & Zhao, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.197 · Cited by 372 · see citing works
Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning
Roemmele, M., Bejan, C. A., & Gordon, A. S. (2011) · Cited by 336 · see citing works
Probing Classifiers: Promises, Shortcomings, and Advances
Belinkov, Y. (2021) · Computational Linguistics · doi:10.1162/coli_a_00422 · Cited by 311 · see citing works
GLU Variants Improve Transformer
Shazeer, N. (2020) · arXiv · doi:10.48550/arxiv.2002.05202 · Cited by 303 · see citing works
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Pilehvar, M. T., & Camacho-Collados, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1128 · Cited by 296 · see citing works
A Gold Standard Dependency Corpus for English
Silveira, N., Dozat, T., de Marneffe, M.-C., Bowman, S. R., Connor, M. K., Bauer, J., & Manning, C. D. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4esbhdz7y5u5 · Cited by 237 · see citing works
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · see citing works
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading ComprehensionPreprint
Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., & Durme, B. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12885 · Cited by 201 · see citing works
Understanding the Difficulty of Training Transformers
Liu, L., Liu, X., Gao, J., Chen, W., & Han, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.463 · Cited by 190 · see citing works
How Can We Accelerate Progress Towards Human-like Linguistic Generalization?
Linzen, T. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.465 · Cited by 148 · see citing works
Transformers without Tears: Improving the Normalization of Self-AttentionPreprint
Nguyen, T. Q., & Salazar, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05895 · Cited by 124 · see citing works
Construction of the Turkish National Corpus (TNC)
Aksan, Y., Aksan, M., Koltuksuz, A. H., Sezer, T., Mersinli, Ü., Demirhan, U. U., Yılmazer, H., Atasoy, G., Seda Ozmersin university, İpek, Y., et al. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/39gnocw3eigg · Cited by 122 · see citing works
Finding the Optimal Vocabulary Size for Neural Machine Translation
Gowda, T., & May, J. (2020) · Findings · doi:10.18653/v1/2020.findings-emnlp.352 · Cited by 100 · see citing works
Weischedel, R., Palmer, M., Marcus, M. P., Hovy, E. H., Pradhan, S., Ramshaw, L., Xue, N., Taylor, A., Kaufman, J. T., Franchini, M., et al. (2013) · Americanae (AECID Library) · doi:10.35111/xmhb-2b84 · Cited by 92 · see citing works
Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristics
Bhargava, P., Валентинович, Д. А., & Rogers, A. (2021) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2021.insights-1.18 · Cited by 89 · see citing works
When Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85 · see citing works
A Surprisingly Robust Trick for the Winograd Schema Challenge
Kocijan, V., Cretu, A.-M., Camburu, O.-M., Yordanov, Y., & Lukasiewicz, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1478 · Cited by 81 · see citing works
Where did we Go Wrong? A Retrospective Look at the British National Corpus
Burnard, L. (2002) · BRILL eBooks · doi:10.1163/9789004334236_007 · Cited by 71 · see citing works
On the importance of pre-training data volume for compact language models
Micheli, V., d’Hoffschmidt, M., & Fleuret, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.632 · Cited by 38 · see citing works
BLiMP: the benchmark of linguistic minimal pairs for English
Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00321 · Cited by 28 · see citing works
NormFormer: Improved Transformer Pretraining with Extra Normalization
Shleifer, S., Weston, J., & Ott, M. (2021) · arXiv · doi:10.48550/arxiv.2110.09456 · Cited by 24 · see citing works
Bajaj, P., Xiong, C., Ke, G., Liu, X., He, D., Tiwary, S., Liu, T., Bennett, P. N., Song, X., & Gao, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.06644 · Cited by 20 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. 2006. The second pascal recognising textual entailment challenge. Proceed- ings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. 16
- Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency, pages 610-623.
- Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth pascal recognizing textual entailment challenge. In In Proc Text Analysis Conference (TAC'09. 16
- Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Eval- uating Neural Language Generation, pages 30-36, Minneapolis, Minnesota. Association for Computa- tional Linguistics. 14 · DOI
Cited by
Sorted by citations
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works
KyrgyzNLP: Challenges, Progress, and Future
Alekseev, A., & Turatali, T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88036-0_1 · Cited by 9 · see citing works
Not all layers are equally as important: Every Layer Counts BERT
Georges Gabriel Charpentier, L., & Samuel, D. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.20 · Cited by 9 · see citing works
Self-citationInvestigating Critical Period Effects in Language Acquisition through Neural Language Models
Constantinescu, I., Pimentel, T., Cotterell, R., & Warstadt, A. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00725 · Cited by 7 · see citing works
Samuel, D. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.19 · Cited by 5 · see citing works
Self-citationFrequency Balanced Datasets Lead to Better Language Models
Zevallos, R., Farrús, M., & Bel, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.527 · Cited by 4 · see citing works
Explicitly Representing Syntax Improves Sentence-to-Layout Prediction of Unexpected Situations
Nuyts, W., Cartuyvels, R., & Moens, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00643 · Cited by 4 · see citing works
Recursos linguísticos para o PLN específico de domínio: o Petrolês
Freitas, C., SOUZA, E. A. D., Castro, M. C., Cavalcanti, T., da Silva, P. F., & Cordeiro, F. C. (2023) · Linguamática · doi:10.21814/lm.15.2.412 · Cited by 4 · see citing works
Exploring lightweight language models for materials informatics with AlchemBERT
Wang, Y., Liu, X., Chen, H., Yang, T., Liu, X., & Wen, X. (2025) · Cell Reports Physical Science · doi:10.1016/j.xcrp.2025.102724 · Cited by 4 · see citing works
CLIMB – Curriculum Learning for Infant-inspired Model Building
Martinez, R. D., McGovern, H., Goriely, Z., Davis, C., Caines, A., Buttery, P., & Beinborn, L. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.10 · Cited by 2 · see citing works
de Souza, E. A., & Freitas, C. (2023) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2023.233980 · Cited by 2 · see citing works
McGill BabyLM Shared Task Submission: The Effects of Data Formatting and Structural Biases
Cheng, Z., Aralikatte, R., Porada, I., Spinoso-Di Piano, C., & Cheung, J. C. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.18 · Cited by 1 · see citing works
Tokenization with Factorized Subword Encoding
Samuel, D., & Øvrelid, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.890
Self-citationExploring Hybrid Pre-training for Automatic Essay Scoring
Carpi, M. M., Silveira, I. C., Mauá, D. D., & Finger, M. (2026) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2026.26589
- Authors
- Samuel, D., Kutuzov, A., Øvrelid, L., & Velldal, E.
- Affiliations
- University of Oslo (NO)
- Venue
- Findings of the Association for Computational Linguistics: EACL 2023
- Published
- 2023
- Pages
- 1954-1974
- OpenAlex
- W4386566517
- Cited by
- 14
- References
- 58
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 1 · 2023: 9
- FWCI
- 2.48
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- BERT, British National Corpus, masked language models, training data size, pre-training, training objectives, model architecture, data efficiency, English language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
