CoChat
Sign Up
Cited by 5Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings

Open access
Samuel, David·2023·Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning
DOI 10.18653/v1/2023.conll-babylm.19
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper explores the use of latent bootstrapping, an alternative self-supervision technique, for pretraining language models.Unlike the typical practice of using self-supervision on discrete subwords, latent bootstrapping leverages contextualized embeddings for a richer supervision signal.We conduct experiments to assess how effective this approach is for acquiring linguistic knowledge from limited resources.Specifically, our experiments are based on the BabyLM shared task, which includes pretraining on two small curated corpora and an evaluation on four linguistic benchmarks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.conll-babylm.19

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 57

  1. Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014. The AMARA corpus: Building parallel language resources for the educa- tional domain. In Proceedings of the Ninth Inter- national Conference on Language Resources and Evaluation (LREC'14), Reykjavik, Iceland. European Language Resources Association (ELRA).
  2. Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. 2023. Self-supervised learning from images with a joint-embedding predic- tive architecture. arXiv preprint arXiv:2301.08243. · arXiv
  3. Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022. data2vec: A general framework for self-supervised learning in speech, vision and language. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1298-1312. PMLR.
  4. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. 2006. The second pascal recog- nising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop on Recog- nising Textual Entailment.
  5. Adrien Bardes, Jean Ponce, and Yann LeCun. 2022. VICReg: Variance-invariance-covariance regulariza- tion for self-supervised learning. In International Conference on Learning Representations.
  6. Yonatan Belinkov. 2022. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Lin- guistics, 48(1):207-219. · DOI
  7. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth pascal recognizing textual entailment challenge. In In Proc Text Analysis Conference (TAC'09.
  8. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org.
  9. Xinlei Chen and Kaiming He. 2020. Exploring sim- ple siamese representation learning. arXiv preprint arXiv:2011.10566. · arXiv
  10. Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15750-15758.
  11. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924-2936, Minneapolis, Min- nesota. Association for Computational Linguistics. · DOI
  12. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment chal- lenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177-190, Berlin, Heidelberg. Springer Berlin Heidelberg.
  13. William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
  14. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48. · DOI · Paper page
  15. Martin Gerlach and Francesc Font-Clos. 2018. A stan- dardized Project Gutenberg corpus for statistical anal- ysis of natural language and quantitative linguistics. Computing Research Repository, arXiv:1812.08092. · DOI
  16. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague. Association for Computational Linguistics.
  17. Ross Girshick. 2015. Fast r-cnn. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), ICCV '15, page 1440-1448, USA. IEEE Computer Society. · DOI · Paper page
  18. Thamme Gowda and Jonathan May. 2020. Finding the optimal vocabulary size for neural machine transla- tion. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 3955-3964, Online. Association for Computational Linguistics. · DOI
  19. Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. 2020. Bootstrap your own latent a new approach to self-supervised learning. In Proceedings of the 34th International Conference on Neural In- formation Processing Systems, NIPS'20, Red Hook, NY, USA.
  20. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Wein- berger. 2017a. On calibration of modern neural net- works. In Proceedings of the 34th International Con- ference on Machine Learning -Volume 70, ICML'17, page 1321-1330. JMLR.org.
  21. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Pi- otr Dollár, and Ross Girshick. 2022. Masked autoen- coders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009.
  22. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for un- supervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 9726-9735. · DOI · Paper page
  23. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
  24. R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In International Conference on Learn- ing Representations.
  25. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. · DOI · Paper page
  26. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span- BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77. · DOI · Paper page
  27. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa- pers), pages 252-262, New Orleans, Louisiana. As- sociation for Computational Linguistics. · DOI
  28. Yann LeCun. 2022. A path towards autonomous ma- chine intelligence version 0.9. 2, 2022-06-27. Open Review, 62.
  29. Hector J. Levesque, Ernest Davis, and Leora Morgen- stern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Confer- ence on Principles of Knowledge Representation and Reasoning, KR'12, page 552-561. AAAI Press.
  30. Pierre Lison and Jörg Tiedemann. 2016. OpenSub- titles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 923-929, Portorož, Slovenia. European Language Resources Association (ELRA).
  31. Brian MacWhinney. 2000. The CHILDES project: The database, volume 2. Psychology Press.
  32. Louis Martin, Benjamin Muller, Pedro Javier Or- tiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: a tasty French language model. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 7203- 7219, Online. Association for Computational Lin- guistics. · DOI
  33. B.W. Matthews. 1975. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA) -Protein Struc- ture, 405(2):442-451. · DOI · Paper page
  34. Vincent Micheli, Martin d'Hoffschmidt, and François Fleuret. 2020. On the importance of pre-training data volume for compact language models. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7853-7858, Online. Association for Computational Linguistics. · DOI
  35. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics. · DOI · Paper page
  36. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Ka- trin Kirchhoff. 2020. Masked language model scor- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguistics. · DOI
  37. David Samuel, Andrey Kutuzov, Samia Touileb, Erik Velldal, Lilja Øvrelid, Egil Rønningstad, Elina Sigdel, and Anna Palatkina. 2023b. NorBench -a bench- mark for Norwegian language models. In Proceed- ings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 618-633, Tórshavn, Faroe Islands. University of Tartu Library.
  38. Noam Shazeer. 2020. GLU variants improve trans- former. CoRR, abs/2002.05202. · arXiv
  39. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
  40. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Tay- lor, Rachel Martin, Marie Meteer, and Carol Van Ess-Dykema. 2000. Dialogue act modeling for au- tomatic tagging and recognition of conversational speech. Computational Linguistics, 26(3):339-371.
  41. Antti Tarvainen and Harri Valpola. 2017. Mean teach- ers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In Proceedings of the 31st International Con- ference on Neural Information Processing Systems, NIPS'17, page 1195-1204, Red Hook, NY, USA.
  42. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Ben- jamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from con- text? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30.
  44. Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov ran- dom field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluat- ing Neural Language Generation, pages 30-36, Min- neapolis, Minnesota. Association for Computational Linguistics. · DOI
  45. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Advances in Neural Information Processing Systems, volume 32.
  46. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics. · DOI · Paper page
  47. Alex Warstadt, Leshem Choshen, Aaron Mueller, Ad- ina Williams, Ethan Wilcox, and Chengxu Zhuang. 2023a. Call for papers -the babylm challenge: Sample-efficient pretraining on a developmentally plausible corpus. Computing Research Repository, arXiv:2301.11796. · arXiv
  48. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Gotlieb Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Adina Williams, Bhargavi Paran- jabe, Tal Linzen, and Ryan Cotterell. 2023b. Find- ings of the 2023 BabyLM Challenge: Sample- efficient pretraining on developmentally plausible corpora. In Proceedings of the 2023 BabyLM Chal- lenge. Association for Computational Linguistics (ACL).
  49. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo- hananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020a. BLiMP: The benchmark of linguis- tic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377- 392. · DOI · Paper page
  50. Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641. · DOI
  51. Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020b. Learning which features matter: RoBERTa acquires a preference for linguistic generalizations (eventually). In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 217-235, Online. Association for Computational Lin- guistics. · DOI
  52. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics. · DOI · Paper page
  53. C. Wilson. 2006. Learning phonology with substantive bias: an experimental and computational study of velar palatalization. Cogn Sci, 30(5):945-982.
  54. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. · DOI · Paper page
  55. Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. 2021. Barlow twins: Self-supervised learning via redundancy reduction. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12310-12320.
  56. PMLR.
  57. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021. When do you need bil- lions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1112-1125, Online. Association for Computational Linguistics. · DOI · Paper page

Cited by

Sorted by citations

Authors
Samuel, D.
Affiliations
University of Oslo (NO)
Corresponding author
Samuel, David
ORCID
Samuel, David
Venue
Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning
Publisher
Association for Computational Linguistics
Published
2023
Pages
193-209
DOI
10.18653/v1/2023.conll-babylm.19
OpenAlex
W4389519508
Cited by
5
References
3
Citations by year
2025: 1 · 2024: 2 · 2023: 2
FWCI
0.89
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • BabyLM Challenge (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Univerzita Karlova v Praze
  • Universitetet i Oslo
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
self-supervised pre-training, language model pretraining, low-resource languages, contextual embeddings, BabyLM, linguistic knowledge acquisition, multilingual language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review