Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measures
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Benchmark performance is not correlated with a greater ability to predict human psycholinguistic data.
“Steuer et al. (2023) found that benchmark performance is not correlated with a greater ability to predict human psycholinguistic data.”
From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Steuer et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · see citing works
Oh, B.-D., & Schuler, W. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00548 · Cited by 107 · see citing works
When Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85 · see citing works
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works
TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
Eldan, R., & Li, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.07759 · Cited by 52 · see citing works
Flagged first, then by citations
Research works printed but not linked: 38
- Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapting pre- trained language models to African languages via multilingual adaptive fine-tuning. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336-4349, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
- Suhas Arehalli, Brian Dillon, and Tal Linzen. 2022. Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities. In Proceedings of the 26th Conference on Computational Natural Lan- guage Learning (CoNLL), pages 301-313, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lap- ata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735, Online. Association for Computational Linguistics. · DOI
- Andrea de Varda and Marco Marelli. 2023. Scaling in cognitive modelling: a multilingual approach to human reading times. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 139- 149, Toronto, Canada. Association for Computational Linguistics. · DOI
- Irene Fernandez Monsalve, Stefan L. Frank, and Gabriella Vigliocco. 2012. Lexical surprisal as a general predictor of reading time. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 398-408, Avignon, France. Association for Compu- tational Linguistics.
- Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven T. Piantadosi, and Evelina Fedorenko. 2021. The Natural Stories corpus: A reading-time corpus of English texts con- taining rare yyntactic constructions. Language Re- sources and Evaluation, 55(1):63-77.
- Jill Gilkerson, Jeffrey A. Richards, Steven F. Warren, Ju- dith K. Montgomery, Charles R. Greenwood, D. Kim- brough Oller, John H. L. Hansen, and Terrance D. Paul. 2017. Mapping the Early Language Envi- ronment Using All-Day Recordings and Automated Analysis. American Journal of Speech-Language Pathology, 26(2):248-265. · DOI · Paper page
- Michael Hahn and Frank Keller. 2016. Modeling human reading with neural attention. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 85-95, Austin, Texas. Association for Computational Linguistics. · DOI
- Micha Heilbron, Benedikt Ehinger, Peter Hagoort, and Floris de Lange. 2019. Tracking Naturalistic Lin- guistic Predictions with Deep Neural Language Mod- els. In 2019 Conference on Cognitive Computational Neuroscience, Berlin, Germany. Cognitive Computa- tional Neuroscience. · DOI
- Kasra Hosseini, Kaspar Beelen, Giovanni Colav- izza, and Mariona Coll Ardanuy. 2021. Neural Language Models for Nineteenth-Century English. ArXiv:2105.11321 [cs. · arXiv
- Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger P. Levy. 2020. A Systematic Assessment of Syntactic Generalization in Neural Language Mod- els. Publisher: arXiv Version Number: 2. · DOI
- Kuan-Jung Huang, Suhas Arehalli, Mari Kugemoto, Christian Muxica, Grusha Prasad, Brian Dillon, and Tal Linzen. 2023. Surprisal does not explain syntac- tic disambiguation difficulty: evidence from a large- scale benchmark. preprint, PsyArXiv. · DOI
- Eshin Jolly. 2018. Pymer4: Connecting R and Python for Linear Mixed Modeling. Journal of Open Source Software, 3(31):862. · DOI
- William Jurayj, William Rudman, and Carsten Eick- hoff. 2022. Garden-Path Traversal in GPT-2. ArXiv:2205.12302 · arXiv
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. ArXiv:2001.08361 [cs, stat]. · arXiv
- Alan Kennedy and Joël Pynte. 2005. Parafoveal-on- foveal effects in normal reading. Vision Research, 45(2):153-168. · DOI
- Tatsuki Kuribayashi, Yohei Oseki, Ana Brassard, and Kentaro Inui. 2022. Context Limitations Make Neu- ral Language Models More Human-Like. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10421-10436, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower Perplexity is Not Always Human-Like. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5203- 5217, Online. Association for Computational Lin- guistics. · DOI
- Alexandra Kuznetsova, Per B. Brockhoff, and Rune H. B. Christensen. 2017. lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13):1-26. · DOI · Paper page
- Jiaxuan Li and Richard Futrell. 2023. A decomposition of surprisal tracks the N400 and P600 brain potentials. Proceedings of the Annual Meeting of the Cognitive Science Society, 45(45).
- Aaron Mueller and Tal Linzen. 2023. How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Bi- ases. Publisher: arXiv Version Number: 1. · DOI
- Byung-Doh Oh and William Schuler. 2023. Transformer-Based LM Surprisal Predicts Hu- man Reading Times Best with About Two Billion Training Tokens. ArXiv:2304.11389 [cs. · arXiv
- Isabel Papadimitriou, Richard Futrell, and Kyle Ma- howald. 2022. When classifying grammatical role, BERT doesn't care about word order... except when it matters. ArXiv:2203.06204 [cs. · arXiv
- Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035.
- Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the Curse of Multilinguality by Pre-training Modular Transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479-3495, Seattle, United States. Association for Computational Lin- guistics. · DOI
- Ofir Press, Noah A. Smith, and Mike Lewis. 2021. Shortformer: Better Language Modeling using Shorter Inputs. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5493-5505, Online. Association for Computational Linguistics. · DOI · Paper page
- Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train Short, Test Long: Attention with Linear Biases En- ables Input Length Extrapolation. ArXiv:2108.12409 · DOI · Paper page
- Brian Roark, Asaf Bachrach, Carlos Cardenas, and Christophe Pallier. 2009. Deriving lexical and syntac- tic expectation-based measures for psycholinguistic modeling via incremental top-down parsing. In Pro- ceedings of the 2009 Conference on Empirical Meth- ods in Natural Language Processing, pages 324-333, Singapore. Association for Computational Linguis- tics.
- Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Ca- rina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative modeling converges on predictive processing. Pro- ceedings of the National Academy of Sciences, 118(45):e2105646118. · DOI · Paper page
- Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cot- terell, and Roger Philip Levy. 2022. Large-Scale Evi- dence for Logarithmic Effects of Word Predictability on Reading Time. preprint, PsyArXiv. · DOI
- Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A Multi-Task Benchmark and Anal- ysis Platform for Natural Language Understanding. ArXiv:1804.07461 [cs. · arXiv
- Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo- hananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020a. BLiMP: The Benchmark of Lin- guistic Minimal Pairs for English. Transactions of the Association for Computational Linguistics, 8:377- 392. · DOI · Paper page
- Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. 2020b. Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually). In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 217-235, Online. Association for Computational Lin- guistics. · DOI
- Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the Predictive Power of Neural Language Models for Human Real- Time Comprehension Behavior. Publisher: arXiv Version Number: 1. · DOI
- Cai Wingfield and Louise Connell. 2022. Understand- ing the role of linguistic distributional knowledge in cognition. volume 37, pages 1220-1270. Routledge. Ludwig Wittgenstein. 1953. Philosophische Unter- suchungen. Suhrkamp Verlag, Frankfurt am Main. · DOI
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics. · DOI · Paper page
- Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. 2023. To Repeat or Not To Repeat: Insights from Scaling LLM under Token- Crisis. · DOI · Paper page
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre- trained Transformer Language Models. Publisher: arXiv Version Number: 4. · DOI · Paper page
Cited by
Sorted by citations
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works
- Authors
- Steuer, J., Mosbach, M., & Klakow, D.
- Venue
- Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning
- Published
- 2023
- Pages
- 114-129
- Cited by
- 2
- References
- 5
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
