When Do You Need Billions of Words of Pretraining Data?
Open accessAbstract
Yian Zhang, Alex Warstadt, Xiaocheng Li, Samuel R. Bowman. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
When restricted to developmentally plausible data volumes, language models no longer perform well on benchmarks for human-like syntactic and semantic behavior.
“When restricted to developmentally plausible data volumes, language models no longer perform well on benchmarks for human-like syntactic and semantic behavior (van Schijndel et al., 2019;Zhang et al., 2021).”
From Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora (2023) · cites this paper as Zhang et al., 2021
The Results section reports that with 100 million words of pretraining data, performance on the BLiMP grammaticality benchmark was within nine accuracy points of humans. The Conclusion says syntactic and semantic abilities largely saturate after 10–100 million words, a volume it describes as comparable to data available to human learners. This contradicts the claim that models at developmentally plausible volumes no longer perform well on those benchmarks. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1× · see citing works
“Miller (1995); Schuler (2005), RTE from Dagan et al. (2006); Bar Haim et al. (2006); Giampiccolo et al. (2007); Bentivogli et al. (2009) Results We plot the results on SuperGLUE in Figure 6.”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Self-citationEnergy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684 · see citing works
Geurts, B. (1999) · Language · doi:10.2307/417746 · Cited by 2,848 · see citing works
5.7 The WORDNET lexical database of English
Ooi, V. B. Y. (2019) · Edinburgh University Press eBooks · doi:10.1515/9781474471459-050 · Cited by 2,034 · see citing works
Semantics in generative grammar
Heim, I., & Kratzer, A. (1998) · Cited by 1,905 · see citing works
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · see citing works
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works
Self-citationA primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552 · see citing works
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Self-citationCamemBERT: a Tasty French Language Model
Martin, L., Müller, B., Suárez, P. O., Dupont, Y., Romary, L., de la Clergerie, É. V., Seddah, D., & Sagot, B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.645 · Cited by 911 · see citing works
VerbNet: A broad-coverage, comprehensive verb lexicon
Schuler, K. K. (2005) · Scholarly Commons (University of Pennsylvania) · doi:10.5281/zenodo.8100881 · Cited by 842 · see citing works
The Third PASCAL Recognizing Textual Entailment Challenge
Giampiccolo, D., Magnini, B., Dagan, I., & Dolan, B. (2007) · ACL-PASCAL@ACL · doi:10.3115/1654536.1654538 · Cited by 801 · see citing works
The power law repealed: The case for an exponential law of practice
Heathcote, A., Brown, S. D., & Mewhort, D. J. K. (2000) · Psychonomic Bulletin & Review · doi:10.3758/bf03212979 · Cited by 735 · see citing works
(2019) · doi:10.18653/v1/n19-1419 · Cited by 601 · see citing works
Hart, B. M., & Risley, T. (1992) · Developmental Psychology · doi:10.1037/0012-1649.28.6.1096 · Cited by 595 · see citing works
The Seventh PASCAL Recognizing Textual Entailment Challenge.
Bentivogli, L., Clark, P. E., Dagan, I., & Giampiccolo, D. (2008) · Cited by 533 · see citing works
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models
Ettinger, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00298 · Cited by 511 · see citing works
Analysis Methods in Neural Language Processing: A Survey
Belinkov, Y., & Glass, J. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00254 · Cited by 485 · see citing works
Tenney, I., Xia, P., Chen, B., Wang, A., Poliak, A., McCoy, R. T., Kim, N., Durme, B. V., Bowman, S. R., Das, D., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.06316 · Cited by 480 · see citing works
Self-citationThe CommitmentBank: Investigating projection in naturally occurring discourse
de Marneffe, M.-C., Simons, M., & Tonhauser, J. (2019) · Digital Access to Libraries · doi:10.18148/sub/2019.v23i2.601 · Cited by 471 · see citing works
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.154 · Cited by 446 · see citing works
Self-citationBoolQ: Exploring the Surprising Difficulty of Natural Yes/No QuestionsPreprint
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10044 · Cited by 425 · see citing works
Salazar, J., Liang, D., Nguyen, T. Q., & Kirchhoff, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.240 · Cited by 414 · see citing works
Representing General Relational Knowledge in ConceptNet 5
Speer, R., & Havasi, C. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3zycidvg6dgg · Cited by 412 · see citing works
Designing and Interpreting Probes with Control Tasks
Hewitt, J., & Liang, P. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1275 · Cited by 393 · see citing works
Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction TasksPreprint
Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., & Goldberg, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.04207 · Cited by 374 · see citing works
SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of NominalsPreprint
Hendrickx, I., Kim, S., Kozareva, Z., Nakov, P., Séaghdha, D. Ó., Padó, S., Pennacchiotti, M., Romano, L., & Szpakowicz, S. (2010) · International Workshop on Semantic Evaluation · doi:10.48550/arxiv.1911.10422 · Cited by 367 · see citing works
Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning
Roemmele, M., Bejan, C. A., & Gordon, A. S. (2011) · Cited by 334 · see citing works
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Pilehvar, M. T., & Camacho-Collados, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1128 · Cited by 296 · see citing works
A Gold Standard Dependency Corpus for English
Silveira, N., Dozat, T., de Marneffe, M.-C., Bowman, S. R., Connor, M. K., Bauer, J., & Manning, C. D. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4esbhdz7y5u5 · Cited by 237 · see citing works
Self-citationIntermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?
Pruksachatkun, Y., Phang, J., Liu, H., Htut, P. M., Zhang, X., Pang, R. Y., Vania, C., Kann, K., & Bowman, S. R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.467 · Cited by 229 · see citing works
Self-citationFine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction TasksPreprint
Adi, Y. (2024) · arXiv (Cornell University) · doi:10.57702/tcr06vve · Cited by 228 · see citing works
Information-Theoretic Probing with Minimum Description Length
Voita, E., & Titov, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.14 · Cited by 197 · see citing works
(2018) · doi:10.63317/2yu5nu76n7bu · Cited by 189 · see citing works
Information-Theoretic Probing for Linguistic Structure
Pimentel, T., Valvoda, J., Maudslay, R. H., Zmigrod, R., Williams, A., & Cotterell, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.420 · Cited by 173 · see citing works
Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge
Rahman, A., & Ng, V. (2012) · Cited by 141 · see citing works
A Systematic Assessment of Syntactic Generalization in Neural Language Models
Hu, J., Gauthier, J., Qian, P., Wilcox, E., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.158 · Cited by 134 · see citing works
Probing for semantic evidence of composition by means of simple classification tasks
Ettinger, A., Elgohary, A., & Resnik, P. (2016) · Workshop on Evaluating Vector-Space Representations for NLP · doi:10.18653/v1/w16-2524 · Cited by 131 · see citing works
The exponential learning equation as a function of successful trials results in sigmoid performance
Leibowitz, N., Baum, B., Enden, G., & Karniel, A. (2010) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2010.01.006 · Cited by 127 · see citing works
Superlative quantifiers and meta-speech acts
Cohen, A., & Krifka, M. (2014) · Linguistics and Philosophy · doi:10.1007/s10988-014-9144-x · Cited by 120 · see citing works
Warstadt, A., Zhang, Y., Li, H.-S., Liu, H., & Bowman, S. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.16 · Cited by 106 · see citing works
Self-citationInvestigating BERT’s Knowledge of Language: Five Analysis Methods with NPIs
Warstadt, A., Cao, Y. H., Grosu, I., Peng, W., Blix, H., Nie, Y., Alsop, A., Bordia, S., Liu, H., Parrish, A., et al. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1286 · Cited by 100 · see citing works
Self-citationInference is Everything: Recasting Semantic Resources into a Unified Evaluation Framework
White, A. S., Rastogi, P., Duh, K., & Durme, B. V. (2017) · International Joint Conference on Natural Language Processing · Cited by 72 · see citing works
Quantity doesn’t buy quality syntax with neural language models
van Schijndel, M., Mueller, A., & Linzen, T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1592 · Cited by 65 · see citing works
WiC: 10, 000 Example Pairs for Evaluating Context-Sensitive RepresentationsPreprint
Pilehvar, M. T., & Camacho-Collados, J. (2018) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.1808.09121 · Cited by 50 · see citing works
Probing Across Time: What Does RoBERTa Know and When?
Liu, Z., Wang, Y., Kasai, J., Hajishirzi, H., & Smith, N. A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.71 · Cited by 42 · see citing works
On the importance of pre-training data volume for compact language models
Micheli, V., d’Hoffschmidt, M., & Fleuret, F. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.632 · Cited by 37 · see citing works
Pretrained Language Model Embryology: The Birth of ALBERT
Chiang, C.-H., Huang, S.-F., & Lee, H.-Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.553 · Cited by 34 · see citing works
Understanding Learning Dynamics Of Language Models with SVCCA
Saphra, N., & Lopez, A. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1329 · Cited by 31 · see citing works
BLiMP: the benchmark of linguistic minimal pairs for English
Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00321 · Cited by 28 · see citing works
Self-citationWhen Bert Forgets How To POS: Amnesic Probing of Linguistic Properties and MLM PredictionsPreprint
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2020) · arXiv (Cornell University) · Cited by 22 · see citing works
Teichert, A. R., Poliak, A., Durme, B. V., & Gormley, M. R. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11165 · Cited by 20 · see citing works
Neural-Davidsonian Semantic Proto-role Labeling
Rudinger, R., Teichert, A. R., Culkin, R., Zhang, S., & Durme, B. V. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1114 · Cited by 14 · see citing works
Evaluating representations by the complexity of learning low-loss predictorsPreprint
Whitney, W. F., Song, M. J., Brandfonbrener, D., Altosaar, J., & Cho, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.07368 · Cited by 12 · see citing works
Precise Task Formalization Matters in Winograd Schema Evaluations
Liu, H., Huang, W., Mungra, D., & Bowman, S. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.664 · Cited by 10 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
5 of the 69 references couldn’t be matched to a known paper.
Research works printed but not linked: 7
- Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognis- ing Textual Entailment.
- Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment chal- lenge. In Textual Analysis Conference (TAC).
- Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh Interna- tional Conference on Language Resources and Eval- uation (LREC-2018).
- Charles Lovering, Rohan Jha, Tal Linzen, and El- lie Pavlick. 2021. Predicting inductive biases of fine-tuned models. In International Conference on Learning Representations.
- Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):1-67.
- Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of plausible alterna- tives: An evaluation of commonsense causal reason- ing. In 2011 AAAI Spring Symposium Series.
- Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Eval- uating Neural Language Generation, pages 30-36, Minneapolis, Minnesota. Association for Computa- tional Linguistics. · DOI
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Self-citationRecent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · see citing works
Probing Classifiers: Promises, Shortcomings, and Advances
Belinkov, Y. (2021) · Computational Linguistics · doi:10.1162/coli_a_00422 · Cited by 311 · see citing works
RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL
Li, H., Zhang, J., Li, C., & Chen, H. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26535 · Cited by 220 · see citing works
The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language ModelsPreprint
Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., & Habash, N. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06678 · Cited by 200 · see citing works
QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 177 · see citing works
Ogueji, K., Zhu, Y., & Lin, J. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.11 · Cited by 124 · see citing works
The limitations and ethical considerations of ChatGPT
Hua, S., Jin, S., & Jiang, S. (2024) · Data Intelligence · doi:10.1162/dint_a_00243 · Cited by 120 · see citing works
What Artificial Neural Networks Can Tell Us about Human Language Acquisition
Warstadt, A., & Bowman, S. R. (2022) · Algebraic Structures in Natural Language · doi:10.1201/9781003205388-2 · Cited by 111 · see citing works
Self-citationLanguage Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
ExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10952 · Cited by 94 · see citing works
Testing the Predictions of Surprisal Theory in 11 Languages
Wilcox, E., Pimentel, T., Meister, C., Cotterell, R., & Lévy, R. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00612 · Cited by 87 · see citing works
Semantic Structure in Deep Learning
Pavlick, E. (2021) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-031120-122924 · Cited by 76 · see citing works
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74 · see citing works
“When restricted to developmentally plausible data volumes, language models no longer perform well on benchmarks for human-like syntactic and semantic behavior (van Schijndel et al., 2019; Zhang et al., 2021).”
Self-citationMcCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70 · see citing works
An Overview of Language Models: Recent Developments and Outlook
Wei, C., Wang, Y.-C., Wang, B., & Kuo, C. ‐. J. (2024) · APSIPA Transactions on Signal and Information Processing · doi:10.1561/116.00000010 · Cited by 55 · see citing works
Changing the World by Changing the Data
Rogers, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.170 · Cited by 51 · see citing works
The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail
Bowman, S. R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.516 · Cited by 45 · see citing works
Self-citationProbing Across Time: What Does RoBERTa Know and When?
Liu, Z., Wang, Y., Kasai, J., Hajishirzi, H., & Smith, N. A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.71 · Cited by 42 · see citing works
Applying large language models for automated essay scoring for non-native Japanese
Li, W., & Liu, H. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03209-9 · Cited by 36 · see citing works
Ding, Y., Li, H., Zhu, F., Wang, Z., Peng, W., & Xie, M. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3299078 · Cited by 31 · see citing works
How much pretraining data do language models need to learn syntax?
Pérez-Mayos, L., Ballesteros, M., & Wanner, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.118 · Cited by 26 · see citing works
An Adaptive Graph Pre-training Framework for Localized Collaborative Filtering
Wang, Y., Li, C., Liu, Z., Li, M., Tang, J., Xie, X., Chen, L., & Yu, P. S. (2022) · ACM Transactions on Information Systems · doi:10.1145/3555372 · Cited by 26 · see citing works
DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains
Labrak, Y., Bazoge, A., Dufour, R., Rouvier, M., Morin, E., Daille, B., & Gourraud, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.896 · Cited by 26 · see citing works
Nangia, N., Sugawara, S., Trivedi, H. K., Warstadt, A., Vania, C., & Bowman, S. R. (2021) · doi:10.18653/v1/2021.acl-long.98 · Cited by 25 · see citing works
Self-citationMVP: Multi-task Supervised Pre-training for Natural Language Generation
Tang, T., Li, J., Zhao, W. X., & Wen, J.-R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.558 · Cited by 23 · see citing works
A Better Way to Do Masked Language Model Scoring
Kauf, C., & Ivanova, A. A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.80 · Cited by 23 · see citing works
Misra, K., Rayz, J. T., & Ettinger, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.213 · Cited by 23 · see citing works
RuCoLA: Russian Corpus of Linguistic Acceptability
Mikhailov, V., Shamardina, T., Ryabinin, M., Pestova, A., Smurov, I., & Artemova, E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.348 · Cited by 21 · see citing works
Neural reality of argument structure constructions
Li, B., Zhu, Z., Thomas, G., Rudzicz, F., & Xu, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.512 · Cited by 21 · see citing works
Context Matters: A Pragmatic Study of PLMs’ Negation Understanding
Gubelmann, R., & Handschuh, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.315 · Cited by 20 · see citing works
Comparing Test Sets with Item Response Theory
Vania, C., Htut, P. M., Huang, W., Mungra, D., Pang, R. Y., Phang, J., Liu, H., Cho, K., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.92 · Cited by 18 · see citing works
Self-citationOpenly accessible LLMs can help us to understand human cognition
Frank, M. C. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01732-4 · Cited by 18 · see citing works
Acceptability Judgements via Examining the Topology of Attention Maps
Cherniavskii, D., Tulchinskii, E., Mikhailov, V., Proskurina, I., Kushnareva, L., Artemova, E., Barannikov, S., Piontkovskaya, I., Piontkovski, D., & Burnaev, E. V. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.7 · Cited by 18 · see citing works
Construction Grammar and Language Models
Madabushi, H. T., Romain, L., Milin, P., & Divjak, D. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009049139.023 · Cited by 16 · see citing works
Rasmussen, S. H. R., Bor, A., Osmundsen, M., & Petersen, M. B. (2023) · British Journal of Political Science · doi:10.1017/s0007123423000042 · Cited by 16 · see citing works
Trained on 100 million words and still in shape: BERT meets British National Corpus
Samuel, D., Kutuzov, A., Øvrelid, L., & Velldal, E. (2023) · Findings of the Association for Computational Linguistics: EACL 2023 · doi:10.18653/v1/2023.findings-eacl.146 · Cited by 14 · see citing works
Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models
Youssef, P., Koraş, O. A., Li, M., Schlötterer, J., & Seifert, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1043 · Cited by 13 · see citing works
Static Embeddings as Efficient Knowledge Bases?
Dufter, P., Kassner, N., & Schütze, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.186 · Cited by 13 · see citing works
Bridging Fairness and Environmental Sustainability in Natural Language Processing
Hessenthaler, M., Strubell, E., Hovy, D., & Lauscher, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.533 · Cited by 13 · see citing works
Should We Be Pre-training? An Argument for End-task Aware Training as an AlternativePreprint
Dery, L. M., Michel, P., Talwalkar, A., & Neubig, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07437 · Cited by 13 · see citing works
From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency
Ohmer, X. I., Bruni, E., & Hupkes, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00529 · Cited by 11 · see citing works
Modeling Brain Representations of Words' Concreteness in Context Using GPT‐2 and Human Ratings
Bruera, A., Tao, Y., Anderson, A. J., Çokal, D., Haber, J., & Poesio, M. (2023) · Cognitive Science · doi:10.1111/cogs.13388 · Cited by 10 · see citing works
Holmes ⌕ A Benchmark to Assess the Linguistic Competence of Language Models
Waldis, A., Waldis, A., Perlitz, Y., Choshen, L., Choshen, L., Hou, Y., & Gurevych, I. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00718 · Cited by 9 · see citing works
Cross-lingual Transfer of Monolingual Models
Gogoulou, E., Ekgren, A., Isbister, T., & Sahlgren, M. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5635surro6fy · Cited by 9 · see citing works
Pre-Training a Language Model Without Human LanguagePreprint
Chiang, C.-H., & Lee, H.-Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11995 · Cited by 9 · see citing works
Influence of LLMs on Learning and Teaching in Higher EducationPreprint
Upadhyay, A., Farahmand, E., Muñoz, I., Khan, M. A., & Witte, N. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4716855 · Cited by 9 · see citing works
Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair
Phang, J., Chen, A., Huang, W., & Bowman, S. R. (2022) · doi:10.18653/v1/2022.dadc-1.8 · Cited by 8 · see citing works
Self-citationInvestigating Critical Period Effects in Language Acquisition through Neural Language Models
Constantinescu, I., Pimentel, T., Cotterell, R., & Warstadt, A. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00725 · Cited by 7 · see citing works
Self-citationA Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese
Liu, Y., Shen, Y., Zhu, H., Xu, L., Qian, Z., Song, S., Zhang, K., Tang, J., Zhang, P., Yang, B., et al. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.648 · Cited by 7 · see citing works
On the Role of Corpus Ordering in Language Modeling
Agrawal, A., Singh, S., Schneider, L. R., & Samuels, M. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.15 · Cited by 7 · see citing works
Are Sample-Efficient NLP Models More Robust?
Liu, N. F., Kumar, A., Liang, P., & Jia, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.144 · Cited by 7 · see citing works
DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainsPreprint
Labrak, Y., Bazoge, A., Dufour, R., Rouvier, M., Morin, E., Daille, B., & Gourraud, P. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.04.03.535368 · Cited by 7 · see citing works
The Effect of Scaling, Retrieval Augmentation and Form on the Factual Consistency of Language Models
Hagström, L., Saynova, D., Norlund, T., Johansson, M., & Johansson, R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.332 · Cited by 6 · see citing works
Decoupling SQL query hardness parsing for text-to-SQL
Yi, J., Chen, G., & Zhou, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129293 · Cited by 6 · see citing works
Ethics consideration sections in natural language processing papers
Benotti, L., & Blackburn, P. R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.299 · Cited by 6 · see citing works
Natural syntax, artificial intelligence and language acquisition
O’Grady, W., & Lee, M. (2023) · Information · doi:10.3390/info14070418 · Cited by 5 · see citing works
Exploring Jiu-Jitsu Argumentation for Writing Peer Review Rebuttals
Purkayastha, S., Lauscher, A., & Gurevych, I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.894 · Cited by 5 · see citing works
Shades of zero: Distinguishing impossibility from inconceivability
Hu, J., Sosa, F., & Ullman, T. D. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104640 · Cited by 5 · see citing works
Exploring Transformers as Compact, Data-efficient Language Models
Fields, C., & Kennington, C. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.35 · Cited by 5 · see citing works
Visual Grounding Helps Learn Word Meanings in Low-Data Regimes
Zhuang, C., Fedorenko, E. G., & Andreas, J. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.71 · Cited by 5 · see citing works
EMTeC: A corpus of eye movements on machine-generated texts
Bolliger, L. S., Haller, P., Cretton, I. C. R., Reich, D. R., Kew, T., & Jäger, L. A. (2025) · Behavior Research Methods · doi:10.3758/s13428-025-02677-4 · Cited by 5 · see citing works
Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale.
Deshpande, V., Pechi, D., Thatte, S., Lialin, V., & Rumshisky, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.326 · Cited by 5 · see citing works
A Proposal of Post-OCR Spelling Correction Using Monolingual Byte-level Language Models
de Araújo, S. S., Bezerra, B. L. D., & Neto, A. F. D. S. (2025) · ACM Symposium on Document Engineering (DocEng) · doi:10.1145/3704268.3748673 · Cited by 4 · see citing works
Not Enough Data to Pre-train Your Language Model? MT to the Rescue!
Urbizu, G., Vicente, I. S., Saralegi, X., & Corral, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.235 · Cited by 4 · see citing works
Frequency Balanced Datasets Lead to Better Language Models
Zevallos, R., Farrús, M., & Bel, N. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.527 · Cited by 4 · see citing works
Tiny Language Models Enriched with Multimodal Knowledge from Multiplex Networks
Fields, C., Natouf, O., McMains, A., Henry, C., & Kennington, C. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.3 · Cited by 4 · see citing works
Explicitly Representing Syntax Improves Sentence-to-Layout Prediction of Unexpected Situations
Nuyts, W., Cartuyvels, R., & Moens, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00643 · Cited by 4 · see citing works
The Rediscovery Hypothesis: Language Models Need to Meet Linguistics
Nikoulina, V., Tezekbayev, M., Kozhakhmet, N., Babazhanova, M., Gallé, M., & Assylbekov, Z. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12788 · Cited by 4 · see citing works
Ad astra or astray: Exploring linguistic knowledge of multilingual BERT through NLI task
Tikhonova, M., Mikhailov, V., Pisarevskaya, D., Malykh, V., & Shavrina, T. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000225 · Cited by 4 · see citing works
Automatic Document Selection for Efficient Encoder Pretraining
Feng, Y., Xia, P., Durme, B. V., & Sedoc, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.647 · Cited by 4 · see citing works
Hints on the data for language modeling of synthetic languages with transformers
Zevallos, R., & Bel, N. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.699 · Cited by 4 · see citing works
Language-Agnostic Bias Detection in Language Models with Bias Probing
Köksal, A., Yalcin, O. F., Akbiyik, A., Kılavuz, M. T., Korhonen, A., & Schuetze, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.848 · Cited by 4 · see citing works
Identifying task groupings for multi-task learning using pointwise V-usable information
Li, Y., Miller, T., Bethard, S. J., & Savova, G. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104881 · Cited by 3 · see citing works
Can BERT eat RuCoLA? Topological Data Analysis to Explain
Proskurina, I., Artemova, E., & Piontkovskaya, I. (2023) · Workshop on Slavic Natural Language Processing (SlavicNLP) · doi:10.18653/v1/2023.bsnlp-1.15 · Cited by 3 · see citing works
Scaling Laws for BERT in Low-Resource Settings
Urbizu, G., Vicente, I. S., Saralegi, X., Agerri, R., & Soroa, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.492 · Cited by 3 · see citing works
LLMs model how humans induce logically structured rules
Loo, A. M., Pavlick, E., & Feiman, R. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104675 · Cited by 3 · see citing works
When Combating Hype, Proceed with Caution.Preprint
Bowman, S. R. (2021) · arXiv (Cornell University) · Cited by 3 · see citing works
Self-citationCLIMB – Curriculum Learning for Infant-inspired Model Building
Martinez, R. D., McGovern, H., Goriely, Z., Davis, C., Caines, A., Buttery, P., & Beinborn, L. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.10 · Cited by 2 · see citing works
Steuer, J., Mosbach, M., & Klakow, D. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.12 · Cited by 2 · see citing works
Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models
Pirozelli, P., José, M. M., Filho, P. D. T. P. D. C., Brandão, A. A. F., & Cozman, F. G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71167-1_2 · Cited by 2 · see citing works
Zhu, M., Liu, Y., Luo, Z., & Zhu, T. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02176-y · Cited by 2 · see citing works
AI-Driven Conservation of the Endangered Twisted Yew (Taxus contorta Griff.) in the Western Himalaya
Din, S., Ali, H., Panagopoulos, Τ., Alam, J., Malik, S., & Sher, H. (2025) · Sustainability · doi:10.3390/su17198541 · Cited by 2 · see citing works
Larger Probes Tell a Different Story: Extending Psycholinguistic Datasets Via In-Context Learning
Shivagunde, N., Lialin, V., & Rumshisky, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.130 · Cited by 2 · see citing works
What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledge
Hagström, L., & Johansson, R. (2022) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2022.acl-srw.19 · Cited by 2 · see citing works
Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways
Govindarajan, V. S., Rodriguez, J. D., Bostrom, K., & Mahowald, K. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.27 · Cited by 1 · see citing works
Training compute-optimal transformer encoder models
Dervishi, M., Allauzen, A., Synnaeve, G., & LeCun, Y. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1804 · Cited by 1 · see citing works
Generative Data Augmentation via Wasserstein Autoencoder for Text Classification
Jin, K., Lee, J., Choi, J., Jang, S., & Kim, Y. (2022) · International Conference on Information and Communication Technology Convergence (ICTC) · doi:10.1109/ictc55196.2022.9952762 · Cited by 1 · see citing works
Acquiring Linguistic Knowledge from Multimodal Input
Amariucai, T., & Warstadt, A. S. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.11 · Cited by 1 · see citing works
Self-citationEdge-Deployed Context-Aware LLM Framework for Low-Latency Bi-Directional Gaze-Speech HRIPreprint
Zhai, Y., & Chian, Y.-T. (2026) · Preprints.org · doi:10.20944/preprints202602.0308.v1 · Cited by 1 · see citing works
Sun, S., Dillon, B., & Iyyer, M. (2022) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2022.insights-1.6
Tran, S. Q., Nguyen-Thuan, P., Nguyen, K. V., & Nguyen, N. L.-T. (2023) · Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop (EACL SRW) · doi:10.18653/v1/2023.eacl-srw.1
Explorando Redes Neurais Profundas para Tarefa de Aceitabilidade Linguística
Santos, H., Alencar, K. P., de Sousa, R. F., & Anchiêta, R. T. (2023) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2023.234644
Foundations of Language Modeling
Beinborn, L., & Hollenstein, N. (2023) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-43260-6_2
NPIs Aren’t Exactly Easy: Variation in Licensing across Large Language Models
DeCarlo, D., Palmer, W., Wilson, M., & Frank, B. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.25
Bootstrapping Small & High Performance Language Models with Unmasking-Removal Training Policy
Yang, Y., Sulem, E., Lee, I., & Roth, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.30
Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling
Zhuang, C., Fedorenko, E. G., & Andreas, J. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.15
Word Vector Representation of Latin Cuengh Based on Root Feature Enhancement
Lyu, W., Chen, J., Qin, X., & Li, J. (2024) · Applied Sciences · doi:10.3390/app15010211
Evaluating L2 Training Methods in Neural Language Models
Lee, J., & Shin, J. (2024) · 어학연구 · doi:10.30961/lr.2024.60.3.323
- Authors
- Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R.
- Affiliations
- New York University (US)
- Venue
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Published
- 2021
- Pages
- 1112-1125
- OpenAlex
- W3176198948
- Cited by
- 85
- References
- 69
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 9 · 2023: 40 · 2022: 16 · 2021: 10
- FWCI
- 8.66
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Funders
- National Science Foundation (1850208)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- pre-training, language models, training data requirements
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
