Targeted Syntactic Evaluation of Language Models

Attention Is All You Need

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404

Long Short-Term Memory

Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318

Finding structure in time

Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212

Recurrent neural network based language model

Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390

SRILM - an extensible language modeling toolkit

Stolcke, A. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2002-303 · Cited by 4,385

Neural Network Acceptability Judgments

Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 1,215

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

Linzen, T., Emmanuel Dupoux, A., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 820

Agreement attraction in comprehension: Representations and processes

WAGERS, M. W., Lau, E. F., & Phillips, C. (2009) · Journal of Memory and Language · doi:10.1016/j.jml.2009.04.002 · Cited by 639

Broken agreement

Bock, K., & Miller, C. (1991) · Cognitive Psychology · doi:10.1016/0010-0285(91)90003-7 · Cited by 558

Colorless Green Recurrent Networks Dream Hierarchically

Gulordava, K., Bojanowski, P., Grave, É., Linzen, T., & Baroni, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1108 · Cited by 538

Recurrent Neural Network Grammars

Dyer, C., Kuncoro, A., Ballesteros, M., & Smith, N. A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1024 · Cited by 517

Gender Bias in Coreference Resolution

Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443

On the State of the Art of Evaluation in Neural Language ModelsPreprint

Melis, G., Dyer, C., & Blunsom, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.05589 · Cited by 397

CCGbank: A Corpus of CCG Derivations and Dependency Structures Extracted from the Penn Treebank

Hockenmaier, J., & Steedman, M. J. (2007) · Computational Linguistics · doi:10.1162/coli.2007.33.3.355 · Cited by 383

Supertagging: an approach to almost parsing

Bangalore, S., & Joshi, A. K. (1999) · Cited by 380

Probabilistic Top-Down Parsing and Language Modeling

Brian Roark, A. (2001) · Computational Linguistics · doi:10.1162/089120101750300526 · Cited by 331

Grammaticality, Acceptability, and Probability: A Probabilistic View of Linguistic Knowledge

Lau, J. H., Clark, A., & Lappin, S. (2016) · Cognitive Science · doi:10.1111/cogs.12414 · Cited by 262

Structures, Not Strings: Linguistics as Part of the Cognitive Sciences

Everaert, M. B. H., Huybregts, M. A. C., Chomsky, N., Berwick, R. C., & Bolhuis, J. J. (2015) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2015.09.008 · Cited by 260

Illusory licensing effects across dependency types: ERP evidence

XIANG, M., Dillon, B., & Phillips, C. (2008) · Brain and Language · doi:10.1016/j.bandl.2008.10.002 · Cited by 244

Interference effects from grammatically unavailable constituents during sentence processing.

Dyke, J. A. V. (2007) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/0278-7393.33.2.407 · Cited by 238

The Importance of Being Recurrent for Modeling Hierarchical Structure

Tran, K. M., Bisazza, A., & Monz, C. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1503 · Cited by 184

FOIL it! Find One mismatch between Image and Language caption

Shekhar, R., Pezzelle, S., Klimovich, Y., Herbelot, A., Nabi, M., Sangineto, E., & Bernardi, R. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1024 · Cited by 145

LSTMs Can Learn Syntax-Sensitive Dependencies Well, But Modeling Structure Makes Them Better

Kuncoro, A., Dyer, C., Hale, J., Yogatama, D., Clark, S. C., & Blunsom, P. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1132 · Cited by 134

Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic ScaffoldPreprint

Swayamdipta, S., Thomson, S., Dyer, C., & Smith, N. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.09528 · Cited by 131

5: Grammatical Illusions and Selective Fallibility in Real-Time Language Comprehension

Phillips, C., WAGERS, M. W., & Lau, E. F. (2011) · BRILL eBooks · doi:10.1163/9781780523750_006 · Cited by 123

Design sensitivity and statistical power in acceptability judgment experiments

Sprouse, J., & Almeida, D. (2017) · Glossa a journal of general linguistics · doi:10.5334/gjgl.236 · Cited by 89

Using a stochastic context-free grammar as a language model for speech recognition

Jurafsky, D., Wooters, C., Segal, J., Stolcke, A., Fosler, E., Tajchaman, G., & Morgan, N. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1995.479396 · Cited by 89

5 Grammatical Illusions and Selective Fallibility in Real-Time Language Comprehension

Phillips, C., WAGERS, M. W., & Lau, E. F. (2011) · Syntax and semantics · doi:10.1108/s0092-4563(2011)0000037009 · Cited by 88

The LAMBADA dataset: Word prediction requiring a broad discourse context

Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., & Fernandez, R. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1144 · Cited by 87

Unbounded Dependency Recovery for Parser Evaluation

Rimell, L., Clark, S. C., & Steedman, M. J. (2009) · Cited by 86

LSTM CCG Parsing

Lewis, M., Lee, K., & Zettlemoyer, L. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1026 · Cited by 72

The Microsoft Research Sentence Completion Challenge

Zweig, G., & Burges, C. J. C. (2011) · Cited by 71

Normal and pathological development of subject–verb agreement in speech production: a study on French children

Franck, J., Cronel-Ohayon, S., Chillier, L., Frauenfelder, U. H., Hamann, C., Rizzi, L., & Zesiger, P. (2003) · Journal of Neurolinguistics · doi:10.1016/s0911-6044(03)00057-5 · Cited by 71

Large-Scale Syntactic Language Modeling with Treelets

Pauls, A., & Klein, D. (2012) · Cited by 53

Evaluation of Dependency Parsers on Unbounded Dependencies

Nivre, J., Rimell, L., McDonald, R., & Gómez‐Rodríguez, C. (2010) · Cited by 53

Assessing Composition in Sentence Vector RepresentationsPreprint

Ettinger, A., Elgohary, A., Phillips, C., & Resnik, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03992 · Cited by 46

How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs

Sennrich, R. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2060 · Cited by 40

Unbounded dependency recovery for parser evaluation

Rimell, L., Clark, S. C., & Steedman, M. J. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699571.1699619 · Cited by 36

Parser Evaluation over Local and Non-Local Deep Dependencies in a Large Corpus

Bender, E. M., Flickinger, D., Oepen, S., & Zhang, Y. (2012) · Cited by 31

Exploring the Syntactic Abilities of RNNs with Multi-task Learning

Enguehard, É., Goldberg, Y., & Linzen, T. (2017) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k17-1003 · Cited by 26

Judging Grammaticality with Tree Substitution Grammar Derivations

Post, M. (2011) · Cited by 23

Can recurrent neural networks learn natural language grammars?

Lawrence, S., Giles, C. L., & Fong, S. (2002) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1996.549183 · Cited by 15

The Emergence of Grammaticality in Connectionist Networks

(2013) · Psychology Press eBooks · doi:10.4324/9781410602367-9 · Cited by 11

Neural Network Acceptability Judgments

Warstadt, A., Singh, A., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1805.12471

Language Models as Knowledge Bases?

Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834

What Does BERT Look at? An Analysis of BERT’s Attention

Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4828 · Cited by 1,953

BERT Rediscovers the Classical NLP Pipeline

Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307

Neural Network Acceptability Judgments

Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 1,215

Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944

An LSTM Adaptation Study of (Un)grammaticality

Chowdhury, S. A., & Zamparelli, R. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4821 · Cited by 7

Cross-Linguistic Syntactic Evaluation of Word Prediction Models

Mueller, A., Nicolai, G., Petrou-Zeniou, P., Talmina, N., & Linzen, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.490 · Cited by 52

Refining Targeted Syntactic Evaluation of Language Models

Newman, B., Ang, K.-S., Gong, J., & Hewitt, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.290 · Cited by 38

Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies

Linzen, T., Emmanuel Dupoux, A., & Goldberg, Y. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00115 · Cited by 820

Language Modeling with Syntactic and Semantic Representation for Sentence Acceptability Predictions

Ek, A., Bernardy, J.-P., & Lappin, S. (2019) · DSpace repository (University of Tartu) · Cited by 6