Exploring the Syntactic Abilities of RNNs with Multi-task Learning
Open accessThe DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Recent work has explored the syntactic abilities of RNNs using the subject-verb agreement task, which diagnoses sensitivity to sentence structure.RNNs performed this task well in common cases, but faltered in complex sentences (Linzen et al., 2016).We test whether these errors are due to inherent limitations of the architecture or to the relatively indirect supervision provided by most agreement dependencies in a corpus.We trained a single RNN to perform both the agreement task and an additional task, either CCG supertagging or language modeling.Multitask training led to significantly lower error rates, in particular on complex sentences, suggesting that RNNs have the ability to evolve more sophisticated syntactic representations than shown before.We also show that easily available agreement training data can improve performance on other syntactic tasks, in particular when only a limited amount of training data is available for those tasks.The multi-task paradigm can also be leveraged to inject grammatical knowledge into language models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963 · cited 1×
“The effectiveness of RNNs in language modeling, in particular LSTMs, has been demonstrated in numerous studies (Mikolov et al., 2010; Sundermeyer et al., 2012; Jozefowicz et al., 2016).”
BackgroundDistributed representations, simple recurrent networks, and grammatical structure
Elman, J. L. (1991) · Machine Learning · doi:10.1007/bf00114844 · Cited by 1,354
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
Hashimoto, K., xiong, C., Tsuruoka, Y., & Socher, R. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1206 · Cited by 535 · cited 1×
“This logic has been applied to various NLP tasks, with generally encouraging results (Collobert and Weston, 2008; Hashimoto et al., 2016; Søgaard and Goldberg, 2016; Martínez Alonso and Plank, 2017; Bingel and Søgaard, 2017).”
MethodRecurrent Neural Network Grammars
Dyer, C., Kuncoro, A., Ballesteros, M., & Smith, N. A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1024 · Cited by 517 · cited 1×
“These deficiencies could be due to fundamental limitations of the architecture, which can only be addressed by switching to more expressive architectures (Socher, 2014; Grefenstette et al., 2015; Dyer et al., 2016).”
BackgroundDeep multi-task learning with low level tasks supervised at lower layers
Søgaard, A., & Goldberg, Y. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p16-2038 · Cited by 445 · cited 1×
“This logic has been applied to various NLP tasks, with generally encouraging results (Collobert and Weston, 2008; Hashimoto et al., 2016; Søgaard and Goldberg, 2016; Martínez Alonso and Plank, 2017; Bingel and Søgaard, 2017).”
MethodSelf-citation
Cited by
Targeted Syntactic Evaluation of Language Models
Marvin, R., & Linzen, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1151 · Cited by 334
“We simply sum the two objective functions with equal weights (Enguehard et al., 2017).”
BackgroundMethodSelf-citationAn Analysis of Encoder Representations in Transformer-Based Machine Translation
Raganato, A., & Tiedemann, J. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5431 · Cited by 277
“Nowadays, given its popularity, recurrent neural networks are the most evaluated networks, mainly investigated on the structures and linguistic properties they are encoding (Linzen et al., 2016; Enguehard et al., 2017; Kuncoro et al., 2017; Gulordava et al., 2018).”
BackgroundQuantity doesn’t buy quality syntax with neural language models
van Schijndel, M., Mueller, A., & Linzen, T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1592 · Cited by 65
“Overall, this body of results points to the limited data efficiency of standard RNNs and Transformers, and indicates that learning syntax from realistic amounts of data—in particular the amount of data available to humans when they learn language—may require syntactically structured architectures or explicit syntactic…”
Self-citationStudying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages
Ravfogel, S., Goldberg, Y., & Linzen, T. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1356 · Cited by 39
“…supervision signals from the prediction of both arguments lead to more robust abstract syntactic representations that transfer across the two tasks (Enguehard et al., 2017); for example, the model may be better able to identify the head of a noun phrase, regardless of whether it is the subject or…”
BackgroundSelf-citation
Metadata
- Authors
- Enguehard, É., Goldberg, Y., & Linzen, T.
- Venue
- Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)
- Published
- 2017
- Pages
- 3-14
- OpenAlex
- W2625014264
- Cited by
- 24
- References
- 5
- FWCI
- 2.29
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Exploring the Syntactic Abilities of RNNs with Multi-task Learning. https://app.cochat.ai/papers/10.18653/v1/k17-1003
- The paper
- Enguehard, É., Goldberg, Y., & Linzen, T. (2017). Exploring the Syntactic Abilities of RNNs with Multi-task Learning. Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017). https://doi.org/10.18653/v1/k17-1003
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.