CoChat

Exploring the Syntactic Abilities of RNNs with Multi-task Learning

Open access
Enguehard, Émile, Goldberg, Yoav, Linzen, Tal · 2017 · Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)
Cited by 24

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Abstract

Recent work has explored the syntactic abilities of RNNs using the subject-verb agreement task, which diagnoses sensitivity to sentence structure.RNNs performed this task well in common cases, but faltered in complex sentences (Linzen et al., 2016).We test whether these errors are due to inherent limitations of the architecture or to the relatively indirect supervision provided by most agreement dependencies in a corpus.We trained a single RNN to perform both the agreement task and an additional task, either CCG supertagging or language modeling.Multitask training led to significantly lower error rates, in particular on complex sentences, suggesting that RNNs have the ability to evolve more sophisticated syntactic representations than shown before.We also show that easily available agreement training data can improve performance on other syntactic tasks, in particular when only a limited amount of training data is available for those tasks.The multi-task paradigm can also be leveraged to inject grammatical knowledge into language models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/k17-1003

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

  • Targeted Syntactic Evaluation of Language Models

    Marvin, R., & Linzen, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1151 · Cited by 334

    “We simply sum the two objective functions with equal weights (Enguehard et al., 2017).”
    BackgroundMethodSelf-citation
  • An Analysis of Encoder Representations in Transformer-Based Machine Translation

    Raganato, A., & Tiedemann, J. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5431 · Cited by 277

    “Nowadays, given its popularity, recurrent neural networks are the most evaluated networks, mainly investigated on the structures and linguistic properties they are encoding (Linzen et al., 2016; Enguehard et al., 2017; Kuncoro et al., 2017; Gulordava et al., 2018).”
    Background
  • Quantity doesn’t buy quality syntax with neural language models

    van Schijndel, M., Mueller, A., & Linzen, T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1592 · Cited by 65

    “Overall, this body of results points to the limited data efficiency of standard RNNs and Transformers, and indicates that learning syntax from realistic amounts of data—in particular the amount of data available to humans when they learn language—may require syntactically structured architectures or explicit syntactic…”
    Self-citation
  • Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages

    Ravfogel, S., Goldberg, Y., & Linzen, T. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1356 · Cited by 39

    “…supervision signals from the prediction of both arguments lead to more robust abstract syntactic representations that transfer across the two tasks (Enguehard et al., 2017); for example, the model may be better able to identify the head of a noun phrase, regardless of whether it is the subject or…”
    BackgroundSelf-citation

See all 24

Metadata

Authors
Enguehard, É., Goldberg, Y., & Linzen, T.
Venue
Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)
Published
2017
Pages
3-14
OpenAlex
W2625014264
Cited by
24
References
5
FWCI
2.29
Type
conference paper
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.