Cited by 449Natural Language Processing TechniquesDomain Adaptation and Few-Shot LearningTopic Modeling

Deep multi-task learning with low level tasks supervised at lower layers

Open access
Søgaard, Anders, Goldberg, Yoav·2016·Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

Abstract

In all previous work on deep multi-task learning we are aware of, all task supervisions are on the same (outermost) layer. We present a multi-task learning architecture with deep bi-directional RNNs, where different tasks supervision can happen at different layers. We present experiments in syntactic chunking and CCG supertagging, coupled with the additional task of POS-tagging. We show that it is consistently better to have POS supervision at the innermost rather than the outermost layer. We argue that this is because “lowlevel” tasks are better kept at the lower layers, enabling the higher-level tasks to make use of the shared representation of the lower-level tasks. Finally, we also show how this architecture can be used for domain adaptation.

Integrity signals

Registry

Crossref · 10.18653/v1/p16-2038

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For POS tagging, accuracies using the first biLM layer are higher than the top layer.

    “However, unlike WSD, accuracies using the first biLM layer are higher than the top layer, consistent with results from deep biL-STMs in multi-task training (Søgaard and Goldberg, 2016;Hashimoto et al., 2017) and MT (Belinkov et al., 2017).”

    From Deep Contextualized Word Representations (2018) · cites this paper as Søgaard and Goldberg, 2016

Cites

Flagged first, then by citations

1 of the 16 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Søgaard, A., & Goldberg, Y.
Affiliations
University of Copenhagen (DK)
Venue
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Published
2016
Pages
231-235
ISSN
0736-587X (ISSN-L)
OpenAlex
W2516255829
Cited by
445
References
16
Citations by year
2026: 5 · 2025: 11 · 2024: 12 · 2023: 32 · 2022: 35 · 2021: 42 · 2020: 87 · 2019: 95 · 2018: 86 · 2017: 37 · 2016: 2 · 2015: 1
FWCI
39.51
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion)
  • Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
multi-task deep learning, bidirectional RNN, part-of-speech tagging, shared representations, domain adaptation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review