The DOI matches its Crossref record; 15 of the 16 references it cites were found in a registry or an index.
Deep multi-task learning with low level tasks supervised at lower layers
Open accessAbstract
In all previous work on deep multi-task learning we are aware of, all task supervisions are on the same (outermost) layer. We present a multi-task learning architecture with deep bi-directional RNNs, where different tasks supervision can happen at different layers. We present experiments in syntactic chunking and CCG supertagging, coupled with the additional task of POS-tagging. We show that it is consistently better to have POS supervision at the innermost rather than the outermost layer. We argue that this is because “lowlevel” tasks are better kept at the lower layers, enabling the higher-level tasks to make use of the shared representation of the lower-level tasks. Finally, we also show how this architecture can be used for domain adaptation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, unlike WSD, accuracies using the first biLM layer are higher than the top layer, consistent with results from deep biL-STMs in multi-task training (Søgaard and Goldberg, 2016;Hashimoto et al., 2017) and MT (Belinkov et al., 2017).”
From Deep Contextualized Word Representations (2018) · cites this paper as Søgaard and Goldberg, 2016
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Natural Language Processing (almost) from ScratchPreprint
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 6,923
A Primer on Neural Network Models for Natural Language Processing
Goldberg, Y. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4992 · Cited by 1,316
Self-citationIntroduction to the CoNLL-2000 shared task
Sang, E. T. K., & Buchholz, S. (2000) · Conference on Computational Natural Language Learning · doi:10.3115/1117601.1117631 · Cited by 713
The CoNLL 2007 Shared Task on Dependency Parsing
Nivre, J., Hall, J., Kübler, S., McDonald, R., Nilsson, J., Riedel, S., & Yüret, D. (2007) · Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) · Cited by 663
Multi-task Sequence to Sequence LearningPreprint
Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, Ł. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06114 · Cited by 606
Opinion Mining with Deep Recurrent Neural Networks
Irsoy, O., & Cardie, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1080 · Cited by 384 · cited 3×
“We experiment with a multi-task learning (MTL) architecture based on deep bi-directional recurrent neural networks (bi-RNNs) (Schuster and Paliwal, 1997; Irsoy and Cardie, 2014).”
MethodRotating your face using multi-task deep neural network
Yim, J., Jung, H., Yoo, B., Choi, C., Park, D., & Kim, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298667 · Cited by 279
Improving multiview face detection with multi-task deep convolutional neural networks
Zhang, C., & Zhang, Z. (2014) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2014.6835990 · Cited by 210
Voting Between Multiple Data Representations for Text Chunking
Shen, H., & Sarkar, A. (2005) · Lecture notes in computer science · doi:10.1007/11424918_40 · Cited by 57
CCG Supertagging with a Recurrent Neural Network
Xu, W., Auli, M., & Clark, S. C. (2015) · doi:10.3115/v1/p15-2041 · Cited by 53
Natural Language Understanding with Distributed RepresentationPreprint
Cho, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.07916 · Cited by 52
Zhu, X., Gibson, B. R., & Rogers, T. T. (2009) · Cited by 19
Cited by
Sorted by citations
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
An Overview of Multi-Task Learning in Deep Neural NetworksPreprint
Ruder, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05098 · Cited by 3,827
A Primer on Neural Network Models for Natural Language Processing
Goldberg, Y. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4992 · Cited by 1,316
Self-citationContextual String Embeddings for Sequence Labeling
Akbik, A., Blythe, D. A. J., & Vollgraf, R. (2018) · International Conference on Computational Linguistics · Cited by 1,086
A survey of unsupervised deep domain adaptation
Wilson, G., & Cook, D. J. (2020) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3400066 · Cited by 987
Metadata
- Authors
- Søgaard, A., & Goldberg, Y.
- Affiliations
- University of Copenhagen (DK)
- Venue
- Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2016
- Pages
- 231-235
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2516255829
- Cited by
- 445
- References
- 16
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 12 · 2023: 32 · 2022: 35 · 2021: 42 · 2020: 87 · 2019: 95 · 2018: 86 · 2017: 37 · 2016: 2
- FWCI
- 39.51
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- multi-task deep learning, bidirectional RNN, part-of-speech tagging, shared representations, domain adaptation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
