The DOI matches its Crossref record; 15 of the 16 references it cites were found in a registry or an index.
Deep multi-task learning with low level tasks supervised at lower layers
Open accessAbstract
In all previous work on deep multi-task learning we are aware of, all task supervisions are on the same (outermost) layer. We present a multi-task learning architecture with deep bi-directional RNNs, where different tasks supervision can happen at different layers. We present experiments in syntactic chunking and CCG supertagging, coupled with the additional task of POS-tagging. We show that it is consistently better to have POS supervision at the innermost rather than the outermost layer. We argue that this is because “lowlevel” tasks are better kept at the lower layers, enabling the higher-level tasks to make use of the shared representation of the lower-level tasks. Finally, we also show how this architecture can be used for domain adaptation.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, unlike WSD, accuracies using the first biLM layer are higher than the top layer, consistent with results from deep biL-STMs in multi-task training (Søgaard and Goldberg, 2016;Hashimoto et al., 2017) and MT (Belinkov et al., 2017).”
From Deep Contextualized Word Representations (2018) · cites this paper as Søgaard and Goldberg, 2016
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172
Natural Language Processing (almost) from ScratchPreprint
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 6,923
A Primer on Neural Network Models for Natural Language Processing
Goldberg, Y. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4992 · Cited by 1,316
Self-citationIntroduction to the CoNLL-2000 shared task
Sang, E. T. K., & Buchholz, S. (2000) · Conference on Computational Natural Language Learning · doi:10.3115/1117601.1117631 · Cited by 713
The CoNLL 2007 Shared Task on Dependency Parsing
Nivre, J., Hall, J., Kübler, S., McDonald, R., Nilsson, J., Riedel, S., & Yüret, D. (2007) · Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) · Cited by 663
Multi-task Sequence to Sequence LearningPreprint
Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, Ł. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06114 · Cited by 606
Opinion Mining with Deep Recurrent Neural Networks
Irsoy, O., & Cardie, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1080 · Cited by 384 · cited 3×
“We experiment with a multi-task learning (MTL) architecture based on deep bi-directional recurrent neural networks (bi-RNNs) (Schuster and Paliwal, 1997; Irsoy and Cardie, 2014).”
MethodRotating your face using multi-task deep neural network
Yim, J., Jung, H., Yoo, B., Choi, C., Park, D., & Kim, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298667 · Cited by 279
Cited by
Sorted by citations
Items 401–449 of 449
On constituent chunking for Turkish
Aslan, Ö., Günal, S., & Dinçer, B. T. (2018) · Information Processing & Management · doi:10.1016/j.ipm.2018.05.004
KU-MTL at SemEval-2018 Task 1: Multi-task Identification of Affect in Tweets
Nyegaard-Signori, T., Helms, C. V., Bjerva, J., & Augenstein, I. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1058
The Perfect Recipe: Add SUGAR, Add Data
Magnolini, S., Balaraman, V., Guerini, M., & Magnini, B. (2018) · Accademia University Press eBooks · doi:10.4000/books.aaccademia.4862
Transfer and Multi-Task Learning for Noun-Noun Compound InterpretationPreprint
Fares, M., Oepen, S., & Velldal, E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.06748
Preface to the Evalita 2018 Proceedings
Caselli, T., Novielli, N., Patti, V., & Rosso, P. (2018) · Accademia University Press eBooks · doi:10.4000/books.aaccademia.4427
A Two-Stage Deep Neural Network for Sequence Labeling
Tan, Y., Yang, L., Niu, S., Zhu, H., & Zhang, Y. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-24265-7_12
Transductive Auxiliary Task Self-Training for Neural Multi-Task Models
Bjerva, J., Kann, K., & Augenstein, I. (2019) · Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo) · doi:10.18653/v1/d19-6128
Concurrent Parsing of Constituency and DependencyPreprint
Zhou, J., Zhang, S., & Zhao, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06379
Multi-Task Learning With Localized Generalization Error Model
Li, W., Zhu, Y., Wang, T., & Ng, W. W. Y. (2019) · International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc48188.2019.8949255
Multitask Learning for Darpa Lorelei’s Situation Frame Extraction Task
Singla, K., & Narayanan, S. S. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054710
Deep Learning With Sentiment Inference For Discourse-Oriented Opinion Analysis
Marasović, A. (2020) · heiDOK (Heidelberg University) · doi:10.11588/heidok.00027699
Transfer Learning in Dialogue Systems
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773.021
Cross-Domain Rating Prediction for Market Development
Otsuka, Y., Kato, M. P., Fujita, S., & Yoshikawa, M. (2020) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wiiat50758.2020.00074
On factuality in neural language models
Nadeem, M. (2021)
Joint Learning for Task-Oriented Representations in Natural Language Processing
橋本, 和. (2018) · Institutional Repositories DataBase (IRDB) · doi:10.15083/00078050
Heterogeneous Transfer Learning
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773.008
Transfer Learning in Urban Computing
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773.025
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773.026
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773.001
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773.009
Analyse automatique d’arguments et apprentissage multi-tâches : un cas d’étude
Mensonides, J.-C., Harispe, S., Montmain, J., & Thireau, V. (2022) · Revue Ouverte d Intelligence Artificielle · doi:10.5802/roia.29
Yang, Q., Zhang, Y., Dai, W., & Pan, S. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781139061773.027
Prohibited items detection on X-ray images with multi-task learning
Xu, L., Ji, Z., Zhou, X., & Shi, Y. (2022) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1145/3532213.3532298
Crude Oil-Related Events Extraction and Processing: A Transfer Learning ApproachPreprint
Lee, M., Soon, L.-K., & Siew, E.-G. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4183577
Noise processing and multitask learning for far‐field dialect classification
Wang, H., Wang, Y., Ma, Y., Qin, C., Zhang, K., Gao, L. C., & Ren, J. (2022) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.7274
Kim, J. Y., Liu, T., & Yacef, K. (2022) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3536220.3558034
Predicting Adverse Neonatal Outcomes for Preterm Neonates with Multi-Task Learning
Lin, J., Chen, J., Lyu, H., Khodak, I., Chhabra, D., Richardson, C. D., Prelipcean, I., Dylag, A. M., & Luo, J. (2023) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh60066.2023.00045
Towards Interpretable Multitask Learning for Splice Site and Translation Initiation Site PredictionPreprint
Kabanga, E. K., Messem, A. V., & Neve, W. D. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.16.562631
Xu, H. (2023) · Springer eBooks · doi:10.1007/978-981-99-5776-7_1
Multilingual Symptom Prediction by Simultaneous Learning Using BERT
Zhu, C., Pahari, N., & Shimada, K. (2023) · International Conference on Asian Language Processing · doi:10.1109/ialp61005.2023.10337285
Finding Adequate Additional Layer of Auxiliary Task in BERT-Based Multi-task Learning
Kitamura, T., & Suzuki, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78090-5_7
Armah, C., & Dehkharghani, R. (2024) · International Symposium on Innovative Approaches in Smart Technologies (ISAS) · doi:10.1109/isas64331.2024.10845733
Chen, K. (2024) · International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC) · doi:10.1109/icairc64177.2024.10900127
Unified Neural Lexical Analysis Via Two‐Stage Span Tagging
Xian, Y., Zhu, Y., Yu, Z., Huang, Y., Guo, J., & Xiang, Y. (2025) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.70015
Enhancing Low-Resource Machine Transliteration Through Three-Way Transfer Learning
Thamilselvan, R., Kalpana, T., Natesan, P., Prasath, S., & Balaji, B. S. (2025) · International Conference on Computing Technologies (ICOCT) · doi:10.1109/icoct64433.2025.11118514
Generalized Gradient Descent is a Hypergraph Functor
Hanks, T., Fairbanks, J., & Klawonn, M. (2025) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.429.12 · Cited by 1
Ashir, A. M., & Taha, M. A. (2025) · Intelligenza Artificiale · doi:10.1177/17248035251345711
Saji, A., Vignesh, R., & Juliet, S. (2025) · International Conference on NexGen Networks and Cybernetics (IC2NC) · doi:10.1109/ic2nc67409.2025.11376233
Lyu, Y., Feng, X., Jiang, P., Li, H., Lai, X., & Zhou, X. (2026) · Research Square · doi:10.21203/rs.3.rs-9200456/v1
Reconnaissance automatique de chaînes de coréférences en français parlé
Grobol, L. (2020) · theses.fr (ABES) · doi:10.70675/c9117547z94a7z4160zbdb1z14d4782be6d0
Neural Transfer Learning for Domain Adaptation in Natural Language Processing
Meftah, S. (2021) · theses.fr (ABES) · doi:10.70675/9111562bzfa37z4fd8zae6bz2e8ae8d5d403
Gugliotta, E. (2022) · theses.fr (ABES) · doi:10.70675/05855e1dzc98az4da9z98cfz9421797337ff
Facing Data Scarcity in Dialogues for Discourse Structure Discovery and Prediction
Li, C. (2023) · theses.fr (ABES) · doi:10.70675/c379d1f5z5324z4865z9c13z84c9508836f1
AdaMoE: Adaptive MoE Loss Framework for Introducing MMLM in Image Restoration
(72874), F. L., Hu, J., Wang, J., Xu, N., & Fu, Z. (2026) · Data Intelligence · doi:10.3724/2096-7004.di.2026.0009
Reentrancy Vulnerability Detection in Smart Contracts Based on Adversarial Examples
Tong, F., Zang, H., Jiang, F., Li, G. D., & Qiu, H. (2026) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2026.3696629
Metadata
- Authors
- Søgaard, A., & Goldberg, Y.
- Affiliations
- University of Copenhagen (DK)
- Venue
- Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Published
- 2016
- Pages
- 231-235
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2516255829
- Cited by
- 445
- References
- 16
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 12 · 2023: 32 · 2022: 35 · 2021: 42 · 2020: 87 · 2019: 95 · 2018: 86 · 2017: 37 · 2016: 2
- FWCI
- 39.51
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- multi-task deep learning, bidirectional RNN, part-of-speech tagging, shared representations, domain adaptation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
