CoChat
Sign Up
Cited by 535Natural Language Processing TechniquesTopic ModelingDomain Adaptation and Few-Shot Learning

A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

Open access
Hashimoto, Kazuma, xiong, caiming, Tsuruoka, Yoshimasa, Socher, Richard·2017·Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/d17-1206
Open this paper in CoChat — free accountOpen PDF

Abstract

Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks.Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model.We introduce a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks.Higher layers include shortcut connections to lower-level task predictions to reflect linguistic hierarchies.We use a simple regularization term to allow for optimizing all model weights to improve one task's loss without exhibiting catastrophic interference of the other tasks.Our single end-to-end model obtains state-of-the-art or competitive results on five different tasks from tagging, parsing, relatedness, and entailment tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d17-1206

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For POS tagging, accuracies using the first biLM layer are higher than the top layer.

    “However, unlike WSD, accuracies using the first biLM layer are higher than the top layer, consistent with results from deep biL-STMs in multi-task training (Søgaard and Goldberg, 2016;Hashimoto et al., 2017) and MT (Belinkov et al., 2017).”

    From Deep Contextualized Word Representations (2018) · cites this paper as Hashimoto et al., 2017

    Section 2.2 assigns POS tagging to the first bi-LSTM layer, and Section 6 says the model can be used as a POS tagger with only that layer. The paper does not compare POS accuracies from its first and top layers, so it supports the lower-layer connection but not the claimed accuracy advantage. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. Chris Alberti, David Weiss, Greg Coppola, and Slav Petrov. 2015. Improved Transition-Based Parsing and Tagging with Neural Networks. In Proceed- ings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1354-1359.
  2. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally Nor- malized Transition-Based Neural Networks. In Pro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 2442-2452.
  3. Giuseppe Attardi and Felice DellOrletta. 2008. Chunk- ing and Dependency Parsing. In Proceedings of LREC 2008 Workshop on Partial Parsing.
  4. Bernd Bohnet. 2010. Top Accuracy and Fast Depen- dency Parsing is not a Contradiction. In Proceed- ings of the 23rd International Conference on Com- putational Linguistics, pages 89-97.
  5. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, and Hui Jiang. 2016. Enhancing and Combining Se- quential and Tree LSTM for Natural Language In- ference. arXiv, cs.CL 1609.06038.
  6. Do Kook Choe and Eugene Charniak. 2016. Parsing as Language Modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2331-2336.
  7. Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen nad Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural Language Processing (Al- most) from Scratch. Journal of Machine Learning Research, 12:2493-2537.
  8. Timothy Dozat and Christopher D. Manning. 2017. Deep Biaffine Attention for Neural Dependency Parsing. In Proceedings of the 5th International Conference on Learning Representations.
  9. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition- Based Dependency Parsing with Stack Long Short- Term Memory. In Proceedings of the 53rd Annual Meeting of the Association for Computational Lin- guistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 334-343.
  10. Jason Eisner. 1996. Efficient Normal-Form Parsing for Combinatory Categorial Grammar. In Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, pages 79-86.
  11. Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-Sequence Attentional Neu- ral Machine Translation. In Proceedings of the 54th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 823-833.
  12. Jonathan Godwin, Pontus Stenetorp, and Sebastian Riedel. 2016. Deep Semi-Supervised Learning with Linguistically Motivated Sequence Labeling Task Hierarchies. arXiv, cs.CL 1612.09113.
  13. Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. 2013. Max- out Networks. In Proceedings of The 30th Inter- national Conference on Machine Learning, pages 1319-1327.
  14. Alex Graves and Jurgen Schmidhuber. 2005. Frame- wise Phoneme Classification with Bidirectional LSTM and Other Neural Network Architectures. Neural Networks, 18(5):602-610.
  15. Kazuma Hashimoto and Yoshimasa Tsuruoka. 2017. Neural Machine Translation with Source-Side La- tent Graph Parsing. In Proceedings of the 2017 Con- ference on Empirical Methods in Natural Language Processing. To appear.
  16. Taku Kudo and Yuji Matsumoto. 2001. Chunking with Support Vector Machines. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics.
  17. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask Me Anything: Dynamic Memory Networks for Natural Language Processing. In Proceedings of The 33rd International Conference on Machine Learning, pages 1378-1387.
  18. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What Do Recurrent Neural Network Grammars Learn About Syntax? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 1249-1258.
  19. Alice Lai and Julia Hockenmaier. 2014. Illinois-LH: A Denotational and Distributional Approach to Se- mantics. In Proceedings of the 8th International Workshop on Semantic Evaluation, pages 329-334.
  20. Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent Convolutional Neural Networks for Text Classification. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 2267-2273.
  21. Zhizhong Li and Derek Hoiem. 2016. Learning with- out Forgetting. CoRR, abs/1606.09282. · arXiv
  22. Wang Ling, Chris Dyer, Alan W Black, Isabel Tran- coso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding Function in Form: Compositional Character Models for Open Vocab- ulary Word Representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1520-1530.
  23. Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task Se- quence to Sequence Learning. In Proceedings of the 4th International Conference on Learning Represen- tations.
  24. Xuezhe Ma and Eduard Hovy. 2016. End-to-end Se- quence Labeling via Bi-directional LSTM-CNNs- CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1064-1074.
  25. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raf- faella Bernardi, Stefano Menini, and Roberto Zam- parelli. 2014. SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment. In Proceedings of the 8th In- ternational Workshop on Semantic Evaluation (Se- mEval 2014), pages 1-8.
  26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed Representa- tions of Words and Phrases and their Composition- ality. In Advances in Neural Information Processing Systems 26, pages 3111-3119.
  27. Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. 2016. Cross-stitch Networks for Multi-task Learning. CoRR, abs/1604.03539. · arXiv
  28. Makoto Miwa and Mohit Bansal. 2016. End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1105- 1116.
  29. Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Des- jardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive Neural Networks. CoRR, abs/1606.04671. · arXiv
  30. Anders Søgaard. 2011. Semi-supervised condensed nearest neighbor for part-of-speech tagging. In Pro- ceedings of the 49th Annual Meeting of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 48-52.
  31. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Re- search, 15:1929-1958.
  32. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Net- works. In Advances in Neural Information Process- ing Systems 27, pages 3104-3112.
  33. Jun Suzuki and Hideki Isozaki. 2008. Semi-Supervised Sequential Labeling and Segmentation Using Giga- Word Scale Unlabeled Data. In Proceedings of the 46th Annual Meeting of the Association for Com- putational Linguistics: Human Language Technolo- gies, pages 665-673.
  34. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved Semantic Representa- tions From Tree-Structured Long Short-Term Mem- ory Networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Lin- guistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1556-1566.
  35. Kristina Toutanova, Dan Klein, Christopher D Man- ning, and Yoram Singer. 2003. Feature-Rich Part- of-Speech Tagging with a Cyclic Dependency Net- work. In Proceedings of the 2003 Human Lan- guage Technology Conference of the North Ameri- can Chapter of the Association for Computational Linguistics, pages 173-180.
  36. Yoshimasa Tsuruoka, Yusuke Miyao, and Jun'ichi Kazama. 2011. Learning with Lookahead: Can History-Based Models Rival Globally Optimized Models? In Proceedings of the Fifteenth Confer- ence on Computational Natural Language Learning, pages 238-246.
  37. David Weiss, Chris Alberti, Michael Collins, and Slav Petrov. 2015. Structured Training for Neural Net- work Transition-Based Parsing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 323-333.
  38. Wenpeng Yin, Hinrich Schtze, Bing Xiang, and Bowen Zhou. 2016. ABCNN: Attention-Based Convolu- tional Neural Network for Modeling Sentence Pairs. Transactions of the Association for Computational Linguistics, 4:259-272.
  39. Xingxing Zhang, Jianpeng Cheng, and Mirella Lapata. 2017. Dependency Parsing as Head Selection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 665-676.
  40. Yuan Zhang and David Weiss. 2016. Stack- propagation: Improved Representation Learning for Syntax. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1557-1566.
  41. Yao Zhou, Cong Liu, and Yan Pan. 2016. Modelling Sentence Pairs with Tree-structured Attentive En- coder. In Proceedings of the 26th International Con- ference on Computational Linguistics, pages 2912- 2922.

Cited by

Sorted by citations

Authors
Hashimoto, K., xiong, C., Tsuruoka, Y., & Socher, R.
Affiliations
Salesforce (United States) (US)
ORCID
Venue
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2017
Pages
1923-1933
DOI
10.18653/v1/d17-1206
OpenAlex
W2556468274
Cited by
535
References
3
Citations by year
2026: 2 · 2025: 8 · 2024: 20 · 2023: 39 · 2022: 46 · 2021: 70 · 2020: 140 · 2019: 106 · 2018: 80 · 2017: 22 · 2016: 1 · 2015: 1
FWCI
43.41
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
joint multi-task learning, neural network growth, skip connections, linguistic hierarchies, catastrophic forgetting, regularization, morphology (linguistics), syntax, semantics, part-of-speech tagging, syntactic analysis, relatedness, logical consequence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review