The Natural Language Decathlon: Multitask Learning as Question Answering
Open accessAbstract
Deep learning has improved performance on many natural language processing (NLP) tasks individually. However, general NLP models cannot emerge within a paradigm that focuses on the particularities of a single metric, dataset, and task. We introduce the Natural Language Decathlon (decaNLP), a challenge that spans ten tasks: question answering, machine translation, summarization, natural language inference, sentiment analysis, semantic role labeling, zero-shot relation extraction, goal-oriented dialogue, semantic parsing, and commonsense pronoun resolution. We cast all tasks as question answering over a context. Furthermore, we present a new Multitask Question Answering Network (MQAN) jointly learns all tasks in decaNLP without any task-specific modules or parameters in the multitask setting. MQAN shows improvements in transfer learning for machine translation and named entity recognition, domain adaptation for sentiment analysis and natural language inference, and zero-shot capabilities for text classification. We demonstrate that the MQAN's multi-pointer-generator decoder is key to this success and performance further improves with an anti-curriculum training strategy. Though designed for decaNLP, MQAN also achieves state of the art results on the WikiSQL semantic parsing task in the single-task setting. We also release code for procuring and processing data, training and evaluating models, and reproducing all experiments for decaNLP.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In-context learning is a key mechanism for foundation model programming.
“In-context learning (McCann et al. 2018; Radford et al. 2018; Brown et al. 2020) is a key mechanism for foundation model programming.”
From DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023) · cites this paper as McCann et al. 2018
The Introduction (p. 1) says natural-language questions can describe tasks and allow a model to generalize to new tasks; the Analysis (p. 8) reports zero-shot classification. These findings relate to using input context to direct a model, but the paper does not identify this as in-context learning or establish it as a key mechanism for foundation model programming. (LLM, conf=0.75)
Experimental — accuracy not yet measuredThe Transformer was initially shown to be effective for machine translation, but has subsequently been used in a wide variety of NLP settings.
“The Transformer was initially shown to be effective for machine translation, but it has subsequently been used in a wide variety of NLP settings (Radford et al., 2018; Devlin et al., 2018; McCann et al., 2018; Yu et al., 2018).”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as McCann et al., 2018
Further Related Work (p. 17) says attention-only models were effective for machine translation. The MQAN section (pp. 5–6) describes using Transformer-derived self-attention layers across decaNLP tasks, but the model also uses recurrent and coattentive layers; it does not establish that the Transformer architecture itself was used across those settings. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe finding that a multi-task model is outperformed by separate models trained on each individual task has previously been observed by Arivazhagan et al. (2019) and McCann et al. (2018).
“The finding that a multi-task model is outperformed by separate models trained on each individual task has previously been observed e.g. by Arivazhagan et al. (2019) and McCann et al. (2018)”
From Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019) · cites this paper as McCann et al. (2018)
Appendix D (pp. 22–23) reports that fully joint training struggles to reach single-task performance on several tasks and that a gap remains. But the Introduction (p. 2) says the multitask MQAN, with the right training strategy, achieves performance comparable to ten separately trained MQANs; Related Work (p. 9) says it performs nearly as well or better on each task. The claim overstates the paper’s finding as a general result that separate models outperform the multitask model. (LLM, conf=0.86)
Experimental — accuracy not yet measuredReferences [61,83] are earlier works that attempt to unify multiple NLP tasks into a QA framework.
“References [61,83] are earlier works that attempt to unify multiple NLP tasks into a QA framework.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [83]
In the majority of previous work, the prompting function consists of applying a template with an input slot and an answer slot and filling the input slot with the input text.
“In the majority of previous work [61,83,105,117], this function consists of a two step process: (1) Apply a template, which is a textual string that has two slots: an input slot [X] for input x and an answer slot [Z] for an intermediate generated answer text z that will later be mapped into y.(2) Fill slot [X] with the input text x.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [83]
The Introduction (p. 1) says this paper frames tasks as question answering using a context, question, and answer; Section 3 (p. 4) describes those three sequences. Our copy does not describe a two-slot template, filling an input slot, or this procedure being used in the majority of previous work (searched for templates, slots, and task-specific questions). (LLM, conf=0.70)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 106 · Cited by 448
Part of this paper’s reference list could not be split into single entries.
Research works printed but not linked: 2
- [2]2. H.-Y. Huang, C. Zhu, Y. Shen, and W. Chen. Fusionnet: Fusing via fully-aware attention with application to machine comprehension. CoRR , abs/1711.07341, 2017. · arXiv
- [14]14. T.-H. Wen, D. Vandyke, N. Mrksic, M. Gasic, L. M. Rojas-Barahona, P.-H. Su, S. Ultes, and S. Young. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562 , 2016. - A. Williams, N. Nangia, and S. R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. CoRR , abs/1704.05426, 2017. - T. Winograd. Understanding natural language. Cognitive psychology , 3(1):1-191, 1972. - Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. S. Corrado, M. Hughes, and J. Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR , abs/1609.08144, 2016. - C. Xiong, S. Merity, and R. Socher. Dynamic Memory Networks for Visual and Textual Question Answering. In ICML , 2016. - C. Xiong, V. Zhong, and R. Socher. Dynamic coattention networks for question answering. ICLR , 2017. - C. Xiong, V. Zhong, and R. Socher. Dcn+: Mixed objective and deep residual coattention for question answering. ICLR , 2018. - A. W. Yu, D. Dohan, M.-T. Luong, R. Zhao, K. Chen, M. Norouzi, and Q. V. Le. Qanet: Combining local convolution with global self-attention for reading compre- hension. In ICLR , 2018a. - H. Yu and T. Munkhdalai. Neural semantic encoders. Proceedings of the conference. Association for Computational Linguistics. Meeting , 1:397-407, 2017a. - H. Yu and T. Munkhdalai. Neural tree indexers for text understanding. ACL , 2017b. - T. Yu, Z. Li, Z. Zhang, R. Zhang, and D. Radev. Typesql: Knowledge-based type-aware neural text-to-sql generation. arXiv preprint arXiv:1804.09769 , 2018b. - Y. Yu, W. Zhang, K. S. Hasan, M. Yu, B. Xiang, and B. Zhou. End-to-end reading comprehension with dynamic answer chunk ranking. CoRR , abs/1610.09996, 2016. - V. Zhong, C. Xiong, and R. Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR , abs/1709.00103, 2017. - V. Zhong, C. Xiong, and R. Socher. Global-locally self-attentive dialogue state tracker. arXiv preprint arXiv:1805.09655 , 2018. - J. Zhou and W. Xu. End-to-end learning of semantic role labeling using recurrent neural networks. In ACL , 2015. - B. Zoph, D. Yuret, J. May, and K. Knight. Transfer learning for low-resource neural machine translation. In EMNLP , 2016. · arXiv
- Authors
- McCann, B., Keskar, N. S., Xiong, C., & Socher, R.
- Affiliations
- Salesforce (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1806.08730
- OpenAlex
- W2809324505
- Cited by
- 448
- References
- 106
- Citations by year
- 2026: 1 · 2025: 11 · 2024: 23 · 2023: 40 · 2022: 79 · 2021: 106 · 2020: 97 · 2019: 77 · 2018: 14
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multi-task learning, question answering, transfer learning, domain adaptation, zero-shot learning, semantic parsing, named entity recognition, sentiment analysis, natural language inference, machine translation, text summarization, semantic role labeling, relation extraction, task-oriented dialogue, WikiSQL
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
