Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems
Open accessAbstract
Task-oriented dialogue systems use four connected modules, namely, Natural Language Understanding (NLU), a Dialogue State Tracking (DST), Dialogue Policy (DP) and Natural Language Generation (NLG). A research challenge is to learn each module with the least amount of samples (i.e., few-shots) given the high cost related to the data collection. The most common and effective technique to solve this problem is transfer learning, where large language models, either pre-trained on text or task-specific data, are fine-tuned on the few samples. These methods require fine-tuning steps and a set of parameters for each task. Differently, language models, such as GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020), allow few-shot learning by priming the model with few examples. In this paper, we evaluate the priming few-shot ability of language models in the NLU, DST, DP and NLG tasks. Importantly, we highlight the current limitations of this approach, and we discuss the possible implication for future work.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Pre-training models on a large corpus results in the model memorizing knowledge in its parameters.
“Pre-training of models on a large corpus is known to result in the model memorizing knowledge in its parameters [169, 201, 221].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [169]
The Introduction (p. 1) says that large models are pre-trained on large corpora before task-specific fine-tuning. Neither that passage nor the discussion of unchanged parameters during few-shot priming (Introduction, p. 2) says that pre-training causes models to memorize knowledge in their parameters. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · see citing works
Big Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · see citing works
Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextPreprint
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2023.7 · Cited by 860 · see citing works
Coucke, A., Saade, A., Ball, A., Bluche, T., Caulier, A., Leroy, D., Doumouro, C., Gisselbrecht, T., Caltagirone, F., Lavril, T., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10190 · Cited by 690 · see citing works
Universal Adversarial Triggers for Attacking and Analyzing NLP
Wallace, E., Feng, S., Kandpal, N., Gardner, M., & Singh, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1221 · Cited by 642 · see citing works
Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
Wu, C.-S., Madotto, A., Hosseini-Asl, E., Xiong, C., Socher, R., & Fung, P. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1078 · Cited by 423 · see citing works
Self-citationZang, X., Rastogi, A., & Chen, J. (2020) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2020.nlp4convai-1.13 · Cited by 195 · see citing works
Personalizing Dialogue Agents via Meta-Learning
Madotto, A., Lin, Z., Wu, C.-S., & Fung, P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1542 · Cited by 183 · see citing works
Self-citationFew-shot Natural Language Generation for Task-Oriented Dialog
Peng, B., Zhu, C., Li, C., Li, X., Li, J., Zeng, M., & Gao, J. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.17 · Cited by 162 · see citing works
Domain Adaptive Dialog Generation via Meta Learning
Qian, K., & Yu, Z. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1253 · Cited by 134 · see citing works
Coach: A Coarse-to-Fine Approach for Cross-domain Slot Filling
Liu, Z., Winata, G. I., Xu, P., & Fung, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.3 · Cited by 102 · see citing works
Self-citationSOLOIST: Building Task Bots at Scale with Transfer Learning and Machine TeachingPreprint
Peng, B., Li, C., Li, J., Shayandeh, S., Lidén, L., & Gao, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05298 · Cited by 96 · see citing works
Meta-Learning for Low-resource Natural Language Generation in Task-oriented Dialogue Systems
Mi, F., Huang, M., Zhang, J., & Faltings, B. V. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/437 · Cited by 89 · see citing works
ToD-BERT: Pre-trained Natural Language Understanding for Task-Oriented DialoguesPreprint
Wu, C.-S., Hoi, S. C. H., Socher, R., & Xiong, C. (2020) · arXiv (Cornell University) · Cited by 47 · see citing works
Template Guided Text Generation for Task-Oriented DialoguePreprint
Kale, M., & Rastogi, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.15006 · Cited by 16 · see citing works
Few-Shot Natural Language Generation by Rewriting TemplatesPreprint
Kale, M., & Rastogi, A. (2020) · arXiv (Cornell University) · Cited by 13 · see citing works
Meta Dialogue Policy LearningPreprint
Xu, Y., Zhu, C., Peng, B., & Zeng, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02588 · Cited by 6 · see citing works
Flagged first, then by citations
1 of the 22 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, I˜ nigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a largescale multi-domain wizard-of-oz dataset for taskoriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 5016-5026.
- Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi, Peter Ku, Anuj Kumar Goyal, Sanchit Agarwal, Shuyang Gao, and Dilek Hakkani-Tur. 2019. Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. arXiv preprint arXiv:1907.01669 . · arXiv
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8):9.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Self-citationBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationTemplate-free Prompt Tuning for Few-shot NER
Ma, R., Zhou, X., Gui, T., Tan, Y., Li, L., Zhang, Q., & Huang, X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.420 · Cited by 136 · see citing works
Language Models are Few-shot Multilingual Learners
Winata, G. I., Madotto, A., Lin, Z., Liu, R., Yosinski, J., & Fung, P. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.1 · Cited by 115 · see citing works
Self-citationTowards Few-shot Fact-Checking via Perplexity
Lee, N., Bang, Y., Madotto, A., & Fung, P. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.158 · Cited by 79 · see citing works
Self-citationKnowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
Cao, B., Lin, H., Han, X., Sun, L., Yan, L., Liao, M., Xue, T., & Xu, J. (2021) · doi:10.18653/v1/2021.acl-long.146 · Cited by 90 · see citing works
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems
Madotto, A., Cahyawijaya, S., Winata, G. I., Xu, Y., Liu, Z., Lin, Z., & Fung, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.215 · Cited by 63 · see citing works
Self-citationLeveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue StateTracking
Lin, Z., Liu, B., Moon, S., Crook, P., Zhou, Z., Wang, Z., Yu, Z., Madotto, A., Cho, E., & Subba, R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.448 · Cited by 66 · see citing works
Self-citationFew-Shot Bot: Prompt-Based Learning for Dialogue SystemsPreprint
Madotto, A., Lin, Z., Winata, G. I., & Fung, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08118 · Cited by 66 · see citing works
Self-citationAre Large Language Models All You Need for Task-Oriented Dialogue?
Hudeček, V., & Dušek, O. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.21 · Cited by 76 · see citing works
Language Model is all You Need: Natural Language Understanding as Question Answering
Namazifar, M., Papangelis, A., Tür, G., & Hakkani-Tür, D. Z. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413810 · Cited by 40 · see citing works
Synthesizing Speech from Intracranial Depth Electrodes using an Encoder-Decoder Framework
Köhler, J., Ottenhoff, M. C., Goulis, S., Angrick, M., Colon, A., Wagner, L., Tousseyn, S., Kubben, P., & Herff, C. (2022) · Neurons Behavior Data analysis and Theory · doi:10.51628/001c.57524 · Cited by 29 · see citing works
Show, Don’t Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue
Gupta, R., Lee, H., Zhao, J., Cao, Y. J., Rastogi, A., & Wu, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.336 · Cited by 23 · see citing works
TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems
Byrne, B., Krishnamoorthi, K., Ganesh, S., & Kale, M. (2021) · doi:10.18653/v1/2021.acl-long.55 · Cited by 19 · see citing works
An Overview on Language Models: Recent Developments and OutlookPreprint
Wei, C., Wang, Y.-C., Wang, B., & Kuo, C. ‐. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.05759 · Cited by 15 · see citing works
A Unified Dialogue User Simulator for Few-shot Data Augmentation
Wan, D., Zhang, Z., Zhu, Q., Liao, L., & Huang, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.277 · Cited by 13 · see citing works
Communicating Natural Programs to Humans and Machines
Acquaviva, S., Pu, Y., Kryven, M., Sechopoulos, T., Wong, C., Ecanow, G., Nye, M., Tessler, M. H., & Tenenbaum, J. B. (2022) · neural information processing systems · doi:10.52202/068431-0270 · Cited by 12 · see citing works
CrossCodeBench: Benchmarking Cross-Task Generalization of Source Code Models
Niu, C., Li, C., Ng, V., & Luo, B. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00055 · Cited by 11 · see citing works
Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems
Sun, W., Guo, S., Zhang, S., Ren, P., Chen, Z., de Rijke, M., & Ren, Z. (2023) · ACM Transactions on Information Systems · doi:10.1145/3596510 · Cited by 11 · see citing works
Consistent Game Content Creation via Function Calling for Large Language Models
Gallotta, R., Liapis, A., & Yannakakis, G. N. (2024) · IEEE Conference on Games (CoG) · doi:10.1109/cog60054.2024.10645599 · Cited by 10 · see citing works
Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
Cao, B., Lin, H., Han, X., Sun, L., Yan, L., Liao, M., Xue, T., & Xu, J. (2021) · arXiv · doi:10.48550/arxiv.2106.09231 · Cited by 10 · see citing works
GenSF: Simultaneous Adaptation of Generative Pre-trained Models and Slot Filling
Mehri, S., & Eskénazi, M. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.51 · Cited by 9 · see citing works
Morandé, S., & Amini, M. (2023) · Qeios · doi:10.32388/0qi028 · Cited by 7 · see citing works
Knowledge-Grounded Conversational Data Augmentation with Generative Conversational Networks
Lin, Y. T., Papangelis, A., Kim, S., & Hakkani-Tür, D. Z. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.3 · Cited by 7 · see citing works
N-Shot Learning for Augmenting Task-Oriented Dialogue State Tracking
Aksu, I., Liu, Z., Kan, M., & Chen, N. F. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.131 · Cited by 6 · see citing works
Self-citationPatFig: Generating Short and Long Captions for Patent Figures
Aubakirova, D., Gerdes, K., & Liu, L. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00305 · Cited by 5 · see citing works
Prompter: Zero-shot Adaptive Prefixes for Dialogue State Tracking Domain Adaptation
Aksu, T., Kan, M., & Chen, N. F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.252 · Cited by 5 · see citing works
Effective transfer learning for low-resource natural language understanding
Liu, Z. (2022) · doi:10.14711/thesis-991013106358603412 · Cited by 5 · see citing works
Self-citationLeveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State TrackingPreprint
Lin, Z., Liu, B., Moon, S., Crook, P., Zhou, Z., Wang, Z., Yu, Z., Madotto, A., Cho, E., & Subba, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04222 · Cited by 3 · see citing works
Self-citationRose, J., Ahmed, I., Keilty, E., Cooper, C., & Selby, P. L. (2022) · doi:10.36227/techrxiv.20029880.v1 · Cited by 3 · see citing works
Inspecting Spoken Language Understanding from Kids for Basic Math Learning at Home
Okur, E., Alba, R. F., Sahay, S., & Nachman, L. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.56 · Cited by 3 · see citing works
Knowledge-aware audio-grounded generative slot filling for limited annotated data
Sun, G., Zhang, C., Vulić, I., Budzianowski, P., & Woodland, P. C. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101707 · Cited by 4 · see citing works
Few‐Shot Contrastive Learning‐Based Multi‐Round Dialogue Intent Classification Method
Wei, F., & Zhang, X. (2024) · Expert Systems · doi:10.1111/exsy.13771 · Cited by 2 · see citing works
DEVELOPMENT THE DATASET FOR AUTOMATIC TRANSLATION SYSTEM
Ghanem, D. (2023) · Journal of Al-Azhar University Engineering Sector · doi:10.21608/auej.2023.297051 · Cited by 2 · see citing works
Enabling Classifiers to Make Judgements Explicitly Aligned with Human Values
Bang, Y., Yu, T., Madotto, A., Lin, Z., Diab, M., & Fung, P. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.27 · Cited by 3 · see citing works
Self-citationStepputtis, S., Campbell, J., Xie, Y., Qi, Z., Zhang, W., Wang, R., Rangreji, S., Lewis, C., & Sycara, K. P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.748 · Cited by 3 · see citing works
Xiao, Q., Zhao, Y., Dang, L., Hao, Y., Che, L., & Wu, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130639 · Cited by 1 · see citing works
N-Shot Learning for Augmenting Task-Oriented Dialogue State TrackingPreprint
Aksu, T., Liu, Z., Kan, M., & Chen, N. F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00293
Self-citation
- Authors
- Madotto, A., Liu, Z., Lin, Z., & Fung, P.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2008.06239
- OpenAlex
- W3049346316
- Cited by
- 36
- References
- 22
- Citations by year
- 2025: 1 · 2024: 3 · 2023: 11 · 2022: 9 · 2021: 11 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- task-oriented dialogue systems, few-shot learning, large language models, priming, dialogue state tracking, dialogue policy, natural language understanding, natural language generation, transfer learning, data scarcity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
