MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
Open accessThe DOI matches its Crossref record; 46 of the 48 references it cites were found in a registry or an index.
Abstract
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica Gašić. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some tasks have clear ground truth answers and can be scored with exact correctness (Du et al., 2024; Dunn et al., 2017), while others are open-ended (dialogue (Budzianowski et al., 2018), summarization (Maharana et al., 2024), preference following (Zhao et al., 2025c)) where multiple outputs are acceptable and reference-based scoring is not reliable.”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The former types of benchmarks are static and only test the conversational agent on pre-collected conversation trajectories[4, 3, 2].”
From $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 4×
“Among them, the first two metrics relate to the dialogue task completion - whether the system has provided an appropriate entity (In-form rate) and then answered all the requested attributes (Success rate); while fluency is measured via BLEU score (Papineni et al., 2002).”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“(2017) which frames the dialogue as a context to response mapping problem, a sequence-to-sequence model (Sutskever et al., 2014) is augmented with a belief tracker and a discrete database accessing component as additional features to inform the word decisions in the decoder.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“In the case of MultiWOZ dataset, the LSTM cell serving as a decoder and an encoder achieved the highest score with the global type of attention (Bahdanau et al., 2014).”
MethodMeasuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“As this requires a multi-annotator metric over a multi-label task, we used Fleiss’ kappa metric (Fleiss, 1971) per single dialogue act.”
MethodA Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926 · cited 2×
“Due to the lack of an explicit goal in the conversation, recent studies have shown that systems trained with this type of corpus not only struggle in generating consistent and diverse responses (Li et al., 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundVinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400 · cited 2×
“Although previous work (Vinyals and Le, 2015) has shown that a large learning system can learn to generate interesting responses from these corpora, the lack of grounding conversations onto an existing knowledge base or APIs limits the usability of developed systems.”
BackgroundLiu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1230 · Cited by 1,228 · cited 3×
“, 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundPOMDP-Based Statistical Spoken Dialog Systems: A Review
Young, S., Gasic, M., Thomson, B., & Williams, J. D. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2225812 · Cited by 882 · cited 4×
“These problems can either be addressed separately (Young et al., 2013), or jointly in an end-to-end fashion (Bordes et al., 2017; Wen et al., 2017; Li et al., 2017).”
BackgroundInfluentialSelf-citationSemantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874 · cited 5×
“To make a fair comparison, we constrained our dataset to only the restaurant sub-domain which contains around 25 k dia-Cam676 Table 5 presents the results on two metrics: BLEU score (Papineni et al., 2002) and slot error rate (SER) (Wen et al., 2015).”
BackgroundMethodInfluentialSelf-citationThe ATIS spoken language systems pilot corpus
Hemphill, C. T., Godfrey, J. J., & Doddington, G. R. (1990) · Proceedings of the workshop on Speech and Natural Language - HLT '90 · doi:10.3115/116580.116613 · Cited by 790 · cited 4×
“One of the earliest dataset collected in this fashion is the ATIS corpus (Hemphill et al., 1990), where conversations between a client and an airline help-desk operator were recorded.”
BackgroundMethodResultInfluential
Research works printed but not linked: 7
- Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. Proceedings of ICLR .
- Paweł Budzianowski, I˜ nigo Casanueva, Bo-Hsiang Tseng, and Milica Gaˇ si´ c. 2018. Towards endto-end multi-domain dialogue modelling. Tech. Rep. CUED/F-INFENG/TR.706, University of Cambridge, Engineering Department .
- Seokhwan Kim, Luis Fernando D'Haro, Rafael E Banchs, Jason D Williams, Matthew Henderson, and Koichiro Yoshino. 2016. The fifth dialog state tracking challenge. In Spoken Language Technology Workshop (SLT), 2016 IEEE , pages 511-517. IEEE.
- Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-end taskcompletion neural dialogue systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , volume 1, pages 733-743.
- David R. Traum. 1999. Foundations of Rational Agency , chapter Speech Acts for Dialogue Agents. Springer.
- Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gaˇ si´ c, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A networkbased end-to-end trainable task-oriented dialogue system. EACL .
- Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference , pages 404-413.
Cited by
Items 201–300 of 1,004
A Survey on Recent Advances in Conversational Data Generation
Soudani, H., Petcu, R., Kanoulas, E., & Hasibi, F. (2026) · ACM Computing Surveys · doi:10.1145/3795686 · Cited by 40
“Within the domain of TOD, the advancement and effectiveness of conversational systems, for example in performing specific tasks like booking a flight or making a reservation, heavily rely on the generation of robust and factually accurate dialogues.”
Continual Sequence Generation with Adaptive Compositional Modules
Zhang, Y., Wang, X., & Yang, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.255 · Cited by 58
The Teacher-Student Chatroom Corpus
Caines, A., Yannakoudakis, H., Edmondson, H., Allen, H., Pérez-Paredes, P., Byrne, B., & Buttery, P. J. (2020) · Linköping electronic conference proceedings · doi:10.3384/ecp2017510 · Cited by 35
LAVA: Latent Action Spaces via Variational Auto-encoding for Dialogue Policy Optimization
Lubis, N., Geishauser, C., Heck, M. C., Lin, H.-C., Moresi, M., van Niekerk, C., & Gašić, M. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.41 · Cited by 45
Self-citationEffective Sequence-to-Sequence Dialogue State Tracking
Zhao, J. W., Mahdieh, M., Zhang, Y., Cao, Y. J., & Wu, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.593 · Cited by 49
Amendable Generation for Dialogue State Tracking
Tian, X., Huang, L., Lin, Y., Bao, S., Huang, H., Yang, Y., Wu, H. R., Wang, F., & Sun, S. (2021) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2021.nlp4convai-1.8 · Cited by 35
Hu, Z., Feng, Y., Tuan, L. A., Hooi, B., & Lipani, A. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615220 · Cited by 35
“1 [3] and Schema Guided Dialogue (SGD) [17] are two typical TOD datasets, and they also have been annotated with satisfaction scores.”
BackgroundCINS: Comprehensive Instruction for Few-Shot Learning in Task-Oriented Dialog Systems
Mi, F., Wang, Y., & Li, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21356 · Cited by 56
A review of dialogue systems: current trends and future directions
Algherairy, A. S., & Ahmed, M. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-09322-1 · Cited by 59
“The data set provides a domain ontology that describes all the items and attributes in external databases [4, 68, 69].”
Dialog Intent Induction with Deep Multi-View Clustering
Perkins, H., & Yang, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1413 · Cited by 49
“As discussed in the introduction, existing goal-oriented dialog datasets mostly concern prede-fined dialog intents in some narrow domains such as restaurant or travel booking (Henderson et al., 2014a; Budzianowski et al., 2018; Serban et al., 2018).”
BackgroundLearning to execute instructions in a Minecraft dialogue
Jayannavar, P., Narayan-Chen, A., & Hockenmaier, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.232 · Cited by 65
“, 2015), focus on slot-value filling tasks (Kim et al., 2016b,a; Budzianowski et al., 2018) which instead require grounding to entities in a knowledge base, or operate within static environments, such as images (Das et al.”
BackgroundConVEx: Data-Efficient and Few-Shot Slot Labeling
Henderson, M., & Vulić, I. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.264 · Cited by 45
Dual Dynamic Memory Network for End-to-End Multi-turn Task-oriented Dialog Systems
Wang, J., Liu, J., Bi, W., Liu, X., He, K., Xu, R., & Yang, M. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.362 · Cited by 42
“1 dataset extends the Multi-WOZ (Budzianowski et al., 2018) by equipping the corresponding KB to every dialog, making it directly applicable for end-to-end generation.”
BackgroundInfluentialFew-shot Named Entity Recognition: Definition, Taxonomy and Research Directions
Moscato, V., Postiglione, M., & Sperlí, G. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3609483 · Cited by 41
“Multiwoz [12] 2018 Dialogues between a user and a virtual assistant covering multiple domains (e.”
BackgroundGraph Convolutional Network with Sequential Attention for Goal-Oriented Dialogue Systems
Banerjee, S., & Khapra, M. M. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00284 · Cited by 34
BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues
Le, H., Sahoo, D., Chen, N. F., & Hoi, S. C. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.145 · Cited by 42
QAConv: Question Answering on Informative Conversations
Wu, C.-S., Madotto, A., Liu, W., Fung, P., & Xiong, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.370 · Cited by 42
Shades of BLEU, Flavours of Success: The Case of MultiWOZ
Nekvinda, T., & Dušek, O. (2021) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2021.gem-1.4 · Cited by 59
Gunrock 2.0: A User Adaptive Social Conversational SystemPreprint
Liang, K.-H., Chau, A., Li, Y., Lu, X., Yu, D., Zhou, M., Jain, I., Davidson, S. R., Arnold, J., Nguyen, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.08906 · Cited by 37
“Current dialog corpora frequently used in dialog research, such as MultiWOZ[3] and PersuasionForGood [34] are all collected through role-play formats on crowdsourcing platforms.”
BackgroundHOOPS: Human-in-the-Loop Graph Reasoning for Conversational Recommendation
Fu, Z., Xian, Y., Zhu, Y., Xu, S., Li, Z., de Melo, G., & Zhang, Y. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463247 · Cited by 45
“[36] based on a large data-driven dialogue corpus [5, 10].”
BackgroundCONDA: a CONtextual Dual-Annotated dataset for in-game toxicity understanding and detection
Weld, H., Huang, G., Lee, J., Zhang, T., Wang, K., Guo, X., Long, S., Poon, J., & Han, S. C. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.213 · Cited by 42
Text-Based Emotion Recognition in English and Polish for Therapeutic Chatbot
Zygadło, A., Kozłowski, M., & Janicki, A. (2021) · Applied Sciences · doi:10.3390/app112110146 · Cited by 38
“Most of the existing dialogue corpora are either domain-specific and taskoriented [31,32] or collected without full control over the content, e.”
BackgroundAceta, C., Fernández, I., & Soroa, A. (2022) · Applied Sciences · doi:10.3390/app12031192 · Cited by 32
BORT: Back and Denoising Reconstruction for End-to-End Task-Oriented Dialog
Sun, H., Bao, J., Wu, Y., & He, X. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.166 · Cited by 43
Overcoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation
Li, D., Chen, Z., Cho, E., Hao, J., Liu, X., Xing, F., Guo, C., & Liu, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.398 · Cited by 65
“0 dataset (Budzianowski et al., 2018).”
Slot Attention with Value Normalization for Multi-Domain Dialogue State Tracking
Wang, Y., Guo, Y., & Zhu, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.243 · Cited by 31
“…two publicly available datasets: MultiWOZ 2.0 and MultiWOZ 2.1, both of which are fully-labeled task-oriented corpora comprised of human-human written conversations and contain 8,438 multi-turn dialogues with each dialogue having 13.68 turns on average in training set (Budzianowski et al., 2018).”
BackgroundMethodInfluentialLearning to Execute Actions or Ask Clarification Questions
Shi, Z., Feng, Y., & Lipani, A. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.158 · Cited by 60
Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models
Bae, S., Kwak, D., Kim, S., Ham, D., Kang, S., Lee, S., & Park, W. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.155 · Cited by 54
Shin, J., Yu, H., Moon, H., Madotto, A., & Park, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.302 · Cited by 60
Learning to Transfer Prompts for Text Generation
Li, J., Tang, T., Nie, J., Wen, J.-R., & Zhao, X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.257 · Cited by 51
Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog
Hung, C.-C., Lauscher, A., Vulić, I., Ponzetto, S. P., & Glavaš, G. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.270 · Cited by 54
Shah, C. S., White, R. W., Thomas, P., Mitra, B., Sarkar, S., & Belkin, N. J. (2023) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3576840.3578288 · Cited by 34
“Task-oriented dialog systems, focus on metrics such as number of slots filled ( 𝑥 of 𝑦 ) [25].”
BackgroundDomain-independent User Simulation with Transformers for Task-oriented Dialogue Systems
Lin, H.-C., Lubis, N., Hu, S., van Niekerk, C., Geishauser, C., Heck, M. C., Feng, S., & Gašić, M. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.47 · Cited by 60
Self-citationSimulated Chats for Building Dialog Systems: Learning to Generate Conversations from Instructions
Mohapatra, B., Pandey, G. S., Contractor, D., & Joshi, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.103 · Cited by 47
“Figure 1: An instruction from the MultiWOZ dialog corpus (Budzianowski et al., 2018) and the corresponding conversation generated by real crowd-sourced workers and crowd-worker simulators.”
BackgroundMethodInfluentialTODSum: Task-Oriented Dialogue Summarization with State TrackingPreprint
Zhao, L., Zheng, F., He, K., Zeng, W., Lei, Y., Jiang, H., Wu, W., Xu, W., Guo, J. Q., & Meng, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12680 · Cited by 39
“Note that we also keep the original dialogue state annotations of MultiWOZ.”
MethodInfluentialPriya, P., Firdaus, M., & Ekbal, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120025 · Cited by 37
Learning Low-Resource End-To-End Goal-Oriented Dialog for Fast and Reliable System Deployment
Dai, Y., Li, H., Tang, C., Li, Y., Sun, J. Q., & Zhu, X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.57 · Cited by 40
“MultiWOZ (Budzianowski et al., 2018) is a widely-used multi-domain Wizard-of-Oz dialog dataset spanning 7 distinct domains and containing 10k dialogs.”
MethodCIMA: A Large Open Access Dialogue Dataset for Tutoring
Stasaski, K., Kao, K., & Hearst, M. A. (2020) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2020.bea-1.5 · Cited by 29
“Multi-domain datasets such as MultiWOZ include 10k dialogues spanning multiple tasks (Budzianowski et al., 2018).”
BackgroundSelf-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems
Mi, F., Zhou, W., Kong, L., Cai, F., Huang, M., & Faltings, B. V. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.142 · Cited by 39
“Two common evaluation metrics (Budzianowski et al., 2018; Wu et al., 2019) are used: slot accuracy and joint goal accuracy.”
BackgroundMethodCasanueva, I., Vulić, I., Spithourakis, G., & Budzianowski, P. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.154 · Cited by 52
Self-citationA Dual Prompt Learning Framework for Few-Shot Dialogue State Tracking
Yang, Y., Lei, W., Huang, P., Cao, J., Li, J., & Chua, T. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583238 · Cited by 36
“Collecting state labels can be costly [3], requiring experts to annotate all (slot, value) information for each turn in dialogues.”
BackgroundA Primer on Seq2Seq Models for Generative Chatbots
Scotti, V., Sbattella, L., & Tedesco, R. (2023) · ACM Computing Surveys · doi:10.1145/3604281 · Cited by 38
“The idea was to ine-tune pre-trained language models on task-oriented dialogues data [24] adding, for example, speciic tokens to represent the additional pieces of information for the dialogue agent or prepending text containing the description of the agent’s belief state (i.”
BackgroundRobust Conversational AI with Grounded Text GenerationPreprint
Gao, J., Peng, B., Li, C., Li, J., Shayandeh, S., Lidén, L., & Shum, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.03457 · Cited by 33
An Introduction to Lifelong Supervised LearningPreprint
Sodhani, S., Faramarzi, M., Mehta, S. V., Malviya, P., Abdelsalam, M., Janarthanan, J., & Chandar, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2207.04354 · Cited by 35
ChainCQG: Flow-Aware Conversational Question Generation
Gu, J., Mirshekari, M., Yu, Z., & Sisto, A. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.177 · Cited by 28
SalesBot: Transitioning from Chit-Chat to Task-Oriented Dialogues
Chiu, S., Li, M., Lin, Y.-T. R., & Chen, Y.-N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.425 · Cited by 48
Communicating Natural Programs to Humans and Machines
Acquaviva, S., Pu, Y., Kryven, M., Sechopoulos, T., Wong, C., Ecanow, G., Nye, M., Tessler, M. H., & Tenenbaum, J. B. (2022) · neural information processing systems · doi:10.52202/068431-0270 · Cited by 71
Soliciting User Preferences in Conversational Recommender Systems via Usage-related Questions
Kostric, I., Balog, K., & Radlinski, F. (2021) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3460231.3478861 · Cited by 41
TWEETSUMM - A Dialog Summarization Dataset for Customer Service
Feigenblat, G., Gunasekara, C., Sznajder, B., Joshi, S., Konopnicki, D., & Aharonov, R. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.24 · Cited by 59
Context-Aware Language Modeling for Goal-Oriented Dialogue Systems
Snell, C., Yang, S., Fu, J., Su, Y., & Levine, S. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.181 · Cited by 36
Abstractive Dialog Summarization with Semantic Scaffolds
Yuan, L., & Zhou, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.00825 · Cited by 29
“We propose an abstractive dialog summarization dataset based on MultiWOZ (Budzianowski et al., 2018).”
MethodInfluentialAttention over Parameters for Dialogue SystemsPreprint
Madotto, A., Lin, Z., Wu, C.-S., Shin, J., & Fung, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.01871 · Cited by 33
“For all three datasets, we use the training/validation/test split provided by the authors and we keep all the real entities in input instead of using their delexicalized version as in [1, 2].”
BackgroundMethodInfluentialConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue SystemsPreprint
Zhu, Q., Zhang, Z., Fang, Y. M., Li, X. R., Takanobu, R., Li, J., Peng, B., Gao, J., Zhu, X., & Huang, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04793 · Cited by 34
“, 2017), MultiWOZ (Budzianowski et al., 2018b), DealOrNoDeal (Lewis et al.”
BackgroundMethodInfluentialMeta-Reinforced Multi-Domain State Generator for Dialogue Systems
Huang, Y., Feng, J., Hu, M., Wu, X., Du, X., & Ma, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.636 · Cited by 28
“Budzianowski et al. (2018) recently introduced a multi-domain dialogue dataset Multi-domain Wizard-of-Oz (MultiWOZ), which is more than one order of magnitude larger than all previous annotated task-oriented corpora with around 10k dialogues and involves more than 7 domains.”
BackgroundModeling Long Context for Task-Oriented Dialogue State Generation
Quan, J., & Xiong, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.637 · Cited by 27
Derivative Based Nonbacktracking Real-World Regex Matching with Backtracking Semantics
Moseley, D., Nishio, M., Rodriguez, J. P., Saarikivi, O., Toub, S., Veanes, M., Wan, T., & Xu, E. Y. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591262 · Cited by 27
“We are investigating ways to reduce the initialization overhead of NonBacktracking through techniques such as code generation.”
MethodMulti-Domain Task-Completion Dialog Challenge
Lee, S., Schulz, H., Atkinson, A., Gao, J., Suleman, K., Asri, L. E., Adada, M., minlie-huang, Sharma, S., Tay, W. W., et al. (2019) · Cited by 27
End-to-End Trainable Non-Collaborative Dialog System
Li, Y., Qian, K., Shi, W., & Yu, Z. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6345 · Cited by 48
Domain-Aware Dialogue State Tracker for Multi-Domain Dialogue Systems
Balaraman, V., & Magnini, B. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3054309 · Cited by 27
“Recent work has shown promise in modeling multi-domain dialogues and has facilitated the release of multi-domain dialogue datasets, such as MultiWOZ (Budzianowski et al. 2018) and FRAMES (El Asri et al. 2017).”
BackgroundParallel Scheduled SamplingPreprint
Duckworth, D., Neelakantan, A., Goodrich, B., Kaiser, Ł., & Samy Bengio, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.04331 · Cited by 29
“We evaluate our method on text response generation task using MultiWOZ [2], a task-oriented dialog dataset.”
BackgroundMethodInfluentialLearning Dialog Policies from Weak Demonstrations
Gordon-Hall, G., Gorinski, P. J., & Cohen, S. B. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.129 · Cited by 27
Unsupervised Abstractive Dialogue Summarization for Tete-a-Tetes
Zhang, X., Zhang, R., Zaheer, M., & Ahmed, A. A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17703 · Cited by 26
“How Robust R U?”: Evaluating Task-Oriented Dialogue Systems on Spoken Conversations
Kim, S., Liu, Y., Jin, D., Papangelis, A., Gopalakrishnan, K., Hedayatnia, B., & Hakkani-Tür, D. Z. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688274 · Cited by 49
KETOD: Knowledge-Enriched Task-Oriented Dialogue
Chen, Z., Liu, B., Moon, S., Sankar, C., Crook, P., & Wang, W. Y. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.197 · Cited by 45
Unraveling ChatGPT: A Critical Analysis of AI-Generated Goal-Oriented Dialogues and Annotations
Labruna, T., Brenna, S., Zaninello, A., & Magnini, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47546-7_11 · Cited by 43
Template Guided Text Generation for Task-Oriented DialoguePreprint
Kale, M., & Rastogi, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.15006 · Cited by 26
“We conduct experiments on 2 datasets - SGD-NLG (Rastogi et al., 2019), MultiWOZ (Budzianowski et al., 2018).”
BackgroundMethodResponse Generation with Context-Aware Prompt LearningPreprint
Gu, X., Yoo, K. M., & Lee, S.-W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.02643 · Cited by 29
“TOD-BERT proposed by Wu et al. (2020) bridges the difference of general text and task-oriented dialogue by unifying nine human-human and multi-turn task-oriented dialogue datasets for language modeling.”
InfluentialUser Simulation for Evaluating Information Access Systems
Balog, K., & Zhai, C. (2023) · arXiv · doi:10.48550/arxiv.2306.08550 · Cited by 26
A Survey on Neural Data-to-Text Generation
Lin, Y., Ruan, T., Liu, J., & Wang, H. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3304385 · Cited by 57
“The KVRET [60] and MultiWOZ [61] are large-scale multi-domain datasets containing thousands of dialogues.”
BackgroundVakulenko, S., Kanoulas, E., & de Rijke, M. (2021) · ACM Transactions on Information Systems · doi:10.1145/3466796 · Cited by 32
“…consider it representative of this dialogue type due to its size (10K dialogues is an order of magnitude larger than all previous annotated task-oriented corpora [9]) and diversity (single-domain and multi-domain dialogues about restaurants, hotels, attractions, taxi, trains, hospitals and police).”
ASSIST: Towards Label Noise-Robust Dialogue State Tracking
Ye, F., Feng, Y., & Yılmaz, E. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.214 · Cited by 27
Show, Don’t Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue
Gupta, R., Lee, H., Zhao, J., Cao, Y. J., Rastogi, A., & Wu, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.336 · Cited by 42
“SDT achieves SotA accuracy and generalization to new APIs across both the Schema-Guided Dataset (SGD) (Rastogi et al., 2020b) and MultiWOZ Leave-One-Out (Budzianowski et al., 2018; Lin et al., 2021b) benchmarks, while being more data-efficient and robust to schema variations.”
MethodInfluentialVisual language integration: A survey and open challenges
Park, S., & Kim, Y. (2023) · Computer Science Review · doi:10.1016/j.cosrev.2023.100548 · Cited by 33
TicketTalk: Toward human-level performance with end-to-end, transaction-based dialog systems
Byrne, B., Krishnamoorthi, K., Ganesh, S., & Kale, M. (2021) · doi:10.18653/v1/2021.acl-long.55 · Cited by 32
Conversational Semantic Role Labeling
Xu, K., Wu, H., Song, L., Zhang, H., Song, L., & Yu, D. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3074014 · Cited by 32
How to Build User Simulators to Train RL-based Dialog SystemsPreprint
Shi, W., Qian, K., Wang, X., & Yu, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01388 · Cited by 25
“We pick the popular restaurant search task from Multiwoz (Budzianowski et al., 2018) and analyze the pros and cons of different user simulator building methods.”
BackgroundMethodInfluentialA Survey on Dialog Management: Recent Advances and ChallengesPreprint
Dai, Y., Yu, H., Jiang, Y., Tang, C., Li, Y., & Sun, J. Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.02233 · Cited by 25
“In the famous MultiWOZ dataset for multi-domain dialogs, the team led by Professor Pascale Fung from Hong Kong University of Science and Technology used the copy network to significantly improve the recognition accuracy of non-enumerative slot values[20].”
BackgroundInfluentialHierarchical Transformer for Task Oriented Dialog Systems
Santra, B., Anusha, P., & Goyal, P. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.449 · Cited by 31
Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systems
Li, Z., Kiseleva, J., & de Rijke, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.316 · Cited by 24
AugNLG: Few-shot Natural Language Generation using Self-trained Data Augmentation
Xu, X., Wang, G., Kim, Y.-B., & Lee, S. (2021) · doi:10.18653/v1/2021.acl-long.95 · Cited by 28
“shot NLG benchmark, built upon RNNLG and MultiWOZ (Budzianowski et al., 2018).”
QA-Driven Zero-shot Slot Filling with Weak Supervision Pretraining
Du, X., He, L., Li, Q., Yu, D., Pasupat, P., & Zhang, Y. (2021) · doi:10.18653/v1/2021.acl-short.83 · Cited by 39
“To obtain the pre-defined ontology and unlabeled conversations, we use MultiWOZ 2.2 (Zang et al., 2020), which is an improved version of Mul-tiWOZ (Budzianowski et al., 2018).”
MethodInfluentialTIAGE: A Benchmark for Topic-Shift Aware Dialog Modeling
Xie, H., Liu, Z., Xiong, C., Liu, Z., & Copestake, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.145 · Cited by 49
LUNA: Learning Slot-Turn Alignment for Dialogue State Tracking
Wang, Y., Zhao, J., Bao, J., Duan, C., Wu, Y., & He, X. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.242 · Cited by 24
Variational Latent-State GPT for Semi-Supervised Task-Oriented Dialog Systems
Liu, H., Cai, Y., Lin, Z., Ou, Z., Huang, Y., & Feng, J. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3240661 · Cited by 27
“1, we follow the original MultiWOZ guidance [36] for individual metrics and follow [37] for the combined score.”
BackgroundMG-ShopDial: A Multi-Goal Conversational Dataset for e-Commerce
Bernard, N., & Balog, K. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591883 · Cited by 44
“Another way is to perform user studies focusing on a task, typically using a crowdsourcing platform to recruit participants as in [8, 12, 48].”
BackgroundMethodInfluentialDiverse Retrieval-Augmented In-Context Learning for Dialogue State Tracking
King, B., & Flanigan, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.344 · Cited by 43
Curate and Generate: A Corpus and Method for Joint Control of Semantics and Style in Neural NLG
Oraby, S., Harrison, V., Ebrahimi, A., & Walker, M. A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1596 · Cited by 29
“Other recent work on collecting datasets for dialog modeling also use largescale crowdsourcing (Budzianowski et al., 2018).”
BackgroundVariational Hierarchical Dialog Autoencoder for Dialog State Tracking Data Augmentation
Yoo, K. M., Lee, H., Dernoncourt, F., Bui, T., Chang, W. H., & Lee, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.274 · Cited by 24
Parallel Interactive Networks for Multi-Domain Dialogue State Generation
Chen, J., Zhang, R., Mao, Y., & Xu, J. J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.151 · Cited by 24
Utterance-level Dialogue Understanding: An Empirical StudyPreprint
Ghosal, D., Majumder, N., Mihalcea, R. F., & Poria, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13902 · Cited by 24
“MultiWOZ (Budzianowski et al., 2018) or Multi-Domain Wizard-of-Oz dataset is a fullylabeled collection of human-human written conversations spanning over multiple domains and topics.”
BackgroundEnd-to-End Learning of Flowchart Grounded Task-Oriented Dialogs
Raghu, D., Agarwal, S., Joshi, S., & Mausam, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.357 · Cited by 35
GlobalWoZ: Globalizing MultiWoZ to Develop Multilingual Task-Oriented Dialogue Systems
Ding, B., Hu, J., Bing, L., Aljunied, M., Joty, S., Si, L., & Miao, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.115 · Cited by 23
Robe, P., Kuttal, S. K., AuBuchon, J., & Hart, J. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549127 · Cited by 27
“Each benchmark targets data for specific applications and domains [43]; for example, MultiWOZ is comprised of 10-thousand tourist reservation dialogs.”
BackgroundThe Importance of Context for Sentiment Analysis in Dialogues
Carvalho, I., Oliveira, H. G., & Silva, C. (2023) · IEEE Access · doi:10.1109/access.2023.3304633 · Cited by 26
“Wikipedia [41], Multi-Woz [42], CamRest [43], and Ubuntu”
Annotation Inconsistency and Entity Bias in MultiWOZ
Qian, K., Beirami, A., Lin, Z., De, A., Geramifard, A., Yu, Z., & Sankar, C. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.35 · Cited by 40
Constraint based Knowledge Base Distillation in End-to-End Task Oriented Dialogs
Raghu, D., Jain, A., Mausam, M., & Joshi, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.448 · Cited by 23
“Table 9 shows responses predicted by CDNET, EER and FG2Seq for an example from the WOZ dataset.”
MethodInfluentialPretraining the Noisy Channel Model for Task-Oriented Dialogue
Liu, Q., Yu, L., Rimell, L., & Blunsom, P. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00390 · Cited by 31
The Effects of System Initiative during Conversational Collaborative Search
Avula, S., Choi, B., & Arguello, J. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3512913 · Cited by 32
Towards personalized persuasive dialogue generation for adversarial task oriented dialogue setting
Tiwari, A., Khandwe, A., Saha, S., Ramnani, R., Maitra, A., & Sengupta, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118775 · Cited by 28
Metadata
- Authors
- Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., & Gašić, M.
- Affiliations
- University of Cambridge (GB)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 5016-5026
- OpenAlex
- W2891732163
- Cited by
- 1,028
- References
- 48
- Citations by year
- 2026: 39 · 2025: 66 · 2024: 86 · 2023: 209 · 2022: 186 · 2021: 196 · 2020: 181 · 2019: 64 · 2018: 1
- FWCI
- 45.11
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/M018946/1, RG80792)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- MultiWOZ, task-oriented dialogue, multi-domain dialogue, dialogue modeling, dialogue dataset, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.