MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
Open accessThe DOI matches its Crossref record; 46 of the 48 references it cites were found in a registry or an index.
Abstract
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica Gašić. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some tasks have clear ground truth answers and can be scored with exact correctness (Du et al., 2024; Dunn et al., 2017), while others are open-ended (dialogue (Budzianowski et al., 2018), summarization (Maharana et al., 2024), preference following (Zhao et al., 2025c)) where multiple outputs are acceptable and reference-based scoring is not reliable.”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The former types of benchmarks are static and only test the conversational agent on pre-collected conversation trajectories[4, 3, 2].”
From $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 4×
“Among them, the first two metrics relate to the dialogue task completion - whether the system has provided an appropriate entity (In-form rate) and then answered all the requested attributes (Success rate); while fluency is measured via BLEU score (Papineni et al., 2002).”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“(2017) which frames the dialogue as a context to response mapping problem, a sequence-to-sequence model (Sutskever et al., 2014) is augmented with a belief tracker and a discrete database accessing component as additional features to inform the word decisions in the decoder.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“In the case of MultiWOZ dataset, the LSTM cell serving as a decoder and an encoder achieved the highest score with the global type of attention (Bahdanau et al., 2014).”
MethodMeasuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“As this requires a multi-annotator metric over a multi-label task, we used Fleiss’ kappa metric (Fleiss, 1971) per single dialogue act.”
MethodA Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926 · cited 2×
“Due to the lack of an explicit goal in the conversation, recent studies have shown that systems trained with this type of corpus not only struggle in generating consistent and diverse responses (Li et al., 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundVinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400 · cited 2×
“Although previous work (Vinyals and Le, 2015) has shown that a large learning system can learn to generate interesting responses from these corpora, the lack of grounding conversations onto an existing knowledge base or APIs limits the usability of developed systems.”
BackgroundLiu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1230 · Cited by 1,228 · cited 3×
“, 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundPOMDP-Based Statistical Spoken Dialog Systems: A Review
Young, S., Gasic, M., Thomson, B., & Williams, J. D. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2225812 · Cited by 882 · cited 4×
“These problems can either be addressed separately (Young et al., 2013), or jointly in an end-to-end fashion (Bordes et al., 2017; Wen et al., 2017; Li et al., 2017).”
BackgroundInfluentialSelf-citationSemantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874 · cited 5×
“To make a fair comparison, we constrained our dataset to only the restaurant sub-domain which contains around 25 k dia-Cam676 Table 5 presents the results on two metrics: BLEU score (Papineni et al., 2002) and slot error rate (SER) (Wen et al., 2015).”
BackgroundMethodInfluentialSelf-citationThe ATIS spoken language systems pilot corpus
Hemphill, C. T., Godfrey, J. J., & Doddington, G. R. (1990) · Proceedings of the workshop on Speech and Natural Language - HLT '90 · doi:10.3115/116580.116613 · Cited by 790 · cited 4×
“One of the earliest dataset collected in this fashion is the ATIS corpus (Hemphill et al., 1990), where conversations between a client and an airline help-desk operator were recorded.”
BackgroundMethodResultInfluential
Research works printed but not linked: 7
- Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. Proceedings of ICLR .
- Paweł Budzianowski, I˜ nigo Casanueva, Bo-Hsiang Tseng, and Milica Gaˇ si´ c. 2018. Towards endto-end multi-domain dialogue modelling. Tech. Rep. CUED/F-INFENG/TR.706, University of Cambridge, Engineering Department .
- Seokhwan Kim, Luis Fernando D'Haro, Rafael E Banchs, Jason D Williams, Matthew Henderson, and Koichiro Yoshino. 2016. The fifth dialog state tracking challenge. In Spoken Language Technology Workshop (SLT), 2016 IEEE , pages 511-517. IEEE.
- Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-end taskcompletion neural dialogue systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , volume 1, pages 733-743.
- David R. Traum. 1999. Foundations of Rational Agency , chapter Speech Acts for Dialogue Agents. Springer.
- Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gaˇ si´ c, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A networkbased end-to-end trainable task-oriented dialogue system. EACL .
- Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference , pages 404-413.
Cited by
Items 701–800 of 1,004
SF-DST: Few-Shot Self-Feeding Reading Comprehension Dialogue State Tracking with Auxiliary Task
Lee, J., & Lee, G. G. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-380 · Cited by 4
“Our model achieved the new best accuracy in a few-shot ex-periment for four domains on MultiWOZ 2.0 [7], and achieved close to the current state-of-the-art in a supervised setting on MultiWOZ 2.1 [8].”
MethodInfluentialZhang, X., Divekar, R. R., Ubale, R., & Zhou, Y. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.26 · Cited by 13
“Several datasets exist to help build task-oriented dialogues such as Schema-Guided-Dialogue (SGD) (Rastogi et al., 2020), MultiWoZ (Budzianowski et al., 2018), Dialogue State Tracking Challenges (DSTC) 1-3 (Williams et al., 2013; Henderson et al., 2014a,b) and DSTC 4-5 (Kim et al., 2017).”
BackgroundMethodResultInfluentialGenerating Synthetic Dialogues from Prompts to Improve Task-Oriented Dialogue Systems
Steindl, S., Schäfer, U., & Ludwig, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42608-7_17 · Cited by 7
GRILLBot: A multi-modal conversational agent for complex real-world tasks
Gemmell, C., Rossetto, F., Mackie, I., Owoicho, P., Fischer, S., & Dalton, J. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.63 · Cited by 4
Large Language Models guided Generative Prompt for Dialogue Generation
Liu, S., Fang, Y., Cheng, H. N., Pan, Y., Liu, Y., & Gao, C. (2023) · International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC) · doi:10.1109/cyberc58899.2023.00013 · Cited by 7
“MultiWOZ [30]: a collection of fully tagged written person-to-person conversations covering a wide range of areas and topics such as attractions, hotels, hospitals, police, restaurants, trains, and taxis.”
BackgroundOptimizing pipeline task-oriented dialogue systems using post-processing networks
Ohashi, A., & Higashinaka, R. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101742 · Cited by 4
Multi-domain dialogue state tracking via dual dynamic graph with hierarchical slot selector
Gong, Y., Kim, H., Hwang, S., Kim, D., & Lee, K.-H. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112754 · Cited by 4
SpeechTaxi: On Multilingual Semantic Speech Classification
Keller, L., & Glavaš, G. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890017 · Cited by 7
“Historically, however, SLU has been heavily influenced by task-oriented dialogue (ToD) [2], [3], resulting in datasets ( [4], [5], inter alia ) built around ToD-specific tasks, intent classification and slot filling.”
Intent Recognition in Dialogue Systems
Ran, Z., Chen, Y., Jiang, Q., & Kunpeng, E. (2024) · Asia-Pacific Artificial Intelligence and Big Data Forum (AIBDF) · doi:10.1145/3718491.3718520 · Cited by 6
RAICL-DSC: Retrieval-Augmented In-Context Learning for Dialogue State Correction
Su, H., Xie, H., Shi, J., Wu, D., Jiang, L., Huang, H., He, Z., Li, Y., Fang, R., Zhao, J., et al. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113423 · Cited by 4
Towards reliable and empathetic depression-diagnosis-oriented chats
Lan, K., Sun, T., Ming, C., Yao, B., Ding, Y., Yan, Y., Li, Y., Luo, C. B., Chen, L., Chen, J., et al. (2025) · Science China Technological Sciences · doi:10.1007/s11431-025-3036-5 · Cited by 5
“These insights underscore the potential for leveraging intent transition patterns to enhance predictive models in depression diagnosis dialogues.”
Modeling Conversational Agents for Service Systems
Sindhgatta, R., Barros, A., & Nili, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33246-4_34 · Cited by 4
Neural Dialogue State Tracking with Temporally Expressive Networks
Chen, J., Zhang, R., Mao, Y., & Xu, J. J. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.142 · Cited by 4
Generalizable and Explainable Dialogue Generation via Explicit Action Learning
Huang, X., Qi, J., Sun, Y., & Zhang, R. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.355 · Cited by 5
Recognising flexible intents and multiple domains in extended human-robot dialogues
Wilcock, G. (2021) · Työväentutkimus Vuosikirja · doi:10.11517/pjsai.jsai2021.0_1n2is5a05 · Cited by 4
Bursztyn, V. S., Healey, J., Lipka, N., Koh, E., Downey, D., & Birnbaum, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.145 · Cited by 6
“…they either frame the problem as a slot-filling task within a predefined feature space, such as Sun and Zhang (2018); Zhang et al. (2018); Budzianowski et al. (2018), which is closer to how people make decisions but not as flexible as real conversations; or they elicit preferences by…”
BackgroundDeep Conversational Recommender Systems: Challenges and Opportunities
Tran, D. H., Sheng, Q. Z., Zhang, W. E., Hamad, S. A., Khoa, N. L. D., & Tran, N. H. K. (2022) · Computer · doi:10.1109/mc.2020.3045426 · Cited by 6
“However, only a handful of them are available, such as the ReDIAL 5 and MultiWOZ 16 This has allowed us to observe an interesting fact: the research works that utilize fluent dialogue data sets tend to focus on generating seamless responses in a multiround setting, where the agent’s response can include zero or more r…”
BackgroundIntegrating Pretrained Language Model for Dialogue Policy Evaluation
Wang, H., Wang, H., Wang, Z., & Wong, K. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747593 · Cited by 4
“All models are evaluated on MultiWOZ [12], a multi-domain, multi-intent task-oriented dialog corpus that contains 7 domains, 13 intents, 25 different slots and 10483 dialog sessions.”
MethodInfluentialSucameli, I., Lenci, A., Magnini, B., Simi, M., & Speranza, M. (2020) · Accademia University Press eBooks · doi:10.4000/books.aaccademia.8915 · Cited by 4
Toward Real-life Dialogue State Tracking Involving Negative Feedback Utterances
Yang, P., Huang, H., Wei, W., & Mao, X.-L. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539385 · Cited by 4
“In recent years, with the release of large multi-domain task-oriented dialogue datasets [1, 16, 27], especially since MultiWOZ series [4, 5, 23] was proposed, the research and exploration of dialogue state tracking have developed rapidly.”
BackgroundKeyword-guided Topic-oriented Conversational Recommender System
Pan, Y., Yin, Y., & Huang, F. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892470 · Cited by 4
“We use public dataset Multi-WOZ 2.0 [34] to verify the effectiveness of our proposed model, which contains 10,438 human-human conversation sessions.”
MethodInfluentialCao, B., Wang, K., Ma, K., Fan, J., Yan, R., & Xu, Y. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2022.3223924 · Cited by 4
“Budzianowski et al. [17] proposed the Multidomain Wizard-of-Oz (Multi-WOZ) data set, which is a large-scale multiturn conversational corpus with dialogues spanning across several domains and topics.”
BackgroundMethodInfluentialVoice-Assisted Food Recall Using Voice Assistants
Liang, X., Batsis, J. A., Yuan, J., Zhu, Y., Driesse, T. M., & Schultz, J. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17902-0_7 · Cited by 4
RSP-DST: Revisable State Prediction for Dialogue State Tracking
Li, Q., Zhang, W., Huang, M., Feng, S., & Wu, Y. (2023) · Electronics · doi:10.3390/electronics12061494 · Cited by 5
Mizukami, M., Sugiyama, H., Arimoto, T., Higashinaka, R., Mitsuda, K., & Kobayashi, T. (2023) · Transactions of the Japanese Society for Artificial Intelligence · doi:10.1527/tjsai.38-3_b-ma2 · Cited by 5
KPT: Keyword-Guided Pre-training for Grounded Dialog Generation
Zhu, Q., Mi, F., Zhang, Z., Wang, Y., Li, Y., Jiang, X., Liu, Q., Zhu, X., & Huang, M. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26646 · Cited by 5
Fang, H., Balakrishnan, A., Jhamtani, H., Bufe, J., Crawford, J., Krishnamurthy, J., Pauls, A., Eisner, J. M., Andreas, J., & Klein, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.351 · Cited by 7
Robust Task-Oriented Dialogue Generation with Contrastive Pre-training and Adversarial Filtering
Yang, S., Huang, X., Lau, J. H., & Erfani, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.88 · Cited by 6
Zhao, M., Wang, L., Ji, H., Jiang, Z., Li, R., Lu, X., & Hu, Z. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110927 · Cited by 4
The Devil is in the Details: On Models and Training Regimes for Few-Shot Intent Classification
Mesgar, M., Tran, T. T., Glavaš, G., & Gurevych, I. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.135 · Cited by 6
Zero and Few-Shot Localization of Task-Oriented Dialogue Agents with a Distilled Representation
Moradshahi, M., Semnani, S. J., & Lam, M. S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.62 · Cited by 4
Multi-Domain Dialogue State Tracking with Top-K Slot Self Attention
Yang, L., Li, J., Li, S., & Shinozaki, T. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.24 · Cited by 4
“MultiWOZ 2.0 (Budzianowski et al., 2018) To make a fair comparison with the models evaluated on these two datasets, we follow the procedure in several previous works (Wu et al., 2019; Lee et al., 2019; Wang et al., 2020; Ye et al., 2021b) to keep consistent.”
MethodInfluentialFeng, S., Lubis, N., Ruppik, B., Geishauser, C., Heck, M. C., Lin, H.-C., van Niekerk, C., Vukovic, R., & Gašić, M. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.8 · Cited by 11
Self-citationFederated Learning for Intent Classification
Cureton, M., & Corradini, A. (2023) · IEEE International Conference on Intelligent Computer Communication and Processing (ICCP) · doi:10.1109/iccp60212.2023.10398713 · Cited by 4
KMc-ToD: Structure knowledge enhanced multi-copy network for task-oriented dialogue system
Ding, Z., Yang, Z., Qiao, Y., & Lin, H. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111662 · Cited by 4
Yan, R., & Wu, W. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17771 · Cited by 7
“With the release of datasets such as MultiWOZ (Budzianowski et al. 2018), there is rapid progress towards overcoming the challenge (Wu et al. 2019a; Chen et al. 2019).”
BackgroundModeling ASR Ambiguity for Dialogue State Tracking Using Word Confusion NetworksPreprint
Pal, V., Guillot, F., Shrivastava, M., Renders, J.-M., & Besacier, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.00768 · Cited by 4
“0 [11] and MultiWoz 2.”
Context-Sensitive Generation Network for Handing Unknown Slot Values in Dialogue State TrackingPreprint
Yang, P., Huang, H., & Mao, X.-L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03923 · Cited by 4
“1 dataset is the latest corrected version of the MultiWOZ dataset [28].”
BackgroundAttention Guided Dialogue State Tracking with Sparse SupervisionPreprint
Liang, S., Poddar, L., & Szarvas, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11958 · Cited by 4
“0 (Budzianowski et al., 2018), which fixes noisy state annotations.”
BackgroundInfluentialA Knowledge-Grounded Dialog System Based on Pre-Trained Language ModelsPreprint
Zhang, W., Chen, J., Wu, H., Wan, S., & Li, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14444 · Cited by 4
“1 (Budzianowski et al. 2018; Eric et al. 2019), domain and entity information are typically stored in a structured database and can be retrieved via an API.”
BackgroundMethodInfluentialSchema-Guided Paradigm for Zero-Shot DialogPreprint
Mehri, S., & Eskénazi, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07056 · Cited by 4
“, 2005), or acting as a tour guide (Budzianowski et al., 2018).”
BackgroundMulti-Task Learning for Situated Multi-Domain End-to-End Dialogue SystemsPreprint
Kung, P.-N., Chang, C., Yang, T.-H., Hsu, H.-K., Liou, Y.-J., & Chen, Y.-N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05221 · Cited by 4
“Multiple multi-domain DST datasets have been released(Budzianowski et al. 2018; Rastogi et al. 2020).”
Learning Dialogue Representations from Consecutive UtterancesPreprint
Zhou, Z., Zhang, D., Xiao, W., Dingwall, N., Ma, X., Arnold, A. O., & Xiang, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.13568 · Cited by 6
“…of 9 publicly available task-oriented datasets: MetaLWOZ (Lee et al., 2019), Schema (Rastogi et al., 2020), Taskmas-ter (Byrne et al., 2019), MWOZ (Budzianowski et al., 2018), MSR-E2E (Li et al., 2018), SMD (Eric and Manning, 2017), Frames (Asri et al., 2017), WOZ (Mrkši´c et al., 2016),…”
Jones, K., Altuncu, E., Franqueira, V. N. L., Wang, Y., & Li, S. (2022) · Kent Academic Repository (University of Kent) · doi:10.48550/arxiv.2208.05757 · Cited by 6
“For instance, the dialogue act INFORM(domain=restaurant,price=cheap) indicates an intent to inform, with slots for the ‘domain’ and the ‘price’, and values of ‘restaurant’, and ‘cheap’ [126].”
BackgroundMethodInfluentialZhao, L., Zheng, F., Zeng, W., He, K., Xu, W., Jiang, H., Wu, W., & Wu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.04362 · Cited by 4
“(2021b), which is a task-oriented dialogue summarization dataset based on the classic dialogue dataset MultiWOZ (Budzianowski et al., 2018).”
BackgroundImproving Generalization in Task-oriented Dialogues with Workflows and Action PlansPreprint
Raimondo, S., Pal, C., Liu, X., Vázquez, D., & Palacios, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.01729 · Cited by 4
“Early examples include ATIS (Hemphill et al., 1990) and DSTC2 (Henderson et al., 2014), while some of the most widely used more modern datasets include Simulated Dialogue (M2M) (Shah et al., 2018), MultiWoz (Budzianowski et al., 2020), TaskMaster (Byrne et al., 2019), Frames (Asri et al., 2017), and the Schema-Guided…”
MethodInfluentialJoTR: A Joint Transformer and Reinforcement Learning Framework for Dialog Policy LearningPreprint
Kwan, W.-C., Wang, H., Wang, H., Wang, Z., Wu, X., Zheng, Y., & Wong, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.00230 · Cited by 4
“Experiments are carried out on MultiWOZ 2.0 (Budzianowski et al. 2018), utilizing a publicly accessible agenda-based user simulator (Zhu et al. 2020), and on the SGD dataset with our developed rule-based simulator.”
MethodInfluentialEvaluating Multi-Turn Bargain Skills in LLM-Based Seller AgentPreprint
Wang, I. Y., Chong, K., Wang, X., Yan, X., Kong, D., Ju, C., Chen, M., Xiao, S., Han, S., & chen, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.06341 · Cited by 4
“Dialogue State Tracking datasets such as MultiWOZ Budzianowski et al. [2018] address explicit task goals, but bargaining often involves implicit and shifting intents.”
Xu, P., Li, S., Sun, A., Zhang, F., Li, Y., Wu, B., Ma, Z., Li, J., Xu, J., Gao, J., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.21244 · Cited by 4
“Traditional evaluation relies on static, pre-collected corpora and thus cannot faithfully reflect robustness and policy effectiveness under dynamic interaction (Chen et al., 2021; Andreas et al., 2020; Budzianowski et al., 2018).”
SAGE: A Service Agent Graph-guided Evaluation BenchmarkPreprint
Shi, L., Dai, Y., Wang, Z., Gao, N., Zhang, W., Wang, C., Wang, Y., He, W., Wang, J., & Xiong, D. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.09285 · Cited by 4
“Second, static interaction paradigms : Relying on fixed datasets like scripts [5, 34, 58], traditional methods fail to test error recovery or cover diverse user behaviors, ranging from cooperative inquiries to adversarial conflicts.”
Conversational AI Systems for Social Good: Opportunities and Challenges
Qi, P., Huang, J., Wu, Y., He, X., & Zhou, B. (2021) · arXiv · doi:10.48550/arxiv.2105.06457 · Cited by 3
“Task-oriented conversations remain a popular problem to tackle, as they help human users to achieve concrete goals, such as booking a restaurant or turning on the light ( e.g. , conversations in the MultiWOZ dataset (Budzianowski et al. 2018)).”
BackgroundInfluentialInspecting Spoken Language Understanding from Kids for Basic Math Learning at Home
Okur, E., Alba, R. F., Sahay, S., & Nachman, L. (2023) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2023.bea-1.56 · Cited by 4
OSTOD: One-Step Task-Oriented Dialogue with activated state and retelling response
Huang, H., Yang, P., Wei, W., Shi, S., & Mao, X.-L. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111677 · Cited by 4
Zhu, S., Rzeszotarski, J. M., & Mimno, D. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720285 · Cited by 6
“Like most other data sources, WILD can also benefit from corpus annotation, e.g., crowdsourcing fine-grained data for intentions or dialog state [4].”
Eyes on the Code: Mapping Critical Thinking Through Eye-Tracking for Student-LLM Coding Interactions
Sqalli, M. T. (2025) · Biannual Conference of the Italian SIGCHI Chapter (CHItaly) · doi:10.1145/3750069.3750397 · Cited by 3
“The strong correlation between linguistic features and cognitive load metrics provides empirical support for developing domain-oriented dialogue systems that consider both linguistic and cognitive factors [14].”
Dialogue system based on spoken conversations with access to an unstructured knowledge base
Masliukhin, S. M. (2023) · Scientific and technical journal of information technologies mechanics and optics · doi:10.17586/2226-1494-2023-23-1-88-95 · Cited by 3
SP-NLG: A Semantic-Parsing-Guided Natural Language Generation Framework
Li, T., Zhang, S., & Li, Z. (2023) · Electronics · doi:10.3390/electronics12081772 · Cited by 3
Overview of the Tenth Dialog System Technology Challenge: DSTC10
Yoshino, K., Chen, Y.-N., Crook, P., Kottur, S., Li, J., Hedayatnia, B., Moon, S., Fei, Z., Li, Z., Zhang, J., et al. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3293030 · Cited by 11
“Recently, more public data sets and benchmarks have become available for dialogue research on task-oriented conversations in various domains [21], [22], [23], [24].”
BackgroundMethodSun, M., Gao, Q., Mou, Y., Dong, G., Liu, R., & Guo, W. (2023) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw59220.2023.10193387 · Cited by 8
Towards Zero-Shot Multilingual Transfer for Code-Switched Responses
Wu, T.-W., Zhao, C., Chang, E., Shi, Y., Chuang, P., Chandra, V., & Juang, B. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.417 · Cited by 5
“It is a multilingual extension of an English ToD dataset for DST, MultiWoZ (Budzianowski et al., 2018).”
MethodKim, Y., Hwang, Y., Shin, J., Bae, H., & Jung, K. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.249 · Cited by 8
“Compared to most existing TOD datasets (Budzianowski et al., 2018; Shah et al., 2018), the Schema-Guided Dialogue (SGD) dataset is designed to share information between domains or slots.”
BackgroundMethodKnowledge-grounded Dialog State Tracking
Yu, D., Wang, M., Cao, Y., Shafey, L. E., Shafran, I., & Soltau, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.250 · Cited by 5
Guo, A., Ohashi, A., Chiba, Y., Tsunomori, Y., Hirai, R., & Higashinaka, R. (2023) · IEEE RO-MAN · doi:10.1109/ro-man57019.2023.10309654 · Cited by 3
“We built our personality-aware NLG module using the popular multi-domain Wizard-of-Oz (MultiWOZ), which is a large-scale human-human dialogue corpus spanning several tourist information consulting tasks [4].”
MethodInfluentialSpeech Act Classification in Collaborative Robotics
Kaszuba, S., Sabbella, S. R., Leotta, F., & Nardi, D. (2023) · IEEE RO-MAN · doi:10.1109/ro-man57019.2023.10309308 · Cited by 3
Dial-M: A Masking-based Framework for Dialogue Evaluation
Dey, S., & Desarkar, M. S. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.7 · Cited by 6
“For example, if the output of database queries (like system-act annotation in MultiWOZ (Budzianowski et al., 2018)) is converted into a natural sentence and used as the condition, Dial-M can be utilized for task-oriented conversation.”
MethodLiu, L., Li, X., & Yang, F. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29830 · Cited by 3
Hart, J., AuBuchon, J., Mason, S. A., & Kuttal, S. K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60606-9_20 · Cited by 3
Graph-based Dynamic Domain Selection for Dialogue State Tracking
Song, S., Zhang, J., Huang, H., Xie, H., Su, H., Li, M., He, Z., Li, Y., & Fang, R. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650239 · Cited by 3
“We evaluated our model on the MultiWOZ 2.0 [15], Multi-WOZ 2.1 [17], and MultiWOZ 2.4 [16] datasets.”
MethodInfluentialImproving Pointer Network based Dialogue State Tracking via Dual Hierarchical Selective Augmentation
Song, S., Xie, H., Su, H., Huang, H., Li, M., He, Z., Li, Y., & Fang, R. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650263 · Cited by 3
“We use MultiWOZ 2.0 [19] and MultiWOZ 2.4 [20] datasets, which are composed of multi-domain dialogues.”
MethodInfluentialDense Paraphrasing for multimodal dialogue interpretation
Tu, J., Rim, K., Ye, B., Lai, K., & Pustejovsky, J. D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1479905 · Cited by 11
The 2nd Futuredial Challenge: Dialog Systems With Retrieval Augmented Generation (Futuredial-RAG)
Cai, Y., Chen, S., Wu, Y., Huang, Y., Feng, J., & Ou, Z. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832299 · Cited by 5
“In recent years, significant progress has been made in building dialog systems with the breakthrough of deep learning methods and the large amount of conversational data being made available for system development [1, 2, 3, 4].”
A Hierarchical Tracker for Multi-Domain Dialogue State Tracking
Li, J., Zhu, S., & Yu, K. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053248 · Cited by 3
“1 [5] show that our approach can achieve stateof-the-art joint goal accuracy, and also enhance the existing DST models significantly.”
BackgroundMulti-Domain Dialogue State Tracking with Hierarchical Task Graph
Shen, T., & Wang, X. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206790 · Cited by 3
“Therefore, DST-HTG achieves both state-of-the-art joint accuracy and slot accuracy on MultiWOZ 2.0 dataset.”
BackgroundInfluentialMemory Attention Neural Network for Multi-domain Dialogue State Tracking
Xu, Z., Chen, Z., Chen, L., Zhu, S., & Yu, K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-60450-9_4 · Cited by 3
Modeling ASR Ambiguity for Neural Dialogue State Tracking
Pal, V., Guillot, F., Shrivastava, M., Renders, J.-M., & Besacier, L. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1783 · Cited by 3
“[10] predicts dialogue states from utterances and schema graphs containing slot relations in edges to achieve state-of-the-art result of MultiWoz 2.0 [11] and MultiWoz 2.1 datasets [12].”
BackgroundInfluentialRobust dialog state tracker with contextual-feature augmentation
Zhang, X., Zhao, X., & Tan, T. (2020) · Applied Intelligence · doi:10.1007/s10489-020-01991-y · Cited by 3
“Considering that building large-scale task-oriented datasets for new domains is costly and time-consuming, we pay attention to the state tracking ability of the model for execution on an unseen domain [4].”
BackgroundSummarizing Behavioral Change Goals from SMS Exchanges to Support Health Coaches
Gupta, I., Eugenio, B. D., Ziebart, B. D., Liu, B., Gerber, B. S., & Sharp, L. K. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.31 · Cited by 11
“As a result, numerous DA schemas have emerged over time (Core and Allen, 1997; Bunt, 2009; El Asri et al., 2017; Budzianowski et al., 2018).”
BackgroundInfluentialPredict-Then-Decide: A Predictive Approach for Wait or Answer Task in Dialogue Systems
Lin, Z., Cui, S., Li, G., Kang, X., Ji, F. Q., Li, F.-L., Zhao, Z., Chen, H., & Zhang, Y. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3110145 · Cited by 3
“Recent years research on task-oriented dialogue systems mainly concentrates on dialogue states [20] and knowledge integration [6], [21] using pipeline or end to end models.”
BackgroundInfluentialA Neural Language Understanding for Dialogue State Tracking
He, Y., & Tang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-82136-4_44 · Cited by 3
Toward Data-Driven Collaborative Dialogue Systems: The JILDA Dataset
Sucameli, I., Lenci, A., Magnini, B., Speranza, M., & Simi, M. (2021) · Italian Journal of Computational Linguistics · doi:10.4000/ijcol.842 · Cited by 3
“1 have been recently published, addressing annotation errors occurring in the original dataset (Ramadan, Budzianowski, and Gasic 2018; Budzianowski et al. 2018; Eric et al. 2020; Zang et al. 2020).”
InfluentialAudio Embeddings Help to Learn Better Dialogue Policies
Zorrilla, A. L., Torres, M. I., & Cuayáhuitl, H. (2021) · Lincoln Repository (University of Lincoln) · doi:10.1109/asru51503.2021.9688296 · Cited by 3
TANet: Thread-Aware Pretraining for Abstractive Conversational Summarization
Yang, Z., Wang, C., Tian, Z., Wu, W., & Li, Z. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.198 · Cited by 4
An effective context‐focused hierarchical mechanism for task‐oriented dialogue response generation
Zhao, M., Jiang, Z., Wang, L., Li, R., Lu, X., Hu, Z., & Chen, D. (2022) · Computational Intelligence · doi:10.1111/coin.12544 · Cited by 3
srVPA: A Multi-Domain Conversational Service Recommendation Approach
Liu, M., Yu, D., Tu, Z., & Wang, Z. (2022) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws55610.2022.00037 · Cited by 3
“Therefore, CrossWOZ-RULE modifies the dialogue policy module based on the rule-based dialogue policy in MultiWOZ [13] to adapt to the CrossWOZ dataset.”
MethodA Survey on Intelligent Question and Answer Systems
Guo, X., Zhao, B., & Ning, B. (2022) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-23902-1_7 · Cited by 4
HVCMM: A Hybrid-View Abstractive Model of Automatic Summary Generation for Teaching
Zhang, H., Liu, S., Li, Y., Ren, W., Tong, H., Cao, J., & Zhou, Q. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3269013 · Cited by 4
“…the field of patents; the BookSum dataset [33] in the field of long-form narrative literature; the SummScreen dataset [34] in the script field; the AMI dataset [35] and the ICSI dataset [36] in the conference field; SAMSum dataset [37]; and the cross-domain task-based dialog dataset Multi-WOZ [38].”
BackgroundLi, B. Z., Eisner, J. M., Pauls, A., & Thomson, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.854 · Cited by 3
Multijugate Dual Learning for Low-Resource Task-Oriented Dialogue System
Li, S., Zhang, X., Zheng, Y., Li, L., & Qiu, X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.702 · Cited by 3
Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances
Kim, T., Lee, Y., Yoon, H., Kang, P., Bang, J., & Kim, M. (2022) · doi:10.18653/v1/2022.seretod-1.1 · Cited by 3
Know Thy Strengths: Comprehensive Dialogue State Tracking Diagnostics
Cho, H., Sankar, C., Lin, C., Sadagopan, K. R., Shayandeh, S., Çelikyılmaz, A., May, J., & Beirami, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.391 · Cited by 11
“As shown in the bar chart on the top side of Figure 1, recent state-of-the-art approaches in DST leaderboards such as MultiWOZ (Budzianowski et al., 2018) are compared to one another on a single metric that provides limited information and their modes of failure.”
BackgroundMethodInfluentialStructure-Aware Dialogue Modeling Methods for Conversational Semantic Role Labeling
Wu, H., Xu, K., & Song, L. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3331576 · Cited by 3
“generation [2], [3], [4], task-oriented dialogue modeling [5], [6] and conversational question answering [7], [8].”
BackgroundDialogue Evaluation with Offline Reinforcement Learning
Lubis, N., Geishauser, C., Lin, H.-C., van Niekerk, C., Heck, M. C., Feng, S., & Gašić, M. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.46 · Cited by 5
Self-citationHuang, C., Qin, P., Lei, W., & Lv, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.670 · Cited by 15
“…unnatural conversations (Lipani et al., 2021; Sekuli´c et al., 2022), constructing the dataset still requires a significant amount of human labor (Budzianowski et al., 2018; Dalton et al., 2020; Adlakha et al., 2022), especially when evaluating specific features of the system that are absent…”
BackgroundMethodInfluentialDo as I Demand, Not as I Say: A Dataset for Developing a Reflective Life-Support Robot
Tanaka, S., Yamasaki, K., Yuguchi, A., Kawano, S., Nakamura, S., & Yoshino, K. (2024) · IEEE Access · doi:10.1109/access.2024.3350174 · Cited by 7
“The available dataset is extremely small compared to a typical dialogue corpus [7], [30].”
BackgroundMethodFact-Aware Summarization with Contrastive Learning for Few-Shot Dialogue State Tracking
Feng, S., Su, H., Xie, H., Wu, D., Huang, H., & Silamu, W. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448311 · Cited by 3
“We evaluate our proposed FaS-DST model on the MultiWOZ 2.0 [18] and MultiWOZ 2.1 [19] datasets.”
MethodInfluentialMODE: a multimodal open-domain dialogue dataset with explanation
Yin, H., Lu, P., Li, Z., Sun, B., & Li, K. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05479-x · Cited by 4
“Task-oriented dialogues aim to accomplish specific tasks, such as food ordering, medical appointment scheduling, or travel recommendations [4, 35].”
Zhao, Y., Dastani, M. M., Long, J., Wang, Z., & Wang, S. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00717 · Cited by 8
“We also performed further validation on the Multiwoz dataset (Budzianowski et al., 2018).”
Span Attention for Entity-Consistent Task-Oriented Dialogue Response Generation
Chen, J., Dong, X., Xie, W., Gong, T., Wang, F. L., & Hao, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887791 · Cited by 3
“The effectiveness of the proposed TOD model is validated using three publicly available task-oriented dialogue datasets: CamRest [18], SMD [17], and MWOZ [19].”
Metadata
- Authors
- Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., & Gašić, M.
- Affiliations
- University of Cambridge (GB)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 5016-5026
- OpenAlex
- W2891732163
- Cited by
- 1,028
- References
- 48
- Citations by year
- 2026: 39 · 2025: 66 · 2024: 86 · 2023: 209 · 2022: 186 · 2021: 196 · 2020: 181 · 2019: 64 · 2018: 1
- FWCI
- 45.11
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/M018946/1, RG80792)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- MultiWOZ, task-oriented dialogue, multi-domain dialogue, dialogue modeling, dialogue dataset, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.