MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
Open accessThe DOI matches its Crossref record; 46 of the 48 references it cites were found in a registry or an index.
Abstract
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica Gašić. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some tasks have clear ground truth answers and can be scored with exact correctness (Du et al., 2024; Dunn et al., 2017), while others are open-ended (dialogue (Budzianowski et al., 2018), summarization (Maharana et al., 2024), preference following (Zhao et al., 2025c)) where multiple outputs are acceptable and reference-based scoring is not reliable.”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The former types of benchmarks are static and only test the conversational agent on pre-collected conversation trajectories[4, 3, 2].”
From $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 4×
“Among them, the first two metrics relate to the dialogue task completion - whether the system has provided an appropriate entity (In-form rate) and then answered all the requested attributes (Success rate); while fluency is measured via BLEU score (Papineni et al., 2002).”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“(2017) which frames the dialogue as a context to response mapping problem, a sequence-to-sequence model (Sutskever et al., 2014) is augmented with a belief tracker and a discrete database accessing component as additional features to inform the word decisions in the decoder.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“In the case of MultiWOZ dataset, the LSTM cell serving as a decoder and an encoder achieved the highest score with the global type of attention (Bahdanau et al., 2014).”
MethodMeasuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“As this requires a multi-annotator metric over a multi-label task, we used Fleiss’ kappa metric (Fleiss, 1971) per single dialogue act.”
MethodA Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926 · cited 2×
“Due to the lack of an explicit goal in the conversation, recent studies have shown that systems trained with this type of corpus not only struggle in generating consistent and diverse responses (Li et al., 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundVinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400 · cited 2×
“Although previous work (Vinyals and Le, 2015) has shown that a large learning system can learn to generate interesting responses from these corpora, the lack of grounding conversations onto an existing knowledge base or APIs limits the usability of developed systems.”
BackgroundLiu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1230 · Cited by 1,228 · cited 3×
“, 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundPOMDP-Based Statistical Spoken Dialog Systems: A Review
Young, S., Gasic, M., Thomson, B., & Williams, J. D. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2225812 · Cited by 882 · cited 4×
“These problems can either be addressed separately (Young et al., 2013), or jointly in an end-to-end fashion (Bordes et al., 2017; Wen et al., 2017; Li et al., 2017).”
BackgroundInfluentialSelf-citationSemantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874 · cited 5×
“To make a fair comparison, we constrained our dataset to only the restaurant sub-domain which contains around 25 k dia-Cam676 Table 5 presents the results on two metrics: BLEU score (Papineni et al., 2002) and slot error rate (SER) (Wen et al., 2015).”
BackgroundMethodInfluentialSelf-citationThe ATIS spoken language systems pilot corpus
Hemphill, C. T., Godfrey, J. J., & Doddington, G. R. (1990) · Proceedings of the workshop on Speech and Natural Language - HLT '90 · doi:10.3115/116580.116613 · Cited by 790 · cited 4×
“One of the earliest dataset collected in this fashion is the ATIS corpus (Hemphill et al., 1990), where conversations between a client and an airline help-desk operator were recorded.”
BackgroundMethodResultInfluential
Research works printed but not linked: 7
- Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. Proceedings of ICLR .
- Paweł Budzianowski, I˜ nigo Casanueva, Bo-Hsiang Tseng, and Milica Gaˇ si´ c. 2018. Towards endto-end multi-domain dialogue modelling. Tech. Rep. CUED/F-INFENG/TR.706, University of Cambridge, Engineering Department .
- Seokhwan Kim, Luis Fernando D'Haro, Rafael E Banchs, Jason D Williams, Matthew Henderson, and Koichiro Yoshino. 2016. The fifth dialog state tracking challenge. In Spoken Language Technology Workshop (SLT), 2016 IEEE , pages 511-517. IEEE.
- Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-end taskcompletion neural dialogue systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , volume 1, pages 733-743.
- David R. Traum. 1999. Foundations of Rational Agency , chapter Speech Acts for Dialogue Agents. Springer.
- Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gaˇ si´ c, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A networkbased end-to-end trainable task-oriented dialogue system. EACL .
- Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference , pages 404-413.
Cited by
Items 401–500 of 1,004
Acoustics Based Intent Recognition Using Discovered Phonetic Units for Low Resource Languages
Gupta, A., Li, X., Rallabandi, S. K., & Black, A. W. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9415112 · Cited by 14
“To work with Romance languages, we create an intent recognition dataset from the MultiWoz dataset [14].”
MethodOverview of the Eighth Dialog System Technology Challenge: DSTC8
Kim, S., Galley, M., Gunasekara, C., Lee, S., Atkinson, A., Peng, B., Schulz, H., Gao, J., Li, J., Adada, M., et al. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3078368 · Cited by 17
“Automatic metrics: For each incomplete test dialogue, a set of 128 complete single-domain MultiWOZ [17] dialogs is provided to the model, which is then asked to respond to the incomplete test dialog.”
InfluentialFast and Scalable Dialogue State Tracking with Explicit Modular Decomposition
Wang, D., Lin, C., Liu, Q., & Wong, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.27 · Cited by 14
“Dataset We evaluate our model performance based on the MultiWoZ 2.0 dataset (Budzianowski et al., 2018), which contains 10, 000 dialogues of 7 domains and 35 domain-slot pairs.”
MethodInfluentialGehrmann, S., Strobelt, H., Rush, A. M., et al. (2019) · doi:10.18653/v1/p19-3 · Cited by 14
Meta-Learning Triplet Network with Adaptive Margins for Few-Shot Named Entity RecognitionPreprint
Han, C., Zhu, R., Jun, K., Chen, F., Li, X., Gao, M., Cao, X., & Wu, W. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.07739 · Cited by 14
“Media), Restaurant (Review) and Multiwoz (Dialogue), respectively.”
MethodDongre, V., Yang, X., Acikgoz, E. C., Dey, S., Tür, G., & Hakkani-Tur, D. Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.00927 · Cited by 19
“The agent is tested on multi-step tasks in common household environments using Alfworld (Shridhar et al., 2020b), tasked with making reservations in the MultiWoz dialogue setup (Budzianowski et al., 2018), and instructed to purchase products in Webshop (Yao et al., 2022a).”
Where to Go for the Holidays: Towards Mixed-Type Dialogs for Clarification of User Goals
Liu, Z., Xu, J., Lei, Z., Wang, H., Niu, Z.-Y., & Wu, H. J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.73 · Cited by 22
Simulating Follow-Up Questions in Conversational Search
Kiesel, J., Gohsen, M., Mirzakhmedova, N., Hagen, M., & Stein, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-56060-6_25 · Cited by 19
BIPED: Pedagogically Informed Tutoring System for ESL Education
Kwon, S., Kim, S.-J., Park, M., Lee, S., & Kim, K. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.186 · Cited by 27
AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction
Wang, H., Wang, R., Xue, B., Xia, H., Cao, J., Liu, Z., Pan, J. Z., & Wong, K. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.856 · Cited by 35
“To maximize the authenticity of user instructions and minimize human efforts, we prioritize using existing task-oriented dialogue datasets (Rastogi et al., 2020; Budzianowski et al., 2018).”
MethodLarge Language Models as User-Agents For Evaluating Task-Oriented-Dialogue Systems
Kazi, T., Lyu, R., Zhou, S., Hakkani-Tür, D. Z., & Tür, G. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832298 · Cited by 13
“We use MultiWOZ [18], which is a well-known dataset of human-to-human written TOD conversations across 7 different domains.”
Henderson, M., Vulić, I., Casanueva, I., Budzianowski, P., Gerz, D., Coope, S., Spithourakis, G., Wen, T.-H., Mrkšić, N., & Su, P.-H. (2019) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d19-3031 · Cited by 16
“As a consequence, despite recent data collection efforts to enable training of task-oriented systems across multiple domains (El Asri et al., 2017; Budzianowski et al., 2018), annotated datasets are still few and far between, as”
BackgroundMethodInfluentialSelf-citationGrounding Dialogue Systems via Knowledge Graph Aware Decoding with Pre-trained Transformers
Chaudhuri, D., Rony, M. R. A. H., & Lehmann, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77385-4_19 · Cited by 25
HyKnow: End-to-End Task-Oriented Dialog Modeling with Hybrid Knowledge Management
Gao, S., Takanobu, R., Peng, W., Liu, Q., & Huang, M. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.139 · Cited by 13
Zhao, Y., Wang, Z., Zhu, C., & Wang, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.354 · Cited by 18
GAR-Net: A Graph Attention Reasoning Network for conversation understanding
Xu, H., Yuan, Z., Zhao, K., Xu, Y., Zou, J., & Gao, K. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.108055 · Cited by 14
Zhao, L., Zheng, F., Zeng, W., He, K., Xu, W., Jiang, H., Wu, W., & Wu, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.357 · Cited by 23
Wang, Q., Ding, L., Cao, Y., Zhan, Y., Lin, Z., Wang, S., Tao, D., & Guo, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.114 · Cited by 23
Cross-domain coreference modeling in dialogue state tracking with prompt learning
Xu, H.-D., Mao, X.-L., Yang, P., Sun, F., & Huang, H. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111189 · Cited by 13
Dialog Acts for Task Driven Embodied Agents
Gella, S., Padmakumar, A., Lange, P. L., & Hakkani-Tür, D. Z. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.13 · Cited by 13
Task Conditioned BERT for Joint Intent Detection and Slot-Filling
Tavares, D., Azevedo, P., Semedo, D., Sousa, R., & Magalhães, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-49008-8_37 · Cited by 13
Towards Universal Dialogue Act Tagging for Task-Oriented Dialogues
Paul, S., Goel, R., & Hakkani-Tür, D. Z. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1866 · Cited by 13
Wizard of Oz Interface to Study System Initiative for Conversational Search
Avula, S., & Arguello, J. (2020) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3343413.3377941 · Cited by 14
“In intermittent turn-by-turn conversations, users are given their own search interfaces or lookup engines to search for information to contribute to the dialog [5, 12, 29].”
BackgroundChartDialogs: Plotting from Natural Language Instructions
Shao, Y., & Nakashole, N. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.328 · Cited by 18
“Recently, a multi-domain goal-oriented dataset covering restaurant, attraction, hospital, police, hotel, taxi and train domains was introduced in (Budzianowski et al., 2018).”
BackgroundMethodGRTr: Generative-Retrieval Transformers for Data-Efficient Dialogue Domain Adaptation
Shalyminov, I., Sordoni, A., Atkinson, A., & Schulz, H. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3074779 · Cited by 14
“For evaluation purposes, the challenge organizers also provide M ulti WOZ [33], another multi-domain, multi-task dialogue dataset.”
BackgroundBiomedical Data-to-Text Generation via Fine-Tuning Transformers
Yermakov, R., Drago, N., & Ziletti, A. (2021) · International Conference on Natural Language Generation · doi:10.18653/v1/2021.inlg-1.40 · Cited by 21
UniDS: A Unified Dialogue System for Chit-Chat and Task-oriented Dialogues
Zhao, X., He, B., Wang, Y., Li, Y., Mi, F., Liu, Y., Jiang, X., Liu, Q., & Chen, H. (2022) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2022.dialdoc-1.2 · Cited by 23
Multi-Domain Dialogue State Tracking based on State GraphPreprint
Zeng, Y., & Nie, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11137 · Cited by 13
“We use two publicly available datasets MultiWOZ 2.0 (Budzianowski et al., 2018) and MultiWOZ 2.1 (Eric et al., 2019)”
InfluentialKiseleva, J., Li, Z., Aliannejadi, M., Mohanty, S., ter Hoeve, M., Burtsev, M., Skrynnik, A., Zholus, A., Panov, A. I., Srinet, K., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06536 · Cited by 16
“…by suggesting competitions, such as the Dialog System Technology Challenge (DSTC-8) [Kim et al., 2019]; benchmarks and experimental platforms, e.g., Convlab, which offers the annotated MultiWOZ dataset [Budzianowski et al., 2018] and associated pre-trained reference models [Lee et al., 2019].”
BackgroundMTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language ModelsPreprint
Wang, P., Wu, Y., Wang, Z., Liu, J., Song, X., Peng, Z., Deng, K., Zhang, C., Wang, J., Peng, J., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.11710 · Cited by 13
“The multi-turn dialogue datasets include MultiWOZ (Budzianowski et al., 2018), SGD (Rastogi et al., 2020b), TaskMaster (Byrne et al., 2019) and MetaLWOZ (Shalyminov et al., 2020).”
MethodLee, H., & Jeong, O. (2023) · Sensors · doi:10.3390/s23020685 · Cited by 15
Improving Dialogue State Tracking by Discerning the Relevant Context
Sharma, S., Choubey, P. K., & Huang, R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1057 · Cited by 12
“We also use dialogues from restaurant domain in MultiWoZ 2.0 dataset (Budzianowski et al., 2018) for secondary evaluation.”
MethodInfluentialKeeping Up Appearances: Computational Modeling of Face Acts in Persuasion Oriented Discussions
Dutt, R., Joshi, R., & Rosé, C. P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.605 · Cited by 24
“It also sets itself apart from other goal-oriented conversations such as restaurant reservations and cab booking (Budzianowski et al., 2018) since in those cases the hearer is obligated to address what might otherwise come across as an FTA (request/ booking), and thus in those cases non-compliance…”
BackgroundYang, P., Huang, H., & Mao, X.-L. (2021) · doi:10.18653/v1/2021.acl-long.193 · Cited by 12
“MultiWOZ2.”
BackgroundInfluentialDORA: Towards policy optimization for task-oriented dialogue system with efficient context
Jeon, H., & Lee, G. G. (2021) · Computer Speech & Language · doi:10.1016/j.csl.2021.101310 · Cited by 12
Attend, Memorize and Generate: Towards Faithful Table-to-Text Generation in Few Shots
Zhao, W., Ye, L., Wan, Y., & Yu, P. S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.347 · Cited by 12
“…et al., 2009), biographies (Lebret et al., 2016; Wang et al., 2018), restaurant descriptions (Novikova et al., 2017), task-oriented conversations (Budzianowski et al., 2018; Williams et al., 2013) as well as health-care descriptions (DiMarco et al., 2007; Hasan and 1 All the source code and…”
BackgroundPrompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection
Varshney, V., Patidar, M., Kumar, R., Vig, L., & Shroff, G. M. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.84 · Cited by 20
“In our experiments, we have taken K =0 and K =1 to create the dataset for intent detection from SGD and MWOZ datasets.”
InfluentialVector Representations of Idioms in Conversational Systems
Adewumi, T., Liwicki, F., & Liwicki, M. (2022) · Sci · doi:10.3390/sci4040037 · Cited by 16
NatCS: Eliciting Natural Customer Support Dialogues
Gung, J., Moeng, E., Rose, W., Gupta, A., Zhang, Y., & Mansour, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.613 · Cited by 24
MetaASSIST: Robust Dialogue State Tracking with Meta Learning
Ye, F., Wang, X., Huang, J., Li, S., Stern, S., & Yılmaz, E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.76 · Cited by 12
Xie, H., Su, H., Song, S., Huang, H., Zou, B., Deng, K., Lin, J., Zhang, Z., & He, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.56 · Cited by 12
“The MultiWOZ dataset includes 8,438 multi-turn dialogues in training set with an average of 13.5 turns per dialogue.”
MethodInfluentialContextual Dynamic Prompting for Response Generation in Task-oriented Dialog Systems
Swamy, S., Tabari, N., Chen, C., & Gangadharaiah, R. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.226 · Cited by 12
Dialogue summarization enhanced response generation for multi-domain task-oriented dialogue systems
Wang, L., Zhao, M., Ji, H., Jiang, Z., Li, R., Hu, Z., & Lu, X. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103668 · Cited by 12
Effective Data Augmentation Approaches to End-to-End Task-Oriented Dialogue
Quan, J., & Xiong, D. (2019) · International Conference on Asian Language Processing · doi:10.1109/ialp48816.2019.9037690 · Cited by 12
“, via talking to myself [26] or MultiWOZ [27].”
BackgroundHuman-Human Health Coaching via Text Messages: Corpus, Annotation, and Analysis
Gupta, I., Eugenio, B. D., Ziebart, B. D., Baiju, A., Liu, B., Gerber, B. S., Sharp, L. K., Nabulsi, N. A., & Smart, M. H. (2020) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2020.sigdial-1.30 · Cited by 26
“Moreover, for data collection, pre-defined scenarios are given to the users and thus, the users’ responses are not as spontaneous as they would be in a real-life situation (Asri et al., 2017; Budzianowski et al., 2018).”
BackgroundDialogue State Tracking with Multi-Level Fusion of Predicted Dialogue States and Conversations
Zhou, J., Wu, H., Lin, Z., Li, G., & Zhang, Y. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.24 · Cited by 12
Is a Wizard-of-Oz Required for Robot-Led Conversation Practice in a Second Language?
Engwall, O., Lopes, J., & Cumbal, R. (2022) · International Journal of Social Robotics · doi:10.1007/s12369-021-00849-8 · Cited by 12
Stabilized In-Context Learning with Pre-trained Language Models for Few Shot Dialogue State Tracking
Chen, D., Qian, K., & Zhou, Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.115 · Cited by 21
Dialogue Act Classification in Team Communication for Robot Assisted Disaster Response
Anikina, T., & Kruijff‐Korbayová, I. (2019) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w19-5946 · Cited by 12
Adewumi, T., Adeyemi, M., Aremu, A., Peters, B., Buzaaba, H., Oyerinde, S., Rufai, A. M., Ajibade, B., Gwadabe, T., Traore, M. M. K., et al. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191208 · Cited by 20
“MultiWOZ is a collection of human-human written conversations that span multiple domains and topics.”
BackgroundMethodInfluentialAn emotion-sensitive dialogue policy for task-oriented dialogue system
Zhu, H., Wang, X., Wang, Z., & Xv, K. (2024) · Scientific Reports · doi:10.1038/s41598-024-70463-x · Cited by 11
“It covers the entire MultiWOZ dataset, which contains over 10,000 dialogues.”
A Survey on the Recent Advancements in Human-Centered Dialog Systems
Oruche, R., Goruganthu, S. K., Akula, R., Cheng, X., Goni, M. A., Shibo, B. W., Kee, K. F., Zampieri, M., & Calyam, P. P. (2025) · ACM Computing Surveys · doi:10.1145/3729220 · Cited by 17
MUSE: A Multimodal Conversational Recommendation Dataset with Scenario-Grounded User Profiles
Wang, Z., Yang, X., Liu, Y., Feng, S., Wang, D., & Zhang, Y. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.58 · Cited by 24
Khosla, S., Vashishth, S., Lehman, J. F., & Rosé, C. P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.626 · Cited by 11
A Comparative Study on Schema-Guided Dialogue State Tracking
Cao, J., & Zhang, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.62 · Cited by 11
“Among various extended versions for MultiWOZ dataset (2.0—2.3, Budzianowski et al., 2018; Eric et al., 2020; Zang et al., 2020; Han et al., 2020) , besides rectifying the annotation errors, M ULTI WOZ 2.2 also introduced the schema-guided annotations, which covers 8 domains, 19 intents, 36 slots.”
BackgroundInfluentialWould you like to tell me more? Generating a corpus of psychotherapy dialogues
Mousavi, S. M., Cervone, A., Danieli, M., & Riccardi, G. (2021) · doi:10.18653/v1/2021.nlpmc-1.1 · Cited by 19
“…dialogue agents are either acquiring user interaction data via user simulators and hand-designed policies (Li et al., 2016), or to collect large sets of human-human conversations in different user-agnostic settings (Budzianowski et al., 2018; Gopalakrishnan et al., 2019; Zhang et al., 2018).”
BackgroundMCIC: Multimodal Conversational Intent Classification for E-commerce Customer Service
Yuan, S., Shen, X., Zhao, Y., Liu, H., Yan, Z., Liu, R., & Chen, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17120-8_58 · Cited by 11
“MultiWOZ [16] is a large-scale multi-domain dataset that consists of around 10K crowd-sourced human-to-human dialogues with 13 intent types.”
BackgroundA machine learning based framework to identify unseen classes in open-world text classification
Parmar, J., Chouhan, S. S., & Raychoudhury, V. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103214 · Cited by 11
Łajewska, W., & Balog, K. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615132 · Cited by 14
“Research on conversational response generation has attracted a lot of attention in task-oriented dialogue systems [2, 19, 24], question answering [1], open-domain chatbots [10, 33], and most recently in conversational information seeking, as part of TREC CAsT’22 [21].”
BackgroundLAD: Language Models as Data for Zero-Shot Dialog
Mehri, S., Altün, Y., & Eskénazi, M. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.55 · Cited by 32
Wu, Q., Gung, J., Shu, R., & Zhang, Y. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.24 · Cited by 23
Fast Domain Adaptation for Goal-Oriented Dialogue Using a Hybrid Generative-Retrieval Transformer
Shalyminov, I., Sordoni, A., Atkinson, A., & Schulz, H. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053599 · Cited by 11
“Evaluation results on MetaLWOz and MultiWoz are shown in Tables 1 and 2, respectively.”
MethodInfluentialCorrective Guidance and Learning for Dialogue Management
Rohmatillah, M., & Chien, J. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482333 · Cited by 11
“0 [3], known as a multidomain, multi-intent task-oriented dialog corpus [33] that contained 7 domains, 13 intents, 25 slot types, 10,483 dialog sessions, and 71,544 dialog turns.”
BackgroundDIRECT: Direct and Indirect Responses in Conversational Text Corpus
Takayama, J., Kajiwara, T., & Arase, Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.170 · Cited by 14
“1 (Budzianowski et al., 2018; Eric et al., 2020) and collected pragmatic paraphrases using crowdsourcing.”
BackgroundA persona aware persuasive dialogue policy for dynamic and co-operative goal setting
Tiwari, A., Saha, T., Saha, S., Sengupta, S., Maitra, A., Ramnani, R., & Bhattacharyya, P. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.116303 · Cited by 11
Kim, Y.-M., Lee, T., & Na, S.-O. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03400-y · Cited by 11
“In addition to online shopping [31], e-commerce [32], and automatic diagnosis datasets [33], a large-scale multidomain dataset was recently released for seven different domains (attraction, hospital, police, hotel, restaurant, taxi, and train) [34].”
BackgroundDey, S., Kummara, R., & Desarkar, M. S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.35 · Cited by 11
Huang, Z., Li, F., Yao, J., & Chen, Z.-G. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-09328-9 · Cited by 11
“To verify the effectiveness of our MGCRL model, we mainly evaluated it on two mainstream datasets: Multi-WOZ 2.0 [61] and MultiWOZ 2.1 [62].”
InfluentialTable-to-Dialog: Building Dialog Assistants to Chat With People on Behalf of You
Haihong, E., Zhan, Z., & Song, M. (2020) · IEEE Access · doi:10.1109/access.2020.2998432 · Cited by 12
“MD-DAS: a multi-domain dialogue architecture [14] for Context-to-Text generation encodes the policy implicitly and generates response from the hidden representation and context.”
BackgroundMethodInfluentialPrabhumoye, S., Li, M., Urbanek, J., Dinan, E., Kiela, D., Weston, J., & Szlam, A. D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02878 · Cited by 13
“, 2014), taxi, train, and hotel (Budzianowski et al., 2018) or trip (El Asri et al.”
BackgroundMethodInfluentialDiversifying Task-oriented Dialogue Response Generation with Prototype Guided ParaphrasingPreprint
Lippe, P., Ren, P., Haned, H., Voorn, B., & de Rijke, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.03391 · Cited by 13
Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language ModelsPreprint
Samarinas, C., Promthaw, P., Nijasure, A., Zeng, H., Killingback, J., & Zamani, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.14772 · Cited by 16
Evaluating Bias in Spoken Dialogue LLMs for Real-World Decisions and Recommendations
Wu, Y., Wang, T., Peng, Y., Chao, Y.-W., Zhuang, X., Wang, X., Yin, S., & Ma, Z. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461562 · Cited by 13
Learning Goal-oriented Dialogue Policy with opposite Agent Awareness
Zhang, Z., Liao, L., Zhu, X., Chua, T., Liu, Z., Huang, Y., & Huang, M. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.16 · Cited by 24
MPDD: A Multi-Party Dialogue Dataset for Analysis of Emotions and Interpersonal Relationships.
Chen, Y., Huang, H., & Chen, H. (2020) · Language Resources and Evaluation · Cited by 10
End-to-End Task-Oriented Dialog Modeling With Semi-Structured Knowledge Management
Gao, S., Takanobu, R., Bosselut, A., & Huang, M. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3153255 · Cited by 12
“In terms of the task completion rate, we measure whether the system provides correct entities (Inform rate) and answers all the requested information (Success rate) in a dialog, following MultiWOZ [20].”
BackgroundMethodAudio Embedding-Aware Dialogue Policy Learning
Zorrilla, A. L., Torres, M. I., & Cuayáhuitl, H. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3225658 · Cited by 10
“Recent dialogue corpora released in the last few years (e.g. MultiWOZ [48], STAR [49] or SGD [50]) have focused on text based dialogue modelling and none include audio.”
BackgroundGao, J., Cui, J., Wu, H., Xiang, L., Zhao, H., Li, X., Fang, M., Yang, Y., & He, Z. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130135 · Cited by 10
Naik, R., Srinivasan, A., Agarwal, S., & He, E. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-0068-9_15 · Cited by 11
Zaman, E. A. K., Isa, N. A. N. M., Jawaddi, S. N. A., & Ismail, A. B. (2026) · Soft Computing · doi:10.1007/s00500-026-11411-x · Cited by 11
“This study utilizes the MultiWOZ 2.2 dataset (Budzianowski et al. 2018), obtained from the Huggingface repository.”
InfluentialHou, Z., Liu, B., Zhao, R., Ou, Z., Liu, Y., Chen, X., & Zheng, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.238 · Cited by 10
Data Augmentation for Copy-Mechanism in Dialogue State Tracking
Song, X., Zang, L., & Hu, S. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77961-0_59 · Cited by 10
Goal Awareness for Conversational AI: Proactivity, Non-collaborativity, and Beyond
Deng, Y., Lei, W., Huang, M., & Chua, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-tutorials.1 · Cited by 22
“…including open-domain dialogue (ODD) systems (Zhang et al., 2018a; Li et al., 2017; Roller et al., 2021), task-oriented dialogue (TOD) systems (Budzianowski et al., 2018; Lei et al., 2018; Su et al., 2022), conversational question answering (CQA) systems (Choi et al., 2018; Reddy et al.,…”
BackgroundPluDG: enhancing task-oriented dialogue system with knowledge graph plug-in module
Dong, X., & Chen, J. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1707 · Cited by 10
“We evaluate our model on three publicly available benchmark datasets: CamRest ( Wen et al., 2016 ), In-Car Assistant ( Eric & Manning, 2017 ), MultiWOZ 2.1 ( Budzianowski et al., 2018 ).”
MethodInfluentialDiSTRICT: Dialogue State Tracking with Retriever Driven In-Context Tuning
Venkateswaran, P., Duesterwald, E., & Isahagian, V. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.310 · Cited by 10
Yu, X., Wu, Q., Qian, K., & Zhou, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.759 · Cited by 20
“In our experiments, we apply KRLS on the MultiWoZ (Budzianowski et al., 2018) dataset, to improve performance on tasks such as restaurant booking and hotel reservation.”
BackgroundMethodInfluentialOWI: Open-World Intent Identification Framework for Dialog Based System
Parmar, J., Soni, S., & Chouhan, S. S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66665-1_21 · Cited by 10
Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance
van Niekerk, C., Malinin, A., Geishauser, C., Heck, M. C., Lin, H.-C., Lubis, N., Feng, S., & Gašić, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.623 · Cited by 16
“We first investigate which states to use for the supervised pretraining (Section 3.3): oracle states, i.e., the dialogue state labels from the MultiWOZ corpus, or estimated belief states, e.g., those predicted by a EnD-SetSUMBT model.”
BackgroundInfluentialSelf-citationMA-DST: Multi-Attention Based Scalable Dialog State TrackingPreprint
Kumar, A., Ku, P., Goyal, A. K., Metallinou, A., & Hakkani-Tür, D. Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08898 · Cited by 12
“This enables the model to produce unseen slot values, and it achieves state-of-the art results on the MultiWOZ public benchmark (Budzianowski et al. 2018; Eric et al. 2019).”
InfluentialSPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and GenerationPreprint
He, W., Dai, Y., Yang, M., Sun, J. Q., Huang, F., Si, L., & Li, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.06664 · Cited by 10
“MultiWOZ [5] is one of the most challenging datasets for DST, which contains 10k dialogs spanning 7 distinct domains.”
BackgroundPrescriptive Process Monitoring in Intelligent Process Automation with Chatbot OrchestrationPreprint
Zeltyn, S., Shlomov, S., Yaeli, A., & Oved, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.06564 · Cited by 10
“These datasets contain many different tasks, such as intent prediction [19], slot filling [20], dialogue state tracking [21], and dialog act [22, 23].”
Background$τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024) · arXiv · doi:10.48550/arxiv.2406.12045 · Cited by 9
“The former types of benchmarks are static and only test the conversational agent on pre-collected conversation trajectories[4, 3, 2].”
BackgroundFewJoint: few-shot learning for joint dialogue understanding
Hou, Y., Wang, X., Chen, C., Li, B., Che, W., & Chen, Z. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01604-9 · Cited by 10
“The second method of constructing data is similar to a classical method of dialog data collection, namely the Wizard-of-Oz [6, 16, 29, 75], where the machine learns from annotator-typed conversation logs.”
MethodAURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks
Maben, L. M., Lakshmy, G. G., Radhakrishnan, S., Arora, S., & Watanabe, S. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434725 · Cited by 13
“While large language models have been extensively employed for dialogue state tracking and response generation on text-based benchmarks such as MultiWOZ (Multi-Domain Wizard-of-Oz) [8], their application to spoken dialogue remains relatively underexplored.”
Schema-Guided Paradigm for Zero-Shot Dialog
Mehri, S., & Eskénazi, M. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.52 · Cited by 19
A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slots
Zhang, S., Hu, Y., Wu, Y., Wu, J., Li, Y., Sun, J. Q., Yuan, C., & Wang, X. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.27 · Cited by 9
Challenges to Open-Domain Constituency Parsing
Yang, S., Cui, L., Ning, R., Wu, D., & Zhang, Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.11 · Cited by 20
“In addition, given much research interest in NLP for dialogue (Budzianowski et al., 2018), law (Chalkidis et al., 2019) and review (Oved and Levy, 2021) domains, we manually label constituent structures for five typical domains (i.e., dialogue, forum, law, literature, review), resulting in a test…”
BackgroundAutomated Generation of Human-readable Natural Arabic Text from RDF Data
Touma, R., Hajj, H. M., Hajj, W. E., & Shaban, K. B. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3582262 · Cited by 9
“The model achieved SOTA results on multiple D2T datasets including WebNLG [11], MultiWoz [4], and ToTTo [17] benchmarks.”
MethodDialogue State Distillation Network with Inter-slot Contrastive Learning for Dialogue State Tracking
Xu, J., Song, D., Liu, C., Hui, S. C., Li, F., Ju, Q., He, X., & Jian, X. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26620 · Cited by 9
Parameter-Efficient Fine-Tuning Method for Task-Oriented Dialogue Systems
Mo, Y., Yoo, J., & Kang, S. (2023) · Mathematics · doi:10.3390/math11143048 · Cited by 9
“The evaluation was conducted using the benchmark dataset Multi-WOZ 2.0 [12].”
MethodInfluential
Metadata
- Authors
- Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., & Gašić, M.
- Affiliations
- University of Cambridge (GB)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 5016-5026
- OpenAlex
- W2891732163
- Cited by
- 1,028
- References
- 48
- Citations by year
- 2026: 39 · 2025: 66 · 2024: 86 · 2023: 209 · 2022: 186 · 2021: 196 · 2020: 181 · 2019: 64 · 2018: 1
- FWCI
- 45.11
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/M018946/1, RG80792)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- MultiWOZ, task-oriented dialogue, multi-domain dialogue, dialogue modeling, dialogue dataset, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.