MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
Open accessThe DOI matches its Crossref record; 46 of the 48 references it cites were found in a registry or an index.
Abstract
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica Gašić. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some tasks have clear ground truth answers and can be scored with exact correctness (Du et al., 2024; Dunn et al., 2017), while others are open-ended (dialogue (Budzianowski et al., 2018), summarization (Maharana et al., 2024), preference following (Zhao et al., 2025c)) where multiple outputs are acceptable and reference-based scoring is not reliable.”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The former types of benchmarks are static and only test the conversational agent on pre-collected conversation trajectories[4, 3, 2].”
From $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 4×
“Among them, the first two metrics relate to the dialogue task completion - whether the system has provided an appropriate entity (In-form rate) and then answered all the requested attributes (Success rate); while fluency is measured via BLEU score (Papineni et al., 2002).”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“(2017) which frames the dialogue as a context to response mapping problem, a sequence-to-sequence model (Sutskever et al., 2014) is augmented with a belief tracker and a discrete database accessing component as additional features to inform the word decisions in the decoder.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“In the case of MultiWOZ dataset, the LSTM cell serving as a decoder and an encoder achieved the highest score with the global type of attention (Bahdanau et al., 2014).”
MethodMeasuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“As this requires a multi-annotator metric over a multi-label task, we used Fleiss’ kappa metric (Fleiss, 1971) per single dialogue act.”
MethodA Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926 · cited 2×
“Due to the lack of an explicit goal in the conversation, recent studies have shown that systems trained with this type of corpus not only struggle in generating consistent and diverse responses (Li et al., 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundVinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400 · cited 2×
“Although previous work (Vinyals and Le, 2015) has shown that a large learning system can learn to generate interesting responses from these corpora, the lack of grounding conversations onto an existing knowledge base or APIs limits the usability of developed systems.”
BackgroundLiu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1230 · Cited by 1,228 · cited 3×
“, 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundPOMDP-Based Statistical Spoken Dialog Systems: A Review
Young, S., Gasic, M., Thomson, B., & Williams, J. D. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2225812 · Cited by 882 · cited 4×
“These problems can either be addressed separately (Young et al., 2013), or jointly in an end-to-end fashion (Bordes et al., 2017; Wen et al., 2017; Li et al., 2017).”
BackgroundInfluentialSelf-citationSemantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874 · cited 5×
“To make a fair comparison, we constrained our dataset to only the restaurant sub-domain which contains around 25 k dia-Cam676 Table 5 presents the results on two metrics: BLEU score (Papineni et al., 2002) and slot error rate (SER) (Wen et al., 2015).”
BackgroundMethodInfluentialSelf-citationThe ATIS spoken language systems pilot corpus
Hemphill, C. T., Godfrey, J. J., & Doddington, G. R. (1990) · Proceedings of the workshop on Speech and Natural Language - HLT '90 · doi:10.3115/116580.116613 · Cited by 790 · cited 4×
“One of the earliest dataset collected in this fashion is the ATIS corpus (Hemphill et al., 1990), where conversations between a client and an airline help-desk operator were recorded.”
BackgroundMethodResultInfluential
Research works printed but not linked: 7
- Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. Proceedings of ICLR .
- Paweł Budzianowski, I˜ nigo Casanueva, Bo-Hsiang Tseng, and Milica Gaˇ si´ c. 2018. Towards endto-end multi-domain dialogue modelling. Tech. Rep. CUED/F-INFENG/TR.706, University of Cambridge, Engineering Department .
- Seokhwan Kim, Luis Fernando D'Haro, Rafael E Banchs, Jason D Williams, Matthew Henderson, and Koichiro Yoshino. 2016. The fifth dialog state tracking challenge. In Spoken Language Technology Workshop (SLT), 2016 IEEE , pages 511-517. IEEE.
- Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-end taskcompletion neural dialogue systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , volume 1, pages 733-743.
- David R. Traum. 1999. Foundations of Rational Agency , chapter Speech Acts for Dialogue Agents. Springer.
- Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gaˇ si´ c, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A networkbased end-to-end trainable task-oriented dialogue system. EACL .
- Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference , pages 404-413.
Cited by
Items 501–600 of 1,004
MoNET: Tackle State Momentum via Noise-Enhanced Training for Dialogue State Tracking
Zhang, H., Bao, J., Sun, H., Wu, Y., Li, W., Cui, S. R., & He, X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.33 · Cited by 11
Qixiang, G., Dong, G., Mou, Y., Wang, L., Zeng, C., Guo, D., Sun, M., & Xu, W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.157 · Cited by 9
“MultiWOZ 2.0 dataset (Budzianowski et al., 2018) provides turn-level annotations of dialogue states in 7 different domains.”
BackgroundInfluentialEnd-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics
Okur, E., Sahay, S., Alba, R. F., & Nachman, L. (2022) · Workshop on Mathematical Natural Language Processing (MathNLP) · doi:10.18653/v1/2022.mathnlp-1.7 · Cited by 10
An Adaptive Prompt Generation Framework for Task-oriented Dialogue System
Gao, J. Y., Xiang, L., Wu, H., Zhao, H. X., Tong, Y., & He, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.76 · Cited by 13
“(Schick and Schütze, 2021) (Budzianowski et al., 2020; Peng et al., 2021) 3.1 Data Preparation In this section, we describe the steps involved in preparing the data for our experiments.”
MethodInfluentialA Self-Learning Framework for Large-Scale Conversational AI Systems
Liu, X., Guo, C., Yao, B., & Sarikaya, R. (2024) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2024.3363971 · Cited by 11
“Typically, for each domain, it requires tens of thousands of conversation utterances [13], [14], [15].”
BackgroundConfidence Estimation For LLM-Based Dialogue State Tracking
Sun, Y.-J., Dey, S., Hakkani-Tür, D. Z., & Tür, G. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832237 · Cited by 9
“Our experiments use MultiWOZ [25], a multi-domain task-oriented dialogue dataset.”
LIDA: Lightweight Interactive Dialogue Annotator
Collins, E., Rozanov, N., & Zhang, B. (2019) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/d19-3021 · Cited by 9
“Following the same constraints imposed in MultiWOZ (Budzianowski et al., 2018) and DSTC (Henderson et al., 2014), this turn segmenter assumes that there are only two participants in the dialogue: the user and the sys-tem, and that the user asks the first query.”
BackgroundMethodInfluentialHigh-Quality Dialogue Diversification by Intermittent Short Extension Ensembles
Tang, Z., Kulkarni, H., & Yang, G. H. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.163 · Cited by 11
Towards Enriching Responses with Crowd-sourced Knowledge for Task-oriented Dialogue
He, Y., Liao, L., Zhang, Z., & Chua, T. (2021) · doi:10.1145/3475959.3485392 · Cited by 13
“Specifically, we apply an LSTM-based neural network [7] to encode each word segment 𝑑 𝑗 in tips together with its ranking position embedding 𝑧 𝑗 into the hidden state 𝑠 𝑗 .”
BackgroundInfluentialUnsupervised Slot Schema Induction for Task-oriented Dialog
Yu, D., Wang, M., Cao, Y., Izhak Shafran, J., Shafey, L. E., & Soltau, H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.86 · Cited by 20
Wang, H., Kwan, W.-C., Li, M., Zhou, Z., & Wong, K. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101637 · Cited by 11
Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking
Kim, T., Yoon, H., Lee, Y., Kang, P., & Kim, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.33 · Cited by 9
Graph Neural Network Policies and Imitation Learning for Multi-Domain Task-Oriented Dialogues
Cordier, T., Urvoy, T., Lefèvre, F., & Rojas-Barahona, L. M. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.10 · Cited by 9
“What do others think?”: Task-Oriented Conversational Modeling with Subjective Knowledge
Zhao, C., Gella, S., Kim, S., Jin, D., Hazarika, D., Papangelis, A., Hedayatnia, B., Namazifar, M., Liu, Y., & Hakkani-Tür, D. Z. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.28 · Cited by 18
Where is the context? -- A critique of recent dialogue datasetsPreprint
Mosig, J. E. M., Vlasov, V., & Nichol, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.10473 · Cited by 9
Meta Dialogue Policy LearningPreprint
Xu, Y., Zhu, C., Peng, B., & Zeng, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.02588 · Cited by 9
“Dataset and task settings We use the benchmark multi-domain dialogue dataset MultiWoz 2.0 [10] for the evaluation.”
BackgroundInfluentialSimulated Chats for Task-oriented Dialog: Learning to Generate Conversations from Instructions.Preprint
Mohapatra, B., Pandey, G. S., Contractor, D., & Joshi, S. (2020) · arXiv (Cornell University) · Cited by 9
Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented DialoguePreprint
Chen, M., Crook, P. A., & Roller, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06905 · Cited by 11
ViWOZ: A Multi-Domain Task-Oriented DialogueSystems Dataset For Low-resource LanguagePreprint
Van, P. N., Hoang, T. C., Manh, D. N., Khanh, H. B., Minh, Q. N., & Tran, L. Q. (2023) · Research Square · doi:10.21203/rs.3.rs-2711150/v1 · Cited by 9
Li, C., Hu, H., Yan, Z., Kan, M., & Li, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.07682 · Cited by 11
Bohnet, B., Swersky, K., Liu, R., Awasthi, P., Nova, A., Snaider, J., Sedghi, H., Parisi, A., Collins, M., Lazaridou, A., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.00179 · Cited by 16
Babu, C. V. S., Anniyappa, C. S. A., & Dharma Sastha, B. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-3860-5.ch008 · Cited by 8
Towards Transparent Interactive Semantic Parsing via Step-by-Step Correction
Mo, L., Lewis, A., Sun, H., & White, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.28 · Cited by 13
Contextual Semantic Parsing for Multilingual Task-Oriented Dialogues
Moradshahi, M., Tsai, V., Campagna, G., & Lam, M. S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.63 · Cited by 11
Reinforcing personalized persuasion in task-oriented virtual sales assistant
Raut, A., Tiwari, A., Das, S., Saha, S., Maitra, A., Ramnani, R., & Sengupta, S. (2023) · PLoS ONE · doi:10.1371/journal.pone.0275750 · Cited by 11
“ATIS [54] Task-oriented Flight booking ✓ × × × × MultiWoz [55] Task-oriented Service booking ✓ × × × × Persona-Chat [56] Chit-Chat × × × × ✓”
TextBox 2.0: A Text Generation Library with Pre-trained Language Models
Tang, T., Li, J., Chen, Z., Hu, Y., Yu, Z., Dai, W., Zhao, W. X., Nie, J., & Wen, J.-R. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.42 · Cited by 14
Semi-Supervised Knowledge-Grounded Pre-training for Task-Oriented Dialog Systems
Zeng, W., He, K., Wang, Z., Fu, D., Dong, G., Geng, R., Wang, P., Wang, J., Sun, C., Wu, W., et al. (2022) · doi:10.18653/v1/2022.seretod-1.6 · Cited by 13
StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses
Li, J.-N., Tu, Q., Mao, C., Yu, Z., Wen, J.-R., & Yan, R. (2024) · neural information processing systems · doi:10.52202/079017-2733 · Cited by 20
Towards AI Conversing: FloodBot using Deep Learning Model Stacks
Basnyat, B., Singh, N., Roy, N., Gangopadhyay, A., & Raglin, A. (2020) · IEEE International Conference on Smart Computing · doi:10.1109/smartcomp50058.2020.00025 · Cited by 9
“We experimented with multiple non domain specific Question and Answering corpus (Q&A) from various sources [10, 5, 9, 18].”
MethodAction State Update Approach to Dialogue Management
Stoyanchev, S., Keizer, S., & Doddipatla, R. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414888 · Cited by 9
“While it is recognized as a key part of discourse interpretation, reference resolution has not been the focus of the recent dialogue challenges and referring expressions are not induced or annotated in most of the recent taskoriented dialogue corpora [6, 7, 8].”
CSAGN: Conversational Structure Aware Graph Network for Conversational Semantic Role Labeling
Wu, H., Xu, K., & Song, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.177 · Cited by 8
“…improvements on conversation-based tasks, such as dialogue response generation (Li et al., 2017; Dinan et al., 2019; Wu et al., 2019), task-oriented dialogue modeling (Mrkši´c et al., 2017; Budzianowski et al., 2018) and conversational question answering (Choi et al., 2018; Reddy et al., 2019).”
BackgroundGuo, A., Ohashi, A., Hirai, R., Chiba, Y., Tsunomori, Y., & Higashinaka, R. (2021) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2021.nlp4convai-1.25 · Cited by 11
“In this work, we performed a case study analysis of user personality and task performance us-ing a rule-based dialogue system on the prevalent Multi-Domain Wizard-of-Oz (MultiWOZ) task (Budzianowski et al., 2018; Eric et al., 2020).”
MethodDatabase Search Results Disambiguation for Task-Oriented Dialog Systems
Qian, K., Kottur, S., Beirami, A., Shayandeh, S., Crook, P., Geramifard, A., Yu, Z., & Sankar, C. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.85 · Cited by 8
Strathearn, C., & Gkatzia, D. (2022) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2022.dialdoc-1.21 · Cited by 10
MidMed: Towards Mixed-Type Dialogues for Medical Consultation
Shi, X., Liu, Z., Wang, C., Leng, H., Xue, K., Zhang, X., & Zhang, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.453 · Cited by 24
AnyTOD: A Programmable Task-Oriented Dialog System
Zhao, J., Cao, Y., Gupta, R., Lee, H., Rastogi, A., Wang, M., Soltau, H., Izhak Shafran, J., & Wu, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1006 · Cited by 8
Wu, Y., Dong, G., & Xu, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.741 · Cited by 15
Transfer Learning for Context-Aware Spoken Language Understanding
Chen, Q., Zhuo, Z., Wang, W., & Xu, Q. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003902 · Cited by 8
“We explore Microsoft Dialogue Challenge (MDC) 4 [31] and MultiWOZ 2.0 (WOZ) 5 [32] datasets as other dialogue corpora for evaluating the proposed TL approaches.”
InfluentialPRAL: A Tailored Pre-Training Model for Task-Oriented Dialog Generation
Gu, J., Wu, Q., Wu, C., Shi, W., & Yu, Z. (2021) · doi:10.18653/v1/2021.acl-short.40 · Cited by 9
Generation and Extraction Combined Dialogue State Tracking with Hierarchical Ontology Integration
Li, X., Li, Q., Wu, W., & Yin, Q. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.171 · Cited by 8
“0, 84.2% in MultiWOZ2.”
BackgroundMethodInfluentialZhao, N., Li, H., Wu, Y., & He, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.825 · Cited by 17
“To mitigate this problem, researches (Zhou et al., 2018; Budzianowski et al., 2018; Zhang et al., 2018; Di-nan et al., 2019; Quan et al., 2020) build the dialogue datasets through crowd-sourcing by asking annotators to talk to each other according to given dialogue objectives.”
BackgroundToward Self-Learning End-to-End Task-oriented Dialog Systems
Zhang, X., Peng, B., Gao, J., & Meng, H. M. L. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.49 · Cited by 8
“Table 1: Data statistics of four single-domain dialog datasets (Peng et al., 2020a; Budzianowski et al., 2018).”
BackgroundMethodInfluentialA Tailored Pre-Training Model for Task-Oriented Dialog GenerationPreprint
Gu, J., Wu, Q., Wu, C., Shi, W., & Yu, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13835 · Cited by 8
“Our model outperforms or achieve comparable results with HDSA and LaRL.”
InfluentialRobustness through Data Augmentation Loss Consistency
Huang, T., Halbe, S., Sankar, C., Amini, P., Kottur, S., Geramifard, A., Razaviyayn, M., & Beirami, A. (2023) · Transactions on machine learning research · doi:10.48550/arxiv.2110.11205 · Cited by 10
“More surprisingly, DAIR also outperforms SimpleTOD on the original MultiWOZ 2.2 test set with no distribution shift, which we attribute to better robustness to the named entity memorization problem observed by (Qian et al., 2021).”
InfluentialLong Dialog Summarization: An AnalysisPreprint
Mullick, A., Bhowmick, A. K., Raghav, R. S., Kokku, R., Dey, P., Goyal, P., & Ganguly, N. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.16986 · Cited by 11
“There exists several benchmarked conversational datasets with summary - QmSum (Zhong et al., 2021), Multiwoz (Budzianowski et al., 2018; Eric et al., 2020), SParC (Yu et al., 2019), Sum-Screen (Tv series summary) (Chen et al., 2021), SQuALITY (Pang et al., 2021) etc.”
BackgroundA dataset of synthetic art dialogues with ChatGPT
Gil-Martín, M., Luna-Jiménez, C., Esteban-Romero, S., Estecha-Garitagoitia, M., Fernández-Martínez, F., & D’Haro, L. F. (2024) · Scientific Data · doi:10.1038/s41597-024-03661-x · Cited by 13
Zhou, K., Gligorić, K., Cheng, M., Lam, M. S., Raman, V., Aminu, B., Woo, C., Brockman, M., Cha, H., & Jurafsky, D. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.67 · Cited by 8
CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluation
Dai, Y., He, W., Li, B., Wu, Y., Cao, Z., An, Z., Sun, J. Q., & Li, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.274 · Cited by 15
Shi, D., You, C., Huang, J., Li, T., & Xiong, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29861 · Cited by 22
CallNavi, A challenge and empirical study on LLM function calling and routing
Song, Y., Tang, X., Lothritz, C., Ezzini, S., Klein, J., Bissyandé, T. F., Boytsov, A., Ble, U., & Goujon, A. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756975 · Cited by 10
“Recent advancements have explored structured output generation in applications such as code generation [7], table completion [15], and multi-turn dialogue [6].”
Dialogue State Tracking Based on Hierarchical Slot Attention and Contrastive Learning
Zhou, Y., Zhao, G., & Qian, X. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557581 · Cited by 7
“0[1] and MultiWOZ2.”
Wang, J., Li, H., Wang, L., & Wu, C. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08409-z · Cited by 8
Liu, Y., Mayfield, R., & Huang, Y. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579602 · Cited by 9
“We refer to ontology as the same concept in dialog state tracking task [Budzianowski et al.(2018)]. The ontology is based on the event ontology of Automatic Content Extraction (ACE) [Walker et al.(2006)] dataset.”
MethodUncertainty-Aware Self-Training for Low-Resource Neural Sequence Labeling
Wang, J., Wang, C., Huang, J. G., Gao, M., & Zhou, A. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26603 · Cited by 11
Multi-granularity Prompts for Topic Shift Detection in Dialogue
Lin, J.-Y., Fan, Y., Chu, X., Li, P., & Zhu, Q. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-4752-2_42 · Cited by 12
Liu, Y., Jiang, X., Yin, Y., Wang, Y., Mi, F., Liu, Q., Wan, X., & Wang, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.1 · Cited by 29
“Multiple User SimulaTors agenda-based methods and neural networks-based methods on the popular restaurant search task from MultiWOZ (Budzianowski et al., 2018).”
BackgroundMethodPrompter: Zero-shot Adaptive Prefixes for Dialogue State Tracking Domain Adaptation
Aksu, T., Kan, M., & Chen, N. F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.252 · Cited by 7
Improved Instruction Ordering in Recipe-Grounded Conversation
Le, D., Guo, R., Xu, W. H., & Ritter, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.561 · Cited by 12
Bebensee, B., & Lee, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.6 · Cited by 12
A Systematic Study of Performance Disparities in Multilingual Task-Oriented Dialogue Systems
Hu, S., Zhou, H., Yuan, M., Gritta, M., Zhang, G., Iacobacci, I., Korhonen, A., & Vulić, I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.422 · Cited by 16
Multi-domain gate and interactive dual attention for multi-domain dialogue state tracking
Xu, J., Zhang, R., & Peng, M. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111383 · Cited by 7
Multi-turn dialogue comprehension from a topic-aware perspective
Ma, X., Xu, Y., Zhao, H., & Zhang, Z. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127385 · Cited by 16
Fischer, S., Gemmell, C., Tecklenburg, N., Mackie, I., Rossetto, F., & Dalton, J. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671622 · Cited by 12
“In comparison, the Wizard of Tasks (WoT) dataset [5] contains conversations between crowd workers acting as students and teachers within the cooking and DIY domains.”
BackgroundMethodKhan, M. A., Prasad, B. K., Qi, G., Song, W. G., Ye, F., Ali, Z., Ullah, I., & Kefalas, P. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05571-2 · Cited by 7
“However, with the presence of complex and numerous candidate sets of slot-value pairs in large datasets, such as the 4510 value in MultiWoZ 2.0 [5, 52] increasing the number of nodes results in higher memory usage and exponentially growing computational time.”
InfluentialDu, Y., Wang, H., Zhao, Z., Liang, B., Wang, B., Zhong, W., Wang, Z., & Wong, K. (2024) · SIGHAN Workshop on Chinese Language Processing (SIGHAN) · doi:10.18653/v1/2024.sighan-1.18 · Cited by 72
“) ✓ ✗ ✗ ✗ ✗ QA on tables and text Multi-Woz (Budzianowski et al., 2018) ✗ ✗ ✗ ✓ ✗ Task-oriented Dialogue Persona-Chat (Zhang et al., 2018) ✗ ✓ ✗ ✓ ✗ Consistent personality dialogue DailyDialog (Li et al., 2017) ✗ ✗ ✗ ✓ ✗ Multi-turn dialogues on daily life Personal-Dialogue (Zheng et al., 2019) ✗ ✓…”
Sub-SA: Strengthen In-Context Learning via Submodular Selective Annotation
Qian, J., Sun, M., Zhou, S., Zhao, Z., Hun, R., & Chiang, P. Y. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia240720 · Cited by 8
Learning from Mistakes: Combining Ontologies via Self-Training for Dialogue Generation
Reed, L., Harrison, V., Oraby, S., Hakkani-Tür, D. Z., & Walker, M. A. (2020) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2020.sigdial-1.3 · Cited by 10
Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles
van Niekerk, C., Heck, M. C., Geishauser, C., Lin, H.-C., Lubis, N., Moresi, M., & Gašić, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.277 · Cited by 11
Self-citationN-Shot Learning for Augmenting Task-Oriented Dialogue State Tracking
Aksu, I., Liu, Z., Kan, M., & Chen, N. F. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.131 · Cited by 7
“We evaluate our framework with MultiWOZ, a multi-domain dialogue dataset (Budzianowski et al., 2018).”
MethodSurvey of Available Datasets for Designing Task Oriented Dialogue Agents
Thakkar, M., & Pise, N. N. (2019) · International Conference on Mechatronics, Remote Sensing, Information Systems and Industrial Information Technologies (ICMRSISIIT) · doi:10.1109/icmrsisiit46373.2020.9405898 · Cited by 7
“7 Taskmaster-1 (2019) [28] Spanning Over 6 Domains ( Ordering Pizza, Creating Auto Repair Appointments, Setting Up Ride Service, Ordering Movie Tickets, Ordering Coffee Drinks And Making Restaurant Reservations) Two Methods Two Person Dialogues on Woz Setup and Dialogues Written By Crowdworkers No of Dialogues: 13,215…”
BackgroundAct-Aware Slot-Value Predicting in Multi-Domain Dialogue State Tracking
Su, R., Wu, T.-W., & Juang, B.-H. F. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-138 · Cited by 7
What BERT Based Language Model Learns in Spoken Transcripts: An Empirical Study
Kumar, A., Sundararaman, M. N., & Vepa, J. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.25 · Cited by 10
A Stack-Propagation Framework With Slot Filling for Multi-Domain Dialogue State Tracking
Wang, Y., He, T., Mei, J., Fan, R., & Tu, X. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3183081 · Cited by 7
“Comparative experiments are conducted on two publicly available datasets for DST, namely, MultiWOZ 2.0 [33] and MultiWOZ 2.1 [14].”
MethodInfluentialAchieving Conversational Goals with Unsupervised Post-hoc Knowledge Injection
Majumder, B. P., Jhamtani, H., Berg-Kirkpatrick, T., & McAuley, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.224 · Cited by 10
Multi-modal multi-hop interaction network for dialogue response generation
Zhou, J., Tian, J., Wang, R., Wu, Y., Yan, M., He, L., & Huang, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120267 · Cited by 7
Diable: Efficient Dialogue State Tracking as Operations on Tables
Lesci, P., Fujinuma, Y., Hardalov, M., Shang, C., & Màrquez, L. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.615 · Cited by 7
Continual Learning for Natural Language Generations with Transformer Calibration
Yang, P., Li, D., & Li, P. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.4 · Cited by 11
AARGH! End-to-end Retrieval-Generation for Task-Oriented Dialog
Nekvinda, T., & Dušek, O. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.29 · Cited by 7
VIST5: An Adaptive, Retrieval-Augmented Language Model for Visualization-oriented Dialog
Voigt, H., Carvalhais, N., Meuschke, M., Reichstein, M., Zarrie, S., & Lawonn, K. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.5 · Cited by 14
“The average dialog has a number of 11.6 user turns, which is higher than the average number of user turns in current task-oriented dialog datasets such as Multi-WOZ (Budzianowski et al., 2018).”
BackgroundInfluentialRevise the NLU: A Prompting Strategy for Robust Dialogue System
Rohmatillah, M., & Chien, J. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446138 · Cited by 10
“Unfortunately, the performance of recently available NLU for multi-domain TOD systems like in the MultiWOZ [16] is quite limited.”
BackgroundMalviya, S., Mishra, R., Barnwal, S. K., & Tiwary, U. S. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3065833 · Cited by 9
Jurassic is (Almost) All You Need: Few-Shot Meaning-to-Text Generation for Open-Domain Dialogue
Reed, L., Li, C., Ramirez, A. M., Wu, L., & Walker, M. A. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-5538-9_7 · Cited by 11
CookDial: a dataset for task-oriented dialogs grounded in procedural documents
Jiang, Y., Zaporojets, K., Deleu, J., Demeester, T., & Develder, C. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03692-0 · Cited by 7
“In specific domain contexts, such as task-oriented CAs to assist users in finding/booking restaurants etc., a slot-filling approach is commonly adopted [3, 15–17]: a predefined template of slots is to be filled by string values.”
BackgroundInfluentialEskénazi, M., & Zhao, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06026 · Cited by 8
“For example, the structured intermediate labels of belief states and dialog acts in the multi-domain MultiWOZ corpus (Budzianowski et al., 2018) resulted in research into domain generalizability (Mehri et al., 2019b) and controlled generation (Chen et al., 2019).”
BackgroundMethodInfluentialMulti-Domain Dialogue State Tracking - A Purely Transformer-Based Generative Approach.Preprint
Zeng, Y., & Nie, J. (2020) · arXiv (Cornell University) · Cited by 7
Tan, C.-H., Yang, X., Zheng, Z., Li, T., Feng, Y., Gu, J.-C., Liu, Q., Liu, D., Ling, Z.-H., & Zhu, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11937 · Cited by 7
“The validation dataset, as well as training dataset, consist of only seen domain data, which is an augmented version of MultiWoz 2.1 (Budzianowski et al. 2018; Eric et al. 2020) introducing knowledge-seeking turn for dialogue.”
MethodInfluentialLearning Goal-oriented Dialogue Policy with Opposite Agent AwarenessPreprint
Zhang, Z., Liao, L., Zhu, X., Chua, T., Liu, Z., Huang, Y., & Huang, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.09731 · Cited by 8
“For the cooperative task, we used MultiWOZ (Budzianowski et al., 2018), a large-scale linguistically rich multi-domain goal-oriented dialogue dataset, which contains 7 domains, 13 intents and 25 slot types.”
MethodInfluentialA Hybrid Task-Oriented Dialog System with Domain and Task Adaptive PretrainingPreprint
Zhang, B., Lyu, Y., Ding, N., Shen, T., Jia, Z., Han, K., & Knight, K. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.04506 · Cited by 8
“Many recent work attempt to use end-to-end neural models, such as sequence-to-sequence and GPT-2, to solve task-oriented dialog problems and achieve remarkable results (Wen et al. 2016b; Budzianowski et al. 2018; Zhang, Ou, and Yu 2019; Peng et al. 2020; Ham et al. 2020a).”
SYNERGY: Building Task Bots at Scale Using Symbolic Knowledge and Machine TeachingPreprint
Peng, B., Li, C., Zhang, Z., Li, J., Zhu, C., & Gao, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.11514 · Cited by 7
“Following Budzianowski et al. (2018) and Peng et al. (2020a), Inform , Success , and BLEU scores are reported.”
BackgroundMethodTowards Better Understanding of User Satisfaction in Open-Domain Conversational SearchPreprint
Chu, Z., Ai, Q., Wang, Z., Liu, Y., Huang, Y., Zhang, R., Zhang, M., & Ma, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.02659 · Cited by 8
“Task-oriented conversational information need is defined based on an ontology in MultiWOZ.”
BackgroundMethodLexical Entrainment for Conversational Systems
Shi, Z., Sen, P., & Lipani, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.22 · Cited by 8
Acquiring New Knowledge Without Losing Old Ones for Effective Continual Dialogue Policy Learning
Wang, H., Zhang, Y., Yang, Y., Zheng, Y., & Wong, K. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3344727 · Cited by 8
“[67].”
MethodInfluentialOvercoming Catastrophic Forgetting by Exemplar Selection in Task-oriented Dialogue System
Chen, C., Li, R., Hu, Y., Chen, Y., Qin, C., & Zhang, Q. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.5 · Cited by 9
A multi-agent collaborative algorithm for task-oriented dialogue systems
Sun, J., Kou, J., Shi, W., & Hou, W. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02374-2 · Cited by 7
“Comparative experiments conducted on the MultiWOZ corpus [14] demonstrate that MACD can formulate more efficacious multi-agent collaborative training strategies.”
MTSS: Learn from Multiple Domain Teachers and Become a Multi-Domain Dialogue Expert
Peng, S., Ji, F. Q., Lin, Z., Cui, S., Chen, H., & Zhang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6384 · Cited by 6
“Dataset We choose MultiWOZ (Budzianowski et al. 2018), a multi-domain human-human conversation corpus, Models Multi-domain BLEU Inform Success”
MethodInfluentialSummary Grounded Conversation Generation
Gunasekara, C., Feigenblat, G., Sznajder, B., Joshi, S., & Konopnicki, D. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.329 · Cited by 11
DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Li, Y., Peng, B., He, P., Galley, M., Zhou, Y., & Gao, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.76 · Cited by 13
SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams
Wu, T.-L., Kottur, S., Madotto, A., Azab, M., Rodriguez, P., Damavandi, B., Peng, N., & Moon, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.345 · Cited by 18
“…or embodied dialogs (Das et al., 2017a; Padmakumar et al., 2022) are often constructed via a two-player game where participants interact with the environment and converse with each other ( i . e . in a Wizard of Oz (WOZ) (Mrkši´c et al., 2017; Budzianowski et al., 2018a) role-playing fashion).”
BackgroundLiu, H., Peng, H., Ou, Z., Li, J., Huang, Y., & Feng, J. (2022) · doi:10.18653/v1/2022.seretod-1.9 · Cited by 6
Patients Facing Large Language Models in Oncology: A Narrative Review
Raynaud, C., Wu, D., Levy, J., Marengo, M., & Bibault, J. (2024) · JCO Clinical Cancer Informatics · doi:10.1200/cci-24-00149 · Cited by 8
Metadata
- Authors
- Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., & Gašić, M.
- Affiliations
- University of Cambridge (GB)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 5016-5026
- OpenAlex
- W2891732163
- Cited by
- 1,028
- References
- 48
- Citations by year
- 2026: 39 · 2025: 66 · 2024: 86 · 2023: 209 · 2022: 186 · 2021: 196 · 2020: 181 · 2019: 64 · 2018: 1
- FWCI
- 45.11
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/M018946/1, RG80792)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- MultiWOZ, task-oriented dialogue, multi-domain dialogue, dialogue modeling, dialogue dataset, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.