MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
Open accessThe DOI matches its Crossref record; 46 of the 48 references it cites were found in a registry or an index.
Abstract
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica Gašić. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some tasks have clear ground truth answers and can be scored with exact correctness (Du et al., 2024; Dunn et al., 2017), while others are open-ended (dialogue (Budzianowski et al., 2018), summarization (Maharana et al., 2024), preference following (Zhao et al., 2025c)) where multiple outputs are acceptable and reference-based scoring is not reliable.”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The former types of benchmarks are static and only test the conversational agent on pre-collected conversation trajectories[4, 3, 2].”
From $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 4×
“Among them, the first two metrics relate to the dialogue task completion - whether the system has provided an appropriate entity (In-form rate) and then answered all the requested attributes (Success rate); while fluency is measured via BLEU score (Papineni et al., 2002).”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“(2017) which frames the dialogue as a context to response mapping problem, a sequence-to-sequence model (Sutskever et al., 2014) is augmented with a belief tracker and a discrete database accessing component as additional features to inform the word decisions in the decoder.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“In the case of MultiWOZ dataset, the LSTM cell serving as a decoder and an encoder achieved the highest score with the global type of attention (Bahdanau et al., 2014).”
MethodMeasuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“As this requires a multi-annotator metric over a multi-label task, we used Fleiss’ kappa metric (Fleiss, 1971) per single dialogue act.”
MethodA Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926 · cited 2×
“Due to the lack of an explicit goal in the conversation, recent studies have shown that systems trained with this type of corpus not only struggle in generating consistent and diverse responses (Li et al., 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundVinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400 · cited 2×
“Although previous work (Vinyals and Le, 2015) has shown that a large learning system can learn to generate interesting responses from these corpora, the lack of grounding conversations onto an existing knowledge base or APIs limits the usability of developed systems.”
BackgroundLiu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1230 · Cited by 1,228 · cited 3×
“, 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundPOMDP-Based Statistical Spoken Dialog Systems: A Review
Young, S., Gasic, M., Thomson, B., & Williams, J. D. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2225812 · Cited by 882 · cited 4×
“These problems can either be addressed separately (Young et al., 2013), or jointly in an end-to-end fashion (Bordes et al., 2017; Wen et al., 2017; Li et al., 2017).”
BackgroundInfluentialSelf-citationSemantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874 · cited 5×
“To make a fair comparison, we constrained our dataset to only the restaurant sub-domain which contains around 25 k dia-Cam676 Table 5 presents the results on two metrics: BLEU score (Papineni et al., 2002) and slot error rate (SER) (Wen et al., 2015).”
BackgroundMethodInfluentialSelf-citationThe ATIS spoken language systems pilot corpus
Hemphill, C. T., Godfrey, J. J., & Doddington, G. R. (1990) · Proceedings of the workshop on Speech and Natural Language - HLT '90 · doi:10.3115/116580.116613 · Cited by 790 · cited 4×
“One of the earliest dataset collected in this fashion is the ATIS corpus (Hemphill et al., 1990), where conversations between a client and an airline help-desk operator were recorded.”
BackgroundMethodResultInfluential
Research works printed but not linked: 7
- Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. Proceedings of ICLR .
- Paweł Budzianowski, I˜ nigo Casanueva, Bo-Hsiang Tseng, and Milica Gaˇ si´ c. 2018. Towards endto-end multi-domain dialogue modelling. Tech. Rep. CUED/F-INFENG/TR.706, University of Cambridge, Engineering Department .
- Seokhwan Kim, Luis Fernando D'Haro, Rafael E Banchs, Jason D Williams, Matthew Henderson, and Koichiro Yoshino. 2016. The fifth dialog state tracking challenge. In Spoken Language Technology Workshop (SLT), 2016 IEEE , pages 511-517. IEEE.
- Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-end taskcompletion neural dialogue systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , volume 1, pages 733-743.
- David R. Traum. 1999. Foundations of Rational Agency , chapter Speech Acts for Dialogue Agents. Springer.
- Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gaˇ si´ c, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A networkbased end-to-end trainable task-oriented dialogue system. EACL .
- Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference , pages 404-413.
Cited by
Items 901–1,000 of 1,004
Joint Modeling for ASR Correction and Dialog State Tracking
Wang, D., Zhang, T., Yuan, C., & Wang, X. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095945 · Cited by 3
“We build a dataset that simulates the ASR noise based on multi-domain DST benchmark Multi-WOZ [10].”
MethodAssis: Online Semi-Automatic Dialog Annotation Tool
Foerste, H. T. S., Purim, A. G. M., de Souza, R. R., & dos Reis, J. C. (2023) · Anais do Simpósio Brasileiro de Sistemas de Informação (SBSI) · doi:10.1145/3592813.3592886 · Cited by 2
“Além de fornecer uma quantidade de dados estruturados e anotados em escala superior a todos os anteriormente disponibilizados, estabelece uma estrutura de organização e anotação de conversas útil, a qual pode ser aplicada na construção de outros conjuntos de dados [4].”
BackgroundA Textual Dataset for Situated Proactive Response Selection
Otani, N., Araki, J., Kim, H., & Hovy, E. H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.214 · Cited by 3
“…English texts, which are more explicit as a semantic representation than just maintaining conversation histories (Lowe et al., 2015; Li et al., 2017; Henderson et al., 2019) and more flexible than structured representations of limited vocabulary (Williams et al., 2013; Budzianowski et al., 2018).”
BackgroundCLSE: Corpus of Linguistically Significant Entities
Chuklin, A., Zhao, J., & Kale, M. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.6 · Cited by 4
Label-Aware Auxiliary Learning for Dialogue State Tracking
Liu, Y., Chen, L., & Yu, K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447189 · Cited by 2
“0 [13] and WOZ [14].”
BackgroundGLMDST: Chinese Dialogue State Tracking Framework Driven by LLM
Jia, Z., & Li, B. (2024) · Chinese Control and Decision Conference (CCDC) · doi:10.1109/ccdc62350.2024.10587519 · Cited by 2
“In recent years, DST model based on neural network has made great progress in English field by us-ing annotated datasets such as MultiWOZ [16] and SGD [17].”
MethodLearning to memorize in neural task-oriented dialogue systems
Wu, C.-S. (2019) · doi:10.14711/thesis-991012730762803412 · Cited by 3
Multi-Granularity Representations of DialogPreprint
Mehri, S., & Eskénazi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.09890 · Cited by 2
“These hidden sizes match the best performing hyperparameters iden-tified by Budzianowski et al. (2018).”
BackgroundMethodInfluentialDomain Transfer in Dialogue Systems without Turn-Level SupervisionPreprint
Bingel, J., Hansen, V. P. B., Gonzalez-Garduño, A. V., Budzianowski, P., Augenstein, I., & Søgaard, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.07101 · Cited by 2
“Furthermore, while preprocessing the data, we found that there are more values present than reported iin the ontology, therefore the number of values presented here is higher than what is reported in Budzianowski et al. (2018).”
MethodSelf-citationA Hybrid Solution to Learn Turn-Taking in Multi-Party Service-based Chat GroupsPreprint
de Bayser, M. G., Guerra, M. A., Cavalin, P., & Pinhanez, C. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.06350 · Cited by 2
“To evaluate the system, we have adapted the multibotwoz corpus [5][4] to a multiple travel service-based bots scenario with dialogue errors in order to simulate the user’s interaction.”
BackgroundMethodLin, Z., Cui, S., Li, G., Kang, X., Ji, F. Q., Li, F.-L., Zhao, Z., Chen, H., & Zhang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09616 · Cited by 2
“• MultiWOZ 2.0 [Budzianowski et al., 2018].”
InfluentialMEEP: An Open-Source Platform for Human-Human Dialog Collection and End-to-End Agent TrainingPreprint
Arkhangorodsky, A., Axelrod, A. E., Chu, C. S., Fang, S., Huang, Y., Nagesh, A., Shi, X., Zhang, B., & Knight, K. K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.04747 · Cited by 2
“WOZ was originally used to explore how human users might ultimately interact with an automatic dialog agent (Jurafsky and Martin, 2018), and is nowadays also used to collect training data (Wen et al., 2017; Asri et al., 2017; Budzianowski et al., 2018).”
BackgroundHigh-Quality Diversification for Task-Oriented Dialogue SystemsPreprint
Tang, Z., Kulkarni, H., & Yang, G. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.00891 · Cited by 4
“We evaluate the proposed approach on the Multiwoz (Budzianowski et al., 2018) expected to 1) provide correct answers to requested information and 2) complete the booking, if asked.”
MethodInfluentialMemory Augmented Optimizers for Deep LearningPreprint
McRae, P., Parthasarathi, P., Assran, M., & Chandar, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10708 · Cited by 2
Improved Goal Oriented Dialogue via Utterance Generation and Look AheadPreprint
Ben-David, E., Carmeli, B., & Anaby-Tavor, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12412 · Cited by 2
“We used the following datasets (see Table 1): MultiWOZ - a large-scale multi-domain WizardOf-Oz dataset for task-oriented dialogue modelling, containing over 8400 dialogues, and spanning seven domains (Budzianowski et al., 2018).”
BackgroundMethodInfluentialA Chit-Chats Enhanced Task-Oriented Dialogue Corpora for Fuse-Motive Conversation SystemsPreprint
Yu, C., Zhang, C., & Sun, Q. L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.05886 · Cited by 2
“) called system action [16], where the Act can be inform,”
BackgroundInvestigating Masking-based Data Generation in Language ModelsPreprint
Ma, E. S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.00008 · Cited by 2
“conversation transcripts or collected via crowdsourcing [4, 5] with no additional cost on human labour.”
BackgroundInvestigating Reinforcement Learning for Communication Strategies in a Task-Initiative SettingPreprint
Khalid, B., & Stone, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01479 · Cited by 2
“slot-filling domains like restaurant booking [2].”
Multi-Turn Dialogue Comprehension from a Topic-Aware PerspectivePreprint
Ma, X., Xu, Y., Zhao, H., & Zhang, Z. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4585917 · Cited by 2
DFA-RAG: Conversational Semantic Router for Large Language Model with Definite Finite AutomatonPreprint
Sun, Y., Hu, J., Cheng, W. D., & Chen, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.04411 · Cited by 9
“MultiWOZ is known for its dense annotations, detailing the purpose and key information of each dialogue.”
MethodInfluentialLarge Language Model Driven RecommendationPreprint
Korikov, A., Sanner, S., Deldjoo, Y., He, Z., McAuley, J., Ramisa, A., Vidal, R., Sathiamoorthy, M., Kasrizadeh, A., Milano, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.10946 · Cited by 3
“…techniques, where the dialogue history would be used to update a dialogue state consisting of variables, or slots (e.g., “ preferred_cuisine: _ ”), that were filled from a set of predefined values (e.g., “ Mexican ”, “ French ”, ...) (Williams et al. , 2014; Budzianowski et al. , 2018).”
MethodBolshinsky, I., Kupiec, S., Sasson, A., Aperstein, Y., & Apartsin, A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.05006 · Cited by 2
Real-World En Call Center Transcripts Dataset with PII RedactionPreprint
Dao, H., Chawla, G., Banda, R., & DeLeeuw, C. (2025) · arXiv (Cornell University) · Cited by 2
Flexible Agent Alignment with Goal Inference from Open-Ended DialogPreprint
Ma, R., Qu, J., Bobu, A., & Hadfield-Menell, D. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2508.15119 · Cited by 2
“Evaluations/Benchmarks: Existing multi-turn dialogue benchmarks [28, 37, 47, 44, 1, 9, 10] do not support real-time dialogue conditioned on evolving preferences and informed by agent actions.”
Unlocking Proactivity in Task-Oriented DialoguePreprint
Zhang, H., Gao, N., Dai, Y., Wu, R., Wang, J., Gao, R. W., Tan, B., Gao, S., Li, Z., & Wang, C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.22240 · Cited by 2
Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response EvaluationPreprint
Miller, K., Chandra, A., & Saligrama, V. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.06718 · Cited by 2
“%) from Multi-Woz • 52 pairs (10.4”
InfluentialNous: Learning and Certifying Memory Decisions Before Source CalibrationPreprint
Singh, P. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.00094 · Cited by 2
A State-Aware Dual-Tool Architecture for Improved Tool Use in Multi-Turn LLM Dialogue
Pałgan, B., Wawryszczuk, M., Chwał, J., Tkacz, E., & Dzik, R. (2026) · Applied Sciences · doi:10.3390/app16094329 · Cited by 1
Imaezue, G. C., Maram, K. V., Ajayi, D., Alohali, I., & Butta, R. K. (2026) · Journal of Speech, Language, and Hearing Research · doi:10.1044/2026_jslhr-26-00031 · Cited by 1
$τ^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
Barres, V., Dong, H., Ray, S., Si, X., & Narasimhan, K. (2025) · arXiv · doi:10.48550/arxiv.2506.07982 · Cited by 1
“Following a long line of research into language agents [25, 27, 10, 13, 21], LLM tool use [24, 20, 9], and task-oriented dialog [5, 4, 1, 22, 6, 7, 8], τ -bench [26] is a recently introduced benchmark to measure the reliability of language agents in multi-turn task-oriented conversations such as…”
Du, Y., Wang, H., Zhao, Z., Liang, B., Wang, B., Zhong, W., Wang, Z., & Wong, K.-F. (2024) · arXiv · doi:10.48550/arxiv.2402.16288 · Cited by 1
“…✓ ✗ ✗ ✗ ✗ Dialogue QA on world knowledge HybridQA (Chen et al., 2020b) ✓ ✗ ✗ ✗ ✗ Multi-Hop QA on world knowledge OTT-QA (Chen et al., 2020a) ✓ ✗ ✗ ✗ ✗ QA on tables and text Multi-Woz (Budzianowski et al., 2018) ✗ ✗ ✗ ✓ ✗ Task-oriented Dialogue Persona-Chat (Zhang et al., 2018) ✗ ✓ ✗ ✓ ✗…”
BackgroundInfluentialProgressive Dialogue State Tracking for Multi-Domain Dialogue Systems
Wang, J., Liu, M., & Quan, X. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414610 · Cited by 1
“‘-’ means scores are not reported in the original papers. fixes substantial errors in MultiWOZ 2.0 [16].”
BackgroundInfluentialZeng, Z., Chow, C.-Y., & Li, N. (2021) · IEEE International Conference on Smart Computing · doi:10.1109/smartcomp52413.2021.00040 · Cited by 1
Nishimoto, B. E., Cristo, R., Mansano, A. F., Hruschka, E. R., Caridá, V. F., & Costa, A. H. R. (2022) · doi:10.5753/eniac.2022.227625 · Cited by 1
“The experiments were executed in two domains: MultiWOZ [Budzianowski et al. 2018, Eric et al. 2019] and Ita´u bank virtual assistant domain.”
MethodA More Accessible Web with Natural Language Interface
Deng, X. (2023) · International Web for All Conference (W4A) · doi:10.1145/3587281.3587700 · Cited by 2
“Existing semantic parsers on knowledge-based question answering [15], text-to-SQL [13], and task-oriented dialogue systems [3] usually work with fixed and well-formatted environments such as knowledge graphs, databases, and APIs.”
BackgroundA Benchmark for Evaluating Robustness of Spoken Language Understanding Models in Slot Filling
Peng, M., Xu, J., & Peng, M. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096803 · Cited by 1
“We already have many English datasets such as SNIPS[1], MultiWOZ[2] to measure the performance of SLU models.”
MethodA Review of Datasets for NLIDBs
Das, A., & Balabantaray, R. C. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-19-9304-6_21 · Cited by 1
Tracking Must Go On : Dialogue State Tracking with Verified Self-Training
Lee, J., Lee, C., Kim, Y., & Lee, G. G. (2023) · Interspeech (USB) · doi:10.21437/interspeech.2023-2513 · Cited by 1
Sanches, M. F., de Sá, J. M. C., de Souza, R. R., Souza, A. M. D., dos Reis, J. C., & Villas, L. A. (2023) · SN Computer Science · doi:10.1007/s42979-023-02148-7 · Cited by 1
“The MultiWOZ [25] is a relevant English dataset that simulates a dialogue between a user and a virtual assistant.”
BackgroundPICO: An Approach to Build Dataset for Customer Service Chatbot
Li, D., Peng, F., Ma, G., Wang, Z., & Wang, Q. (2023) · IEEE International Conference on Real-time Computing and Robotics (RCAR) · doi:10.1109/rcar58764.2023.10250131 · Cited by 1
CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogs
Aksu, T., Hazarika, D., Mehri, S., Kim, S., Hakkani-Tür, D. Z., Liu, Y., & Namazifar, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.717 · Cited by 3
Modeling the impact of out-of-schema questions in task-oriented dialog systems
Meem, J. A., Rashid, M. S., & Hristidis, V. (2024) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-024-01039-6 · Cited by 1
“Our first dataset, D-MultiWoz (D stands for “Drop-out”), extends the latest version of the MultiWoz (2.2) Budzianowski et al. (2018).”
InfluentialGameTE: A Game-Theoretic Distributed Traffic Engineering in Trustless Multi-Domain SDN
Liu, Y., Hua, J., Zhang, Y., & Zhong, S. (2024) · IEEE International Conference on Distributed Computing Systems (ICDCS) · doi:10.1109/icdcs60910.2024.00118 · Cited by 1
“Considering a network that supports communication between multiple domains, e.g. the global Internet is naturally organized as a large-scale multi-domain network [1], where each ISP corresponds to a single domain.”
BackgroundLin, H., Chen, Y., Tao, W., Chen, M., Xu, X., & Xing, X. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832316 · Cited by 1
“In recent years, with advancements in algorithms and the growth of data, significant progress has been made in building dialog systems [1, 2].”
Enriching Responses with Crowd-Sourced Knowledge for Task-Oriented Conversational Agents
Wei, Z., Liao, L., Xiang, X., & Du, X. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3714474 · Cited by 1
“Scholars widely agree that a good response should comprehensively meet a range of user needs, as evidenced in numerous studies [1, 36, 42].”
InfluentialQin, L., Huang, S., Chen, Q., Liu, Q., Che, W., & Xu, R. (2025) · Frontiers of Computer Science · doi:10.1007/s11704-024-3778-9 · Cited by 1
Intent-driven In-context Learning for Few-shot Dialogue State Tracking
Yi, Z., Xu, Z., & Shen, Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889623 · Cited by 5
“We employ the MultiWOZ 2.1 and 2.4 [18], [19] datasets to experimentally evaluate our proposed IDIC-DST method.”
InfluentialLeveraging Slot-Aware Enhancements for Improved Zero-Shot Dialogue State Tracking
Song, T., Hu, H., Chen, X., & Zi, K. (2024) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic64248.2024.10913312 · Cited by 1
“However, collecting this data is expensive and inefficient [1], so it is particularly important to study how to predict dialogue state under zero-shot conditions.”
A parallel network encoding dialog history template for end-to-end task-oriented dialog
Yang, G., Ma, D., Yuan, J., Li, N., & Bouraoui, Z. (2025) · CCF Transactions on Pervasive Computing and Interaction · doi:10.1007/s42486-024-00179-4 · Cited by 1
Strathearn, C., Yu, Y., & Gkatzia, D. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127304 · Cited by 2
Chen, Y.-C., & Huang, H. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i22.34534 · Cited by 3
Ontology Construction for Task-oriented Dialogue
Vukovic, R. (2024) · Young Researchers' Roundtable on Spoken Dialogue Systems (YRRSDS) · doi:10.18653/v1/2024.yrrsds-1.20 · Cited by 1
Faithfulness of Natural Language Generation
Schmidtová, P. (2024) · Young Researchers' Roundtable on Spoken Dialogue Systems (YRRSDS) · doi:10.18653/v1/2024.yrrsds-1.8 · Cited by 1
Xiao, Q., Zhao, Y., Dang, L., Hao, Y., Che, L., & Wu, Q. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130639 · Cited by 1
Guo, A., Ohashi, A., Hirai, R., Chiba, Y., Tsunomori, Y., & Higashinaka, R. (2025) · Scientific Reports · doi:10.1038/s41598-025-07101-7 · Cited by 4
“…of the three very hard tasks, where “Informable Slot” refers to attributes that users need to specify their search criteria (e.g., area or price range), and “Requestable Slot” refers to additional information that users need to request regarding a given entity (e.g., address or phone number) 37 .”
Hidayat, M. L., Ahmad, I., Assidiqi, M. H., Khanzada, T. J. S., & Meccawy, M. (2025) · International Conference on Innovation in Artificial Intelligence and Internet of Things (AIIT) · doi:10.1109/aiit63112.2025.11082869 · Cited by 2
“[62] Provided detailed annotation guidelines for the MultiWOZ dataset, which is one of the largest and most diverse available task-oriented dialogue datasets encompassing multiple domains, ensuring that dialogue acts, slot-value pairings, and system responses are consistently labelled and comparing…”
Flowchart-Based Decision Making with Large Language Models
Yamanaka, Y., Takahashi, H., & Yamashita, T. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.766 · Cited by 2
“While we assume that accurate flowcharts facilitate the creation of guided chatbots, future work should include performance metrics such as Winrate to validate this assumption across other datasets like MultiWOZ (Budzianowski et al., 2018).”
A generative AI model for modelling and creating an emotion-aware database: Design, and analytics
Cuzzocrea, A., Pilato, G., & Bringas, P. G. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131232 · Cited by 2
Interface Layout of Different AI Agents in Human-Computer Cooperation
Wang, L., Tang, W., Tang, X., Wang, H., Xue, C., & Gan, L. (2025) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2025.2551057 · Cited by 1
When Words Fall Short: The Case for Conversational Interfaces that Don’t Listen
Simpson, J., Stening, H., Patil, G., Nalepka, P., Dras, M., Kallen, R. W., Hosking, S. G., Richardson, M. J., & Richards, D. (2025) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3716553.3750799 · Cited by 1
“For instance, training a conversational agent to learn when to respond to user queries may require the collection and annotation of a considerable amount of training data despite conversational platforms and most recently large language models somewhat ameliorating this issue [8, 26, 35, 39].”
DialogGraph-LLM: Graph-Informed LLMs for End-to-End Audio Dialogue Intent Recognition
Liu, H., Li, J., Guo, C., Chen, H., Huang, Y. J. W., Guo, Y., Yang, H., & Cai, L. (2025) · Frontiers in artificial intelligence and applications · doi:10.3233/faia251182 · Cited by 1
“While valuable text-based dialogue datasets exist [15, 2], and multimodal benchmarks incorporating audio are emerging [41], acquiring extensive and accurately intent labeled audios remains a costly and labor-intensive task.”
SEAR: A Multimodal Dataset for Analyzing AR-LLM-Driven Social Engineering Behaviors
Yu, T., Ye, C., Yang, Z., Zhou, Z., Tang, C., Tao, Z., Zhang, J., Wang, K., Zhou, L., Yang, Y., et al. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3758245 · Cited by 4
Xu, H., Li, Z., Tang, W. J., & Zhang, J. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.85 · Cited by 4
“In practical scenarios, constructing an accurate DST model typically requires substantial labeled data, which is both time-consuming and costly to acquire (Budzianowski et al., 2018).”
InfluentialMIND: A Unified Multi-Modal Context-Knowledge Model for Task-Oriented Dialogue
Dong, X., Chen, J., Wei, C., Ke, C., & Hao, T. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11229289 · Cited by 1
“To evaluate the performance of our MIND model on text-modal datasets, we conduct experiments on the MultiWOZ dataset [17], comparing MIND with the highest-performing baseline models proposed in the past two years.”
Teng, D., Yan, Z., & Che, W. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.130481 · Cited by 1
Tie, Z., Chen, Q., Liu, F., Zhou, J., & He, L. (2025) · Lecture notes in computer science · doi:10.1007/978-981-95-3343-5_38 · Cited by 3
BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues
Jayannavar, P., Ren, L., Hudspeth, M., Sidhu, R., Lambert, C., Cordes, A., Kaplan, E., Narayan-Chen, A., & Hockenmaier, J. (2026) · Computational Linguistics · doi:10.1162/coli.a.602 · Cited by 1
MUSCAT: MUltilingual, SCientific ConversATion Benchmark
Sinhamahapatra, S., Nguyen, T. B., Oğuz, Y., Ugan, E. Y., Niehues, J., & Waibel, A. H. (2026) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2q8inmh4zmpa · Cited by 2
PATHFinder Agent for Tailored Prenatal Care
Balloli, V., Samuel, C., Abdelnabi, S., Peahl, A. F., & Bondi, E. (2026) · ACM Interactive Health Conference · doi:10.1145/3786579.3804996 · Cited by 1
“On the other hand, researchers have also studied task-oriented conversations [6, 7] between humans and automated evaluation of conversational models [8].”
One Model, Multiple Goals: Adaptive Multi-Objective Learning for E-commerce Dialogue Systems
Li, M., Xiang, J., Zhou, E., Gao, L., Li, T., Zhang, Q., Zhang, X., & Chen, X. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818454 · Cited by 1
Memory-Augmented Dialogue State Tracker in Task-Oriented Dialogue System
Li, Q., & Li, X. (2020) · International Conference on Computer Information and Big Data Applications (CIBDA) · doi:10.1109/cibda50819.2020.00100 · Cited by 1
On Task-Level Dialogue Composition of Generative Transformer Model
Parthasarathi, P., Narang, S., & Neelakantan, A. (2020) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2020.insights-1.6 · Cited by 1
Energy-based Neural Modelling for Large-Scale Multiple Domain Dialogue State Tracking
Trinh, A. D., Ross, R., & Kelleher, J. D. (2020) · Workshop on Structured Prediction for NLP (SPNLP) · doi:10.18653/v1/2020.spnlp-1.5 · Cited by 1
“Since we focus on multiple domain dialogue state tracking, we conduct our study on MultiWOZ 2.0 (Budzianowski et al., 2018) and MultiWOZ 2.1 (Eric et al., 2019), two novel chat-based multidomain dialogue datasets.”
MethodInfluentialDomain Adaptation in Dialogue Systems using Transfer and Meta-Learning
Ribeiro, R., Abad, A., & Lopes, J. (2021) · IberSPEECH · doi:10.21437/iberspeech.2021-44 · Cited by 1
Domain Expert Platform for Goal-Oriented Dialog Collection
Goško, D., Znotiņš, A., Skadiņa, I., Grūzītis, N., & Nešpore-Bērzkalne, G. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-demos.35 · Cited by 1
“For English and several other widely used languages, many publicly available dialogue datasets have been created and are reused for different research and development needs (e.g., Budzianowski et al. (2018), Zeng et al. (2020)).”
BackgroundPhrase-Level Action Reinforcement Learning for Neural Dialog Response Generation
Yamazaki, T., & Aizawa, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.446 · Cited by 2
“The action space is represented with hand-crafted dialog acts (Budzianowski et al., 2018; Stolcke et al., 2000) or meaning representations (Balakrish-nan et al., 2019).”
BackgroundMethodInfluentialParthasarathi, P., Pineau, J., & Chandar, S. (2021) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2021.sigdial-1.50 · Cited by 1
Arslan, D., & Eryiğit, G. (2021) · International Conference on Innovations in Intelligent Systems and Applications (INISTA) · doi:10.1109/inista52262.2021.9548636 · Cited by 1
Predicting Next Dialogue Action in Emotionally Loaded Conversation
Deksne, D., & Skadiņš, R. (2021) · Lecture notes in networks and systems · doi:10.1007/978-3-030-89906-6_19 · Cited by 1
Demeter, D., & Downey, D. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.16 · Cited by 3
“Lastly, we note that SPLATs can be constructed for other domains such as WOZ dialogues (Budzianowski et al., 2018) or quantitative behavior finance (Ramiah et al., 2015).”
BackgroundAIA-BDE: um Corpo de Perguntas, Variações e outras Anotações
Oliveira, H. G., & Alves, A. (2021) · Linguamática · doi:10.21814/lm.13.2.350 · Cited by 1
Zeng, Z., Lam, K.-Y., Chow, C.-Y., & Li, N. (2021) · International Journal of Humanoid Robotics · doi:10.1142/s0219843621500195 · Cited by 1
Cache: Modeling Contribution-Aware Context Hierarchically for Long-Range Dialogue State Tracking
Qi, J., Si, Y., Wang, L., & Dang, J. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746560 · Cited by 1
Takayama, J., Kajiwara, T., & Arase, Y. (2022) · Journal of Natural Language Processing · doi:10.5715/jnlp.29.84 · Cited by 1
Probabilistic Dialogue Model Combined With VAE For Task-oriented Dialogue System
Xing, X., & Wang, Z. (2021) · 2021 2nd International Conference on Electronics, Communications and Information Technology (CECIT) · doi:10.1109/cecit53797.2021.00154 · Cited by 1
Khalid, H., & Wade, V. (2022) · International Journal on Natural Language Computing · doi:10.5121/ijnlc.2022.11401 · Cited by 1
“In the experiment, we use switchboard data [31], multiWOZ dataset [32] and persona chat [33].”
MethodA rich task-oriented dialogue corpus in Vietnamese
Luong, T. C., Le-Hong, P., & Tran, O. T. (2022) · Language Resources and Evaluation · doi:10.1007/s10579-022-09618-z · Cited by 1
“The most notable multi-domain English corpus is MultiWOZ (Budzianowski et al., 2018) which greatly stimulated the research on multi-domain dialogue modelling.”
BackgroundMethodSUPPLE: A Dialogue Management Approach Based on Conversation Patterns
Kunneman, F., & Hindriks, K. V. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-19-5538-9_14 · Cited by 1
Dynamic Network Model for Multi-Domain End-to-end Task-Oriented Dialogue System
Zhao, F., Qiu, M. L., Li, X., & Guo, D. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981938 · Cited by 1
“The multi-domain Wizard-of-Oz (MultiWOZ) [4] dataset is the maximal human dialog corpus available, which is widely used in the research of dialog systems.”
BackgroundA Commonsense-Enhanced Document-Grounded Conversational Agent: A Case Study on Task-Based Dialogue
Strathearn, C., & Gkatzia, D. (2022) · Signals and communication technology · doi:10.1007/978-3-031-11035-1_6 · Cited by 1
Task-Oriented Document-Grounded Dialog Systems by HLTPR@RWTH for DSTC9 and DSTC10
Thulke, D., Daheim, N., Dugast, C., & Ney, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3267832 · Cited by 1
“The data that is provided as part of the challenges is based on the MultiWOZ 2.1 dataset [3], [4].”
MethodInfluentialDialogue Specific Pre-training Tasks for Improved Dialogue State Tracking
An, J., & Kim, M. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11283-4 · Cited by 1
“Even though MultiWOZ [25] has been continuously updated from its original version, there still exists noisy labels.”
BackgroundCall Center and Robot Call Center How to Reduce the Cost
Fatullayev, R., Fatullayev, O., & Fatullayev, A. (2023) · Studies in fuzziness and soft computing · doi:10.1007/978-3-031-23476-7_3 · Cited by 1
Multi-Action Dialog Policy Learning from Logged User Feedback
Zhang, S., Zhao, J., Wang, P., Wang, T., Liang, Z., Tao, J., Huang, Y., & Feng, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26636 · Cited by 1
Braunschweiler, N., Doddipatla, R., Keizer, S., & Stoyanchev, S. (2023) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3571884.3597138 · Cited by 1
“, provide and request information, clarify issues, social dialogue acts), and natural language generation (converting symbolic dialogue act representations into natural language) [1, 20, 27].”
BackgroundMethodInfluentialExternal Slot Relationship Memory for Multi-Domain Dialogue State Tracking
Xing, X., Yang, C., Lin, D., Teng, D., Chen, P., & Zhang, X. (2023) · Applied Sciences · doi:10.3390/app13158943 · Cited by 1
Ma, X., Zhang, P., & Zhao, F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.194 · Cited by 1
CMCC: A Comprehensive and Large-Scale Human-Human Dataset for Dialogue Systems
Huang, Y., Wu, X., Chen, S., Hu, W., Zhu, Q., Feng, J., Deng, C., Ou, Z., & Zhao, J. (2022) · doi:10.18653/v1/2022.seretod-1.7 · Cited by 1
“Task-oriented dialogue corpus, such as Frames (Asri et al., 2017), MultiWOZ (Budzianowski et al., 2018), CrossWOZ (Zhu et al., 2020), RiSAWOZ (Quan et al., 2020), are collected by two crowd workers playing the roles of the user and the system, which often leads to be small-scale, and can not sufficiently capture a num…”
BackgroundNavigating Connected Memories with a Task-oriented Dialog System
Kottur, S., Moon, S., Geramifard, A., & Damavandi, B. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.160 · Cited by 1
Self-training with Two-phase Self-augmentation for Few-shot Dialogue Generation
Du, W., Chen, H., & Ji, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.201 · Cited by 1
“The two benchmark datasets are sampled and constructed based on the three datasets: RNNLG (Wen et al., 2016), MultiWOZ (Budzianowski et al., 2018) and SGD (Rastogi et al., 2020b).”
Method
Metadata
- Authors
- Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., & Gašić, M.
- Affiliations
- University of Cambridge (GB)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 5016-5026
- OpenAlex
- W2891732163
- Cited by
- 1,028
- References
- 48
- Citations by year
- 2026: 39 · 2025: 66 · 2024: 86 · 2023: 209 · 2022: 186 · 2021: 196 · 2020: 181 · 2019: 64 · 2018: 1
- FWCI
- 45.11
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/M018946/1, RG80792)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- MultiWOZ, task-oriented dialogue, multi-domain dialogue, dialogue modeling, dialogue dataset, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.