MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling
Open accessThe DOI matches its Crossref record; 46 of the 48 references it cites were found in a registry or an index.
Abstract
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, Milica Gašić. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The majority of evaluation protocols largely simplify experimental assumptions and design static, pre-defined rules, with relatively short and isolated task settings (Cobbe et al., 2021; Chen et al., 2021; Budzianowski et al., 2018).”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Some tasks have clear ground truth answers and can be scored with exact correctness (Du et al., 2024; Dunn et al., 2017), while others are open-ended (dialogue (Budzianowski et al., 2018), summarization (Maharana et al., 2024), preference following (Zhao et al., 2025c)) where multiple outputs are acceptable and reference-based scoring is not reliable.”
From A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents (2026) · cites this paper as Budzianowski et al., 2018
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The former types of benchmarks are static and only test the conversational agent on pre-collected conversation trajectories[4, 3, 2].”
From $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2024) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 4×
“Among them, the first two metrics relate to the dialogue task completion - whether the system has provided an appropriate entity (In-form rate) and then answered all the requested attributes (Success rate); while fluency is measured via BLEU score (Papineni et al., 2002).”
BackgroundMethodInfluentialSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 2×
“(2017) which frames the dialogue as a context to response mapping problem, a sequence-to-sequence model (Sutskever et al., 2014) is augmented with a belief tracker and a discrete database accessing component as additional features to inform the word decisions in the decoder.”
BackgroundMethodNeural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“In the case of MultiWOZ dataset, the LSTM cell serving as a decoder and an encoder achieved the highest score with the global type of attention (Bahdanau et al., 2014).”
MethodMeasuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“As this requires a multi-annotator metric over a multi-label task, we used Fleiss’ kappa metric (Fleiss, 1971) per single dialogue act.”
MethodA Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926 · cited 2×
“Due to the lack of an explicit goal in the conversation, recent studies have shown that systems trained with this type of corpus not only struggle in generating consistent and diverse responses (Li et al., 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundVinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400 · cited 2×
“Although previous work (Vinyals and Le, 2015) has shown that a large learning system can learn to generate interesting responses from these corpora, the lack of grounding conversations onto an existing knowledge base or APIs limits the usability of developed systems.”
BackgroundLiu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1230 · Cited by 1,228 · cited 3×
“, 2016) but are also extremely hard to evaluate (Liu et al., 2016).”
BackgroundPOMDP-Based Statistical Spoken Dialog Systems: A Review
Young, S., Gasic, M., Thomson, B., & Williams, J. D. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2225812 · Cited by 882 · cited 4×
“These problems can either be addressed separately (Young et al., 2013), or jointly in an end-to-end fashion (Bordes et al., 2017; Wen et al., 2017; Li et al., 2017).”
BackgroundInfluentialSelf-citationSemantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gasic, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1199 · Cited by 874 · cited 5×
“To make a fair comparison, we constrained our dataset to only the restaurant sub-domain which contains around 25 k dia-Cam676 Table 5 presents the results on two metrics: BLEU score (Papineni et al., 2002) and slot error rate (SER) (Wen et al., 2015).”
BackgroundMethodInfluentialSelf-citationThe ATIS spoken language systems pilot corpus
Hemphill, C. T., Godfrey, J. J., & Doddington, G. R. (1990) · Proceedings of the workshop on Speech and Natural Language - HLT '90 · doi:10.3115/116580.116613 · Cited by 790 · cited 4×
“One of the earliest dataset collected in this fashion is the ATIS corpus (Hemphill et al., 1990), where conversations between a client and an airline help-desk operator were recorded.”
BackgroundMethodResultInfluential
Research works printed but not linked: 7
- Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. Proceedings of ICLR .
- Paweł Budzianowski, I˜ nigo Casanueva, Bo-Hsiang Tseng, and Milica Gaˇ si´ c. 2018. Towards endto-end multi-domain dialogue modelling. Tech. Rep. CUED/F-INFENG/TR.706, University of Cambridge, Engineering Department .
- Seokhwan Kim, Luis Fernando D'Haro, Rafael E Banchs, Jason D Williams, Matthew Henderson, and Koichiro Yoshino. 2016. The fifth dialog state tracking challenge. In Spoken Language Technology Workshop (SLT), 2016 IEEE , pages 511-517. IEEE.
- Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-end taskcompletion neural dialogue systems. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , volume 1, pages 733-743.
- David R. Traum. 1999. Foundations of Rational Agency , chapter Speech Acts for Dialogue Agents. Springer.
- Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gaˇ si´ c, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A networkbased end-to-end trainable task-oriented dialogue system. EACL .
- Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference , pages 404-413.
Cited by
Items 301–400 of 1,004
Generating Usage-related Questions for Preference Elicitation in Conversational Recommender Systems
Kostric, I., Balog, K., & Radlinski, F. (2024) · ACM Transactions on Recommender Systems · doi:10.1145/3629981 · Cited by 30
“These models are usually trained on a hand-labeled corpus of task-oriented dialogue [6].”
PRESTO: A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogs
Goel, R., Ammar, W., Gupta, A., Vashishtha, S., Sano, M., Surani, F., Chang, M., Choe, H., Greene, D., He, C., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.667 · Cited by 22
NeuralWOZ: Learning to Collect Task-Oriented Dialogue via Model-Based Simulation
Kim, S., Chang, M., & Lee, S.-W. (2021) · doi:10.18653/v1/2021.acl-long.287 · Cited by 26
“We populate corresponding answer options There are two special values, Dontcare to indicate the user has no preference and None to indicate the user is yet to specify a value for this slot (Henderson et al., 2014a; Budzianowski et al., 2018).”
BackgroundMethodInfluentialGeng, B., Yuan, F., Xu, Q., Shen, Y., Xu, R., & Yang, M. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.18653/v1/2021.acl-short.66 · Cited by 28
“…dialogues for training/validation/testing, belonging to calendar scheduling, weather query, and POI navigation domains, (2) Multi-WOZ 2.1 (Budzianowski et al., 2018) that contains 1,839/117/141 dialogues for training/validation/testing, belonging to restaurant, attraction, and hotel…”
Influential[CASPI] Causal-aware Safe Policy Improvement for Task-oriented Dialogue
Ramachandran, G. S., Hashimoto, K., & Xiong, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.8 · Cited by 27
“0 (Budzianowski et al., 2018b).”
BackgroundMethodInfluentialJointly Optimizing State Operation Prediction and Value Generation for Dialogue State TrackingPreprint
Zeng, Y., & Nie, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14061 · Cited by 22
“We use two publicly available datasets MultiWOZ 2.0 (Budzianowski et al., 2018) and MultiWOZ 2.1 (Eric et al., 2019) in our experiments.”
MethodInfluentialCommonsense Reasoning for Conversational AI: A Survey of the State of the ArtPreprint
Richardson, C., & Heck, L. P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.07926 · Cited by 28
“There are many datasets used for dialogue modeling, including PERSONA-CHAT (Zhang et al. 2018), ConvAI2 (Dinan et al. 2020), DailyDialogue (Li et al. 2017), and MultiWOZ (Budzianowski et al. 2018).”
Aspect-Aware Response Generation for Multimodal Dialogue System
Firdaus, M., Thakur, N., & Ekbal, A. (2021) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3430752 · Cited by 21
“The recently released dataset, MultiWOZ [6], is an extension of WOZ2.”
MethodLearning to Learn and Remember Super Long Multi-Domain Task Sequence
Wang, Z., Shen, L., Duan, T., Zhan, D., Fang, L., & Gao, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00782 · Cited by 34
Hierarchical Reinforcement Learning With Guidance for Multi-Domain Dialogue Policy
Rohmatillah, M., & Chien, J. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3235202 · Cited by 21
“MultiWOZ 2.1 was an updated version of MultiWOZ 2.0 [49], known as a multi-domain, multi-intent task-oriented dialog corpus [50] that contained 7 domains which are hotel, attraction, restaurant, train, taxi, police and hospital, 13 user intents, 25 slot types, 10,483 dialog sessions, and 71,544…”
BackgroundInfluentialUnsupervised Dialogue Topic Segmentation with Topic-aware Contrastive Learning
Gao, H., Wang, R., Lin, T.-E., Wu, Y., Yang, M., Huang, F., & Li, Y. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3592081 · Cited by 35
“DialSeg711 [36] is a real-world dataset including 711 English dialogues that combines dialogues from two existing task-oriented dialogue datasets, MultiWOZ [2] and KVRET [9].”
BackgroundConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format
Zhu, Q., Geishauser, C., Lin, H.-C., van Niekerk, C., Peng, B., Zhang, Z., Feng, S., Heck, M. C., Lubis, N., Wan, D., et al. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.9 · Cited by 38
Self-citationSemi-Supervised Dialogue Policy Learning via Stochastic Reward Estimation
Huang, X., Qi, J., Sun, Y., & Zhang, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.62 · Cited by 22
Regularizing Dialogue Generation by Imitating Implicit Scenarios
Feng, S., Ren, X., Chen, H., Sun, B., Li, K., & Sun, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.534 · Cited by 24
Conversational Entity Linking: Problem Definition and Datasets
Joko, H., Hasibi, F., Balog, K., & de Vries, A. P. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463258 · Cited by 33
“Note that MultiWOZ has several versions [10, 25, 62]; we used the latest version,MultiWOZ 2.”
MethodSibyl: Simple yet Effective Agent Framework for Complex Real-world ReasoningPreprint
Wang, Y., Shen, T., Liu, L., & Xie, J. G. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.10718 · Cited by 28
“To efficiently compress the received information, we integrate the concept of dialogue states from task-oriented dialogue systems (Budzianowski et al., 2018; Quan et al., 2020; Moradshahi et al., 2023) into the channel and design a representation language for the compressed information.”
MethodProbing Task-Oriented Dialogue Representation from Language Models
Wu, C.-S., & Xiong, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.409 · Cited by 21
Ghosal, D., Majumder, N., Mihalcea, R. F., & Poria, S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.124 · Cited by 20
“MultiWOZ (Budzianowski et al., 2018) or Multi-Domain Wizard-of-Oz dataset is a fully-labeled collection of human-human conversations spanning over multiple domains and topics.”
BackgroundHybriDialogue: An Information-Seeking Dialogue Dataset Grounded on Tabular and Textual Data
Nakamura, K., Levy, S., Tuan, Y.-L., Chen, W., & Wang, W. Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.41 · Cited by 39
DIRECT: Toward Dialogue-Based Reading Comprehension Tutoring
Huang, J., Lee, Y., & Kwon, O. (2022) · IEEE Access · doi:10.1109/access.2022.3233224 · Cited by 27
Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches, and Future Directions
Jia, Q., Liu, Y., Ren, S., & Zhu, K. Q. (2023) · ACM Computing Surveys · doi:10.1145/3622933 · Cited by 31
“MultiWOZ* and TODSum transform and annotate summaries based on the original MultiWOZ dataset [15].”
MethodLin, H.-C., Geishauser, C., Feng, S., Lubis, N., van Niekerk, C., Heck, M. C., & Gašić, M. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.28 · Cited by 50
Self-citationSUMBT+LaRL: Effective Multi-Domain End-to-End Neural Task-Oriented Dialog System
Lee, H., Jo, S., Kim, H., Jung, S., & Kim, T.-Y. (2021) · IEEE Access · doi:10.1109/access.2021.3105461 · Cited by 20
“Following the conventional delexicalization method in [34], we replaced the user and system utterances with placeholders.”
BackgroundMethodInfluentialLearning Multi-Party Turn-Taking Models from Dialogue LogsPreprint
de Bayser, M. G., Cavalin, P., Pinhanez, C. S., & Zadrozny, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.02090 · Cited by 20
“Conversational systems have entered mainstream applications in recent years, as a result, we have seen a huge number of dyadic chatbots becoming available to users, from chit-chat bots which can converse about generic topics to entertain people, to chatbots which are experts and can provide useful services through nat…”
BackgroundMethodGoal-Oriented Multi-Task BERT-Based Dialogue State TrackerPreprint
Gulyaev, P. A., Elistratova, E., Konovalov, V., Kuratov, Y., Pugachev, L., & Burtsev, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02450 · Cited by 20
“As technology for dialogue state tracking developed, a more complex task was proposed in the MultiWoZ dataset (Budzianowski et al. 2018; Eric et al. 2019).”
MethodResponse-act Guided Reinforced Dialogue Generation for Mental Health Counseling
Srivastava, A., Pandey, I., Akhtar, M. S., & Chakraborty, T. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583380 · Cited by 33
Firdaus, M., Thakur, N., & Ekbal, A. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.210 · Cited by 19
“Though we follow a known approach (Wizard-of-oz) for data creation as done in the existing works (Budzianowski et al., 2018; Peskov et al., 2019; Saha et al., 2018), our MDMMD dataset constitutes of more varied responses belonging to the multiple domains and having both textual and visual…”
MethodHe, W., Sun, Y., Yang, M., Ji, F. Q., Li, C., & Xu, R. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.106667 · Cited by 19
A dynamic goal adapted task oriented dialogue agent
Tiwari, A., Saha, T., Saha, S., Sengupta, S., Maitra, A., Ramnani, R., & Bhattacharyya, P. (2021) · PLoS ONE · doi:10.1371/journal.pone.0249030 · Cited by 22
“Several benchmark conversational datasets were explored for the proposed setting which include ATIS [42], MultiWoz [43], Ubuntu dialogue corpus [44], bAbi [45], cornell-movie corpus [46], Deal or not [47] etc.”
BackgroundLiu, Q., Bai, G., He, S., Liu, C., Liu, K., & Zhao, J. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107186 · Cited by 20
Deep Learning on a Healthy Data Diet: Finding Important Examples for Fairness
Zayed, A., Parthasarathi, P., Mordido, G., Palangi, H., Shabanian, S., & Chandar, S. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26706 · Cited by 44
Feng, Y., Yang, S., Zhang, S., Zhang, J., Xiong, C., Zhou, M., & Wang, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.10342 · Cited by 23
“Following the standard setup ( e.g. , Budzianowski et al., 2018; Mehri et al., 2019), we use four automatic evaluations metrics: 1) Inform rate: the fraction of the dialogues where the system has provided an appropriate entity; 2) Success rate: the fraction of the dialogues where the system…”
BackgroundMethodInfluentialBuilding Persona Consistent Dialogue Agents with Offline Reinforcement Learning
Shea, R., & Yu, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.110 · Cited by 36
LARA: Linguistic-Adaptive Retrieval-Augmentation for Multi-Turn Intent Classification
Liu, J., Keat, T. Y., Fu, B., & Lim, K. H. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-industry.82 · Cited by 18
“Previous works adopt bidirectional contextual LSTM (Ghosal et al., 2021) to create context-aware utter-ance representation on MultiWOZ intent classification (Budzianowski et al., 2018).”
MethodTopic-Guided Conversational Recommender in Multiple Domains
Liao, L., Takanobu, R., Ma, Y., Yang, X., Huang, M., & Chua, T. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3008563 · Cited by 25
Schema2QA: High-Quality and Low-Cost Q&A Agents for the Structured Web
Xu, S., Campagna, G., Li, J., & Lam, M. S. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3411974 · Cited by 23
Gritta, M., Λάμπουρας, Γ., & Iacobacci, I. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00352 · Cited by 28
Dual Slot Selector via Local Reliability Verification for Dialogue State Tracking
Guo, J., Shuang, K., Li, J., & Wang, Z. (2021) · doi:10.18653/v1/2021.acl-long.12 · Cited by 18
“Our proposed DSS-DST achieves state-of-theart joint accuracy on three of the most actively studied datasets: MultiWOZ 2.0 (Budzianowski et al., 2018), MultiWOZ 2.1 (Eric et al., 2019), and MultiWOZ 2.2 (Zang et al., 2020) with joint accuracy of 56.93%, 60.73%, and 58.04%.”
MethodInfluentialDiscovering Dialogue Slots with Weak Supervision
Hudeček, V., Dušek, O., & Yu, Z. (2021) · doi:10.18653/v1/2021.acl-long.189 · Cited by 30
“• MultiWOZ (Budzianowski et al., 2018; Eric et al., 2020) is a multi-domain corpus; we picked two domains – hotel reservation and attraction recommendation – to form WOZ-hotel ( WH ) with 14,435 utterances, 9 slots, 3 intents and WOZ-attr ( WA ) with 7524 utterances, 8 slots and 3 intents…”
BackgroundDDRel: A New Dataset for Interpersonal Relation Classification in Dyadic Dialogues
Jia, Q., Huang, H., & Zhu, K. Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17551 · Cited by 31
Teacher-Student Framework Enhanced Multi-domain Dialogue GenerationPreprint
Peng, S., Huang, X., Lin, Z., Ji, F., Chen, H., & Zhang, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.07137 · Cited by 18
“We use two metrics that are suggested by Budzianowski et al. (2018b), as the estimations for the MultiWOZ dataset in the dialogue context to text task.”
MethodInfluentialModeling Text-visual Mutual Dependency for Multi-modal Dialog GenerationPreprint
Wang, S., Meng, Y., Sun, X., Wu, F. Q., Ouyang, R., Yan, R., Zhang, T., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14445 · Cited by 20
“With the emergence of task-oriented datasets [7, 17, 66, 74], more practice has been devoted to task-oriented dialog generation, which usually involves a pipeline of intent classification [67], dialog state tracking [25–27], dialog policy making [10, 45] and dialog generation [15].”
BackgroundToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models
Shim, J., Seo, G., Lim, C., & Jo, Y. (2025) · International Conference on Learning Representations · doi:10.48550/arxiv.2503.00564 · Cited by 44
“Representative datasets for TOD include MultiWOZ (Budzianowski et al., 2020) and Schema-Guided Dialogue (SGD) (Rastogi et al., 2020).”
Modularized Pre-Training for End-to-End Task-Oriented Dialogue
Qin, L., Xiao, X., Wang, L., Zhang, Y., & Che, W. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3244503 · Cited by 21
Injecting Domain Knowledge in Language Models for Task-oriented Dialogue Systems
Emelin, D., Bonadiman, D., Alqahtani, S., Zhang, Y., & Mansour, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.820 · Cited by 23
Dou, C., Jin, Z. Q., Jiao, W., Zhao, H., Zhao, Y., & Tao, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.336 · Cited by 27
“2) Mars (Sun et al., 2022), an advanced model explicitly crafted for the MultiWoZ (Budzianowski et al., 2018) dataset, renowned for its exceptional performance in addressing the Task-Oriented Dialogue (TOD) task.”
BackgroundMuthusamy, V., Rizk, Y., Kate, K., Venkateswaran, P., Isahagian, V., Gulati, A., & Dube, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.461 · Cited by 40
“There are a few benchmarks available for this setting (Rastogi et al., 2020; Li et al., 2023; Budzianowski et al., 2018) with only one of them being a fully human annotated corpus (Budzianowski et al., 2018).”
BackgroundDongre, V., Gui, C., Garg, S., Nayyeri, H., Tür, G., Hakkani-Tur, D. Z., & Adve, V. (2025) · neural information processing systems · doi:10.52202/085713-0744 · Cited by 19
Dialogue State Induction Using Neural Latent Variable Models
Min, Q., Qin, L., Teng, Z., Liu, X., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/532 · Cited by 17
“In practice, widely used datasets for SLU include ATIS [Hemphill et al. , 1990], while for DST include Multi-WoZ [Budzianowski et al. , 2018].”
BackgroundMethodInfluentialData-to-Text Generation with Iterative Text Editing
Kasner, Z., & Dušek, O. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.9 · Cited by 25
SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systems
Lee, H., Gupta, R., Rastogi, A., Cao, Y. J., Zhang, B., & Wu, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21341 · Cited by 44
“MultiWOZ (Budzianowski et al. 2018), another popular dialogue state tracking benchmark, faces similar issues in the zero-shot leave-one-domain-out setup (Wu et al.”
An Interpretable Neuro-Symbolic Reasoning Framework for Task-Oriented Dialogue Generation
Yang, S., 张, R. 蕊. Z., Erfani, S., & Lau, J. H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.338 · Cited by 21
DAISY: An Implementation of Five Core Principles for Transparent and Accountable Conversational AI
Wahde, M., & Virgolin, M. (2022) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2022.2081762 · Cited by 19
Robots-Dont-Cry: Understanding Falsely Anthropomorphic Utterances in Dialog Systems
Gros, D., Li, Y., & Zhou, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.215 · Cited by 23
A Fast and Robust BERT-based Dialogue State Tracker for Schema-Guided Dialogue DatasetPreprint
Noroozi, V., Zhang, Y., Bakhturina, E., & Kornuta, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.12335 · Cited by 17
“Most of the previously published public datasets, such as MultiWOZ [2] or M2M [16], use a fixed list of defined slots for each domain without any information on the semantics of the slots and other entities in the dataset.”
MethodOPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue
Chen, Z., Liu, Y., Chen, L., Zhu, S., Wu, M., & Yu, K. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00534 · Cited by 20
“0 (Budzianowski et al., 2018) and MultiWOZ2.”
MethodInfluentialContextualize Knowledge Bases with Transformer for End-to-end Task-Oriented Dialogue Systems
Gou, Y., Lei, Y., Liu, L., Dai, Y., & Shen, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.353 · Cited by 23
“Two public multi-turn task-oriented dialogue datasets are used to evaluate our model, i.e., SMD 6 (Eric et al., 2017) and Multi-WOZ 2.1 (Budzianowski et al., 2018).”
InfluentialImproving Limited Labeled Dialogue State Tracking with Self-Supervision
Wu, C.-S., Hoi, S. C. H., & Xiong, C. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.400 · Cited by 16
Shenoy, A., Bodapati, S., & Kirchhoff, K. (2021) · Workshop on e-Commerce and NLP (ECNLP) · doi:10.18653/v1/2021.ecnlp-1.3 · Cited by 18
“…to train a reliable language model, we used a combination of Schema-Guided Dialogue Dataset (Rastogi et al., 2019), MultiWOZ 2.1 (Eric et al., 2019; Budzianowski et al., 2018), MultiDoGo (Peskov et al., 2019) along with anonymized in-house datasets that belong to two e-commerce usecases : retail…”
MethodInfluentialSmåprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learning
Adewumi, T., Brännvall, R., Abid, N., Pahlavan, M., Sabah, S., Liwicki, F., & Liwicki, M. (2022) · Proceedings of the Northern Lights Deep Learning Workshop · doi:10.7557/18.6231 · Cited by 18
DS-TOD: Efficient Domain Specialization for Task-Oriented Dialog
Hung, C.-C., Lauscher, A., Ponzetto, S. P., & Glavaš, G. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.72 · Cited by 34
Graph-Grounded Goal Planning for Conversational Recommendation
Liu, Z., Zhou, D., Liu, H., Wang, H., Niu, Z.-Y., Wu, H., Che, W., Liu, T., & Xiong, H. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3147210 · Cited by 19
“Inspired by the work of MultiWOZ [40], we provide a task template to guide the annotation of each dialog.”
MethodRobust Dialogue State Tracking with Weak Supervision and Sparse Data
Heck, M. C., Lubis, N., van Niekerk, C., Feng, S., Geishauser, C., Lin, H.-C., & Gašić, M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00513 · Cited by 16
“In addition to MultiWOZ version 2.1, we test TripPy-R on 2.0, 2.2, 2.3 and 2.4 (Budzianowski et al., 2018; Zang et al., 2020; Han et al., 2021; Ye et al., 2021a).”
MethodInfluentialSelf-citationEmoUS: Simulating User Emotions in Task-Oriented Dialogues
Lin, H.-C., Feng, S., Geishauser, C., Lubis, N., van Niekerk, C., Heck, M. C., Ruppik, B., Vukovic, R., & Gašić, M. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3592092 · Cited by 36
“It contains user emotion annotations for human-human dialogues from MultiWOZ [2] and 1 𝑘 human-machine dialogues between volunteers and an RNN-based dialogue policy trained on MultiWOZ.”
MethodSelf-citationRewriting the Script: Adapting Text Instructions for Voice Interaction
Hwang, A., Oza, N., Callison-Burch, C., & Head, A. (2023) · Conference on Designing Interactive Systems · doi:10.1145/3563657.3596059 · Cited by 20
UniDU: Towards A Unified Generative Dialogue Understanding Framework
Chen, Z., Chen, L., Chen, B., Qin, L., Liu, Y., Su, Z., Lou, J., & Yu, K. (2022) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2022.sigdial-1.43 · Cited by 19
GCDST: A Graph-based and Copy-augmented Multi-domain Dialogue State Tracking
Wu, P., Zou, B., Jiang, R., & Aw, A. T. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.95 · Cited by 17
Larson, S., Zheng, A., Mahendran, A., Tekriwal, R., Cheung, A., Guldan, E., Leach, K., & Kummerfeld, J. K. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.650 · Cited by 28
“Crowdsourcing is widely used to collect data, including cases where workers are writing new text, such as questions (Rajpurkar et al., 2016), dialog (Budzianowski et al., 2018), and captions (Rus-sakovsky et al., 2015).”
BackgroundNeural Assistant: Joint Action Prediction, Response Generation, and Latent Knowledge ReasoningPreprint
Neelakantan, A., Yavuz, S., Narang, S., Prasad, V., Goodrich, B., Duckworth, D., Sankar, C., & Yan, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14613 · Cited by 18
“71 , significantly higher than the baseline system [2] that gets 18 .”
InfluentialDoc2Bot: Accessing Heterogeneous Documents via Conversational Bots
Fu, H., Zhang, Y., Yu, H., Sun, J., Huang, F., Si, L., Li, Y., & Nguyen, C. T. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.131 · Cited by 18
DeliData: A Dataset for Deliberation in Multi-party Problem Solving
Karadzhov, G., Stafford, T., & Vlachos, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610056 · Cited by 41
“[5], and the third dataset consists of transcripts of a popular TV show, which while containing true multi-party dialogues, are not collaborative.”
BackgroundMulti-Turn Hidden Backdoor in Large Language Model-powered Chatbot Models
Chen, B., Ivanov, N., Wang, G., & Yan, Q. (2024) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3634737.3656289 · Cited by 23
“Though there are a limited number of multi-turn conversational datasets, it would be our future work to test our model with datasets used by task-oriented dialogue systems [6].”
Zhao, T., & Kawahara, T. (2019) · Computer Speech & Language · doi:10.1016/j.csl.2019.03.001 · Cited by 15
“These differences make it difficult to compare models from the two tasks or to evaluate our models on task-oriented datasets such as DSTC2 (Henderson et al., 2014) and MultiWOZ (Budzianowski et al., 2018).”
MALA: Cross-Domain Dialogue Generation with Action Learning
Huang, X., Qi, J., Sun, Y., & Zhang, R. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6306 · Cited by 16
Arun, A., Batra, S., Bhardwaj, V. S., Challa, A., Dönmez, P., Heidari, P., Inan, H., Jain, S. S. B. C., Kumar, A., Mei, S., et al. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-industry.7 · Cited by 15
Few Shot Dialogue State Tracking using Meta-learning
Dingliwal, S., Gao, B., Agarwal, S., Lin, C., Chung, T., & Hakkani-Tür, D. Z. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.148 · Cited by 22
DTCRSKG: A Deep Travel Conversational Recommender System Incorporating Knowledge Graph
Fang, H., Chen, C., Long, Y., Xu, G., & Xiao, Y. (2022) · Mathematics · doi:10.3390/math10091402 · Cited by 16
“It is worth noting that MultiWOZ [43], CrossWOZ [44], and KdConv [45] are dialogue datasets that are related to the field of tourism.”
BackgroundA code-mixed task-oriented dialog dataset for medical domain
Dowlagar, S., & Mamidi, R. (2022) · Computer Speech & Language · doi:10.1016/j.csl.2022.101449 · Cited by 23
IRWoZ: Constructing an Industrial Robot Wizard-of-OZ Dialoguing Dataset
Li, C., Chrysostomou, D., Zhang, X., & Yang, H. (2023) · IEEE Access · doi:10.1109/access.2023.3259325 · Cited by 15
“The IRWoZ contains 401 dialogues, and follows a similar data structure as the most popular dialogue corpus MultiWoZ [18], which can be easily extended to other domains and reused for training various existing dialogue systems, e.g., [19] and [20].”
BackgroundMethodResultInfluential3D Dynamic Image Modeling Based on Machine Learning in Film and Television Animation
Wang, Y. (2023) · Journal of Multimedia Information System · doi:10.33851/jmis.2023.10.1.69 · Cited by 15
Is MultiWOZ a Solved Task? An Interactive TOD Evaluation Framework with User Simulator
Cheng, Q., Li, L., Quan, G., Gao, F., Mou, X., & Qiu, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.90 · Cited by 25
Self-Explanation Prompting Improves Dialogue Understanding in Large Language Models
Gao, H., Lin, T.-E., Li, H., Yang, M., Wu, Y., Ma, W., Huang, F., & Li, Y. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4shdjhs9d9ud · Cited by 23
Tracking Interaction States for Multi-Turn Text-to-SQL Semantic Parsing
Wang, R., Ling, Z.-H., Zhou, J., & Hu, Y. H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17646 · Cited by 24
Show Us the Way: Learning to Manage Dialog from DemonstrationsPreprint
Gordon-Hall, G., Gorinski, P. J., Λάμπουρας, Γ., & Iacobacci, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08114 · Cited by 15
“This problem is aggravated in systems that deal with multiple domains at the same time, as is the case for the MultiWOZ-based “travel agent” system in ConvLab.”
InfluentialDeep Learning Approaches for Multimodal Intent Recognition: A SurveyPreprint
Zhao, J., Wen, Y., Li, Q., Hu, M., Zhou, Y., Xue, J., Wu, J., Gao, Y., Wen, Z., Tao, J., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2507.22934 · Cited by 16
“Ask for help, Advise, Suggest, Invite, Plan, Arrange, Inform, Request, Select, Prevent, BookRestaurant, PlayMusic, Ad-dToPlaylist, Query, Directive, etc. [8, 21, 77, 84, 162, 163] Information and Declaration State facts or describe situations objectively.”
Hu, S., Yen, R., Yu, Z., Zhao, M., Seaborn, K., & Liu, C. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579591 · Cited by 17
“In particular, speech-based systems are expected to move towards greater complexity in response to the heightened demand for dynamic and open-ended interactions, potentially with non-voice complements [6, 63].”
BackgroundHu, S., Zhou, H., Hergul, M., Gritta, M., Zhang, G., Iacobacci, I., Vulić, I., & Korhonen, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00609 · Cited by 39
A multi-agent curiosity reward model for task-oriented dialogue systems
Sun, J., Kou, J., Hou, W., & Bai, Y. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110884 · Cited by 19
Beyond Persuasion: Towards Conversational Recommender System with Credible Explanations
Qin, P., Huang, C., Deng, Y., Lei, W., & Chua, T. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.247 · Cited by 31
Multi-Action Dialog Policy Learning with Interactive Human Teaching
Jhunjhunwala, M., Bryant, C., & Shah, P. (2020) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2020.sigdial-1.36 · Cited by 14
“However, human-human dialogs collected in a Wizard-of-Oz setup (Budzianowski et al., 2018; Eric et al., 2017) cannot cover all dialog states occurring in human-machine interactions, such as dialog states occurring due to system errors.”
BackgroundMethodInfluentialLeyzer: A Dataset for Multilingual Virtual Assistants
Sowański, M., & Janicki, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58323-1_51 · Cited by 14
Adapting Long Context NLM for ASR Rescoring in Conversational Agents
Shenoy, A., Bodapati, S., Sunkara, M., Ronanki, S., & Kirchhoff, K. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1849 · Cited by 19
“Conversations in goal or task oriented conversational interfaces, such as digital personal assistants or chatbots, typically span multiple turns of back and forth between a user and a bot [1, 2].”
BackgroundA Collaborative Multi-agent Reinforcement Learning Framework for Dialog Action Decomposition
Wang, H., & Wong, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.621 · Cited by 14
“We first evaluate 2-agent based models that factor the centralized spaces into two subspaces of the domain and joint intent-slot on 3 different domains sizes of 2, 4, and 7 on MultiWoz.”
BackgroundMethodInfluentialLearning Dialogue Representations from Consecutive Utterances
Zhou, Z., Zhang, D., Xiao, W., Dingwall, N., Ma, X., Arnold, A., & Xiang, B. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.55 · Cited by 26
Multimodal Dialogue State Tracking
Le, H., Chen, N. F., & Hoi, S. C. H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.248 · Cited by 18
A Unified Framework for Slot based Response Generation in a Multimodal Dialogue System
Firdaus, M., Madasu, A., & Ekbal, A. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15915-8 · Cited by 14
“With the release of MultiWoz [6], a task-oriented dialogue dataset, several works have focused on multi-domain dialogue generation.”
BackgroundOPERA: Harmonizing Task-Oriented Dialogs and Information Seeking Experience
Li, M., Peng, B., Gao, J., & Zhang, Z. (2024) · ACM Transactions on the Web · doi:10.1145/3623381 · Cited by 17
“We create a new dataset OB-MultiWOZ by leveraging existing datasets [4, 27].”
InfluentialSocial Intelligence Data Infrastructure: Structuring the Present and Navigating the Future
Li, M., Shi, W., Ziems, C., & Yang, D. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.163 · Cited by 19
Adapting language generation to dialogue environments and users for task-oriented dialogue systems
Ohashi, A., & Higashinaka, R. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100153 · Cited by 15
An Analysis of Mixed Initiative and Collaboration in Information-Seeking Dialogues
Vakulenko, S., Kanoulas, E., & de Rijke, M. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401297 · Cited by 14
Metadata
- Authors
- Budzianowski, P., Wen, T.-H., Tseng, B.-H., Casanueva, I., Ultes, S., Ramadan, O., & Gašić, M.
- Affiliations
- University of Cambridge (GB)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 5016-5026
- OpenAlex
- W2891732163
- Cited by
- 1,028
- References
- 48
- Citations by year
- 2026: 39 · 2025: 66 · 2024: 86 · 2023: 209 · 2022: 186 · 2021: 196 · 2020: 181 · 2019: 64 · 2018: 1
- FWCI
- 45.11
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/M018946/1, RG80792)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- MultiWOZ, task-oriented dialogue, multi-domain dialogue, dialogue modeling, dialogue dataset, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.