The DOI matches its Crossref record, with a small difference in title or year; 36 of the 38 references it cites were found in a registry or an index.
MuTual: A Dataset for Multi-Turn Dialogue Reasoning
Open accessAbstract
Non-task oriented dialogue systems have achieved great success in recent years due to largely accessible conversation data and the development of deep learning techniques.Given a context, current systems are able to yield a relevant and fluent response, but sometimes make logical mistakes because of weak reasoning capabilities.To facilitate the conversation reasoning research, we introduce Mu-Tual, a novel dataset for Multi-Turn dialogue Reasoning, consisting of 8,860 manually annotated dialogues based on Chinese student English listening comprehension exams.Compared to previous benchmarks for non-task oriented dialogue systems, MuTual is much more challenging since it requires a model that can handle various reasoning problems.Empirical results show that state-of-the-art methods only reach 71%, which is far behind the human performance of 94%, indicating that there is ample room for improving reasoning ability.MuTual is available at https://github. com/Nealcly/MuTual. * Contribution during internship at MSRA.M: Ma'am
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“There are some reasoning works in the area of dialogue [39, 83, 272], but few in reducing hallucinations.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [39]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
5th International Conference on Learning Representations (ICLR 17)
Havrylov, S., & Titov, I. (2017) · Cited by 3,503
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Speer, R., Chin, J., & Havasi, C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11164 · Cited by 2,312
What Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4828 · Cited by 1,953
Enhanced LSTM for Natural Language Inference
Chen, Q., Zhu, X., Ling, Z.-H., Wei, S., Jiang, H., & Inkpen, D. Z. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1152 · Cited by 1,315
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177
Neural Responding Machine for Short-Text Conversation
Shang, L., Lu, Z., & Li, H. (2015) · doi:10.3115/v1/p15-1152 · Cited by 1,146
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044
CoQA: A Conversational Question Answering Challenge
Reddy, S., Chen, D., & Manning, C. D. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00266 · Cited by 975
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1421 · Cited by 919
Levesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 879
ConceptNet 5.5: An Open Multilingual Graph of General KnowledgePreprint
Speer, R., Chin, J., & Havasi, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.03975 · Cited by 866
The TREC-8 Question Answering Track Report
Voorhees, E. M. (1999) · Cited by 831
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Zellers, R., Bisk, Y., Schwartz, R., & Choi, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1009 · Cited by 801
From Eliza to XiaoIce: Challenges and opportunities with social chatbots
Shum, H.-Y., He, X.-D., & Li, D. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700826 · Cited by 749
Wu, Y., Wu, W., Chen, X., Zhou, M., & Li, Z. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1046 · Cited by 538
Self-citationThink you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., & Roth, D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1023 · Cited by 407
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393 · cited 2×
“DROP (Dua et al., 2019) and COSMOS (Huang et al., 2019) focus on factual understanding and commonsense comprehension, respectively.”
BackgroundMulti-Turn Response Selection for Chatbots with Deep Attention Matching Network
Zhou, X., Lü, L., Dong, D., Liu, Y., Chen, Y., Zhao, W. X., Yu, D., & Wu, H. R. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1103 · Cited by 341
Neural Text Summarization: A Critical Evaluation
Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1051 · Cited by 335
The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems
Lowe, R., Pow, N., Serban, I. V., & Pineau, J. (2015) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w15-4640 · Cited by 334
BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
Xu, H., Liu, B., Shu, L., & Yu, P. S. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1242 · Cited by 324
Augmenting End-to-End Dialogue Systems With Commonsense Knowledge
Young, T., Cambria, E., Chaturvedi, I., Zhou, H., Biswas, S., & Huang, M. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11923 · Cited by 308
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
Huang, L., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1243 · Cited by 302
DREAM: A Challenge Data Set and Models for Dialogue-Based Reading Comprehension
Sun, K., Yu, D., Chen, J., Yu, D., Choi, Y., & Cardie, C. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00264 · Cited by 252
Dialogue Natural Language Inference
Welleck, S., Weston, J., Szlam, A., & Cho, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1363 · Cited by 249
Modeling Multi-turn Conversation with Deep Utterance AggregationPreprint
Zhang, Z., Li, J., Zhu, P., Zhao, H., & Liu, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.09102 · Cited by 218
Smart Reply: Automated Response Suggestion for EmailPreprint
Kannan, A., Kurach, K., Ravi, S., Kaufmann, T., Tomkins, A., Miklos, B., Corrado, G. S., Lukacs, L., Ganea, M., Young, P., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04870 · Cited by 148
Tao, C., Wu, W., Xu, C., Hu, W., Zhao, D., & Yan, R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1001 · Cited by 128
A Sequential Matching Framework for Multi-Turn Response Selection in Retrieval-Based Chatbots
Wu, Y., Wu, W., Chen, X., Xu, C., Li, Z., & Zhou, M. (2018) · Computational Linguistics · doi:10.1162/coli_a_00345 · Cited by 72
Self-citationImproving Open-Domain Dialogue Systems via Multi-Turn Incomplete Utterance Restoration
Pan, Z., Bai, K., Wang, Y., Zhou, L., & Liu, X. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1191 · Cited by 51
Domain adaptive training BERT for response selectionPreprint
Whang, T., Lee, D., Lee, C., Yang, K., Oh, D., & Lim, H. S. (2019) · arXiv (Cornell University) · Cited by 21
Research works printed but not linked: 3
- Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Pro- cessing Systems 30, pages 5998-6008.
- Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. In 7th International Conference on Learning Represen- tations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
- Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Aug- menting end-to-end dialogue systems with common- sense knowledge. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Arti- ficial Intelligence (IAAI-18), and the 8th AAAI Sym- posium on Educational Advances in Artificial Intel- ligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4970-4977.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Graph Neural Networks for Natural Language Processing: A Survey
Wu, L., Chen, Y., Shen, K., Guo, X., Gao, H., Li, S., Pei, J., & Long, B. (2023) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000096 · Cited by 417
Recent advances in deep learning based dialogue systems: a systematic survey
Ni, J., Young, T., Pandelea, V., Xue, F., & Cambria, E. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10248-8 · Cited by 313
LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
Jian, L., Cui, L., Liu, H., Huang, D., Wang, Y., & Zhang, Y. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/501 · Cited by 431
Self-citationDialogSum: A Real-Life Scenario Dialogue Summarization Dataset
Chen, Y., Liu, Y., Liang, C., & Zhang, Y. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.449 · Cited by 183
Self-citation
Metadata
- Authors
- Cui, L., Wu, Y., Liu, S., Zhang, Y., & Zhou, M.
- Affiliations
- Westlake University (CN)
- Microsoft Research Asia (China) (CN)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 1406-1416
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3034446185
- Cited by
- 110
- References
- 38
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 4 · 2023: 28 · 2022: 22 · 2021: 37 · 2020: 6
- FWCI
- 8.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Westlake University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- dialogue dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
