The Second Conversational Intelligence Challenge (ConvAI2)
Open accessAbstract
We describe the setting and results of the ConvAI2 NeurIPS competition that aims to further the state-of-the-art in open-domain chatbots. Some key takeaways from the competition are: (i) pretrained Transformer variants are currently the best performing models on this task, (ii) but to improve performance on multi-turn conversations with humans, future systems must go beyond single word metrics like perplexity to measure the performance across sequences of utterances (conversations) -- in terms of repetition, consistency and balance of dialogue acts (e.g. how many questions asked vs. answered).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 13
- [1][1] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243 , 2018. - · arXiv
- [2][2] Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. Generative deep neural networks for dialogue: A short review. arXiv preprint arXiv:1611.06216 , 2016. - · arXiv
- [3][3] Oriol Vinyals and Quoc Le. A neural conversational model. arXiv preprint arXiv:1506.05869 , 2015. [13 http://workshop.colips.org/dstc7/](http://workshop.colips.org/dstc7/) - · arXiv
- [4][4] Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155 , 2016. - · arXiv
- [5][5] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055 , 2015. - · arXiv
- [6][6] Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476 , 2017. - · arXiv
- [7][7] Chia-Wei Liu, Ryan Lowe, Iulian Vlad Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. CoRR , abs/1603.08023, 2016. - · arXiv
- [8][8] Ilya Kulikov, Alexander H. Miller, Kyunghyun Cho, and Jason Weston. Importance of a search strategy in neural dialogue modelling. CoRR , abs/1811.00907, 2018. - · arXiv
- [9][9] Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. arXiv preprint arXiv:1612.01627 , 2016. - · arXiv
- [10][10] Jason Weston, Emily Dinan, and Alexander H Miller. Retrieve and refine: Improved sequence generation models for dialogue. arXiv preprint arXiv:1808.04776 , 2018. - · arXiv
- [11][11] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018. - · arXiv
- [12][12] Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. Dialogue natural language inference. arXiv preprint arXiv:1811.00671 , 2018. - · arXiv
- [13][13] Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241 , 2018. · arXiv
Cited by
Sorted by citations
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Challenges in building intelligent open-domain dialog systems
Huang, M., Zhu, X., & Gao, J. (2020) · ACM Transactions on Information Systems · doi:10.1145/3383123 · Cited by 288 · see citing works
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.378 · Cited by 303 · see citing works
What makes a good conversation? How controllable attributes affect human judgments
See, A., Roller, S., Kiela, D., & Weston, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1170 · Cited by 247 · see citing works
Self-citationUSR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Mehri, S., & Eskenazi, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.64 · Cited by 149 · see citing works
Shen, J., Yang, B., Dudley, J. J., & Kristensson, P. O. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511145 · Cited by 48 · see citing works
Personalized Non-Player Characters: A Framework for Character-Consistent Dialogue Generation
Liu, X., Xie, Z., & Jiang, S. (2025) · AI · doi:10.3390/ai6050093 · Cited by 3 · see citing works
Kummara, H. (2026) · Cambridge Open Engage (preprint) · doi:10.33774/coe-2026-xpkvb
A Multi-Task Role-Playing Agent Capable of Imitating Character Linguistic Styles
Chen, S., Si, Q., Yang, C., Liang, Y., Lin, Z., Liu, H., & Wang, W. (2024) · arXiv · doi:10.48550/arxiv.2411.02457
- Authors
- Dinan, E., Logacheva, V., Malykh, V., Miller, A., Shuster, K., Urbanek, J., Kiela, D., Szlam, A., Serban, I., Lowe, R., Prabhumoye, S., Black, A. W., Rudnicky, A., Williams, J., Pineau, J., Burtsev, M., & Weston, J.
- Affiliations
- Meta (United States) (US)
- Moscow Institute of Physics and Technology (RU)
- Université de Montréal (CA)
- Carnegie Mellon University (US)
- Microsoft (United States) (US)
- Corresponding author
- Dinan, Emily
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- Pages
- 187-208
- ISSN
- 2520-131X (ISSN-L)
- arXiv
- 1902.00098
- OpenAlex
- W2913443447
- Cited by
- 355
- References
- 0
- Citations by year
- 2026: 5 · 2025: 23 · 2024: 15 · 2023: 95 · 2022: 54 · 2021: 68 · 2020: 63 · 2019: 32
- FWCI
- 67.41
- Type
- book chapter
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- The Springer series on challenges in machine learning (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Amazon Web Services
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- open-domain chatbots, conversational intelligence, multi-turn dialogue, human-chatbot interaction, pretrained transformer models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
