The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Deep Reinforcement Learning for Dialogue Generation
Open accessAbstract
Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be shortsighted, predicting utterances one at a time while ignoring their influence on future outcomes.Modeling the future direction of a dialogue is crucial to generating coherent, interesting dialogues, a need which led traditional NLP models of dialogue to draw on reinforcement learning.In this paper, we show how to integrate these goals, applying deep reinforcement learning to model future reward in chatbot dialogue.The model simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity, coherence, and ease of answering (related to forward-looking function).We evaluate our model on diversity, length as well as with human judges, showing that the proposed algorithm generates more interactive responses and manages to foster a more sustained conversation in dialogue simulation.This work marks a first step towards learning a neural conversational model based on the long-term success of dialogues.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 2×
“A curriculum learning strategy is adopted (Bengio et al., 2009) as in Ranzato et al.”
MethodSequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732 · Cited by 1,426
Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400
On the semantics and pragmatics of linguistic feedback
ALLWOOD, J., NIVRE, J., & AHLSÉN, E. (1992) · Journal of Semantics · doi:10.1093/jos/9.1.1 · Cited by 442
Cited by
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177
Interpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958
The design and implementation of Xiaoice, an empathetic social chatbot
Zhou, L., Gao, J., Li, D., & Shum, H.-Y. (2020) · Computational Linguistics · doi:10.1162/coli_a_00368 · Cited by 777
Self-citationDeep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Zhao, T., Zhao, R., & Eskenazi, M. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1061 · Cited by 736
QuAC: Question Answering in Context
Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W.-T., Choi, Y., Liang, P., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1241 · Cited by 714
“19 Dialog fits into an increasing interest in open domain dialog, mostly studied in the context of social chit-chat (Li et al., 2016; Ritter et al., 2011; Fang et al., 2017; Ghazvininejad et al., 2018).”
BackgroundA Survey of Deep Learning: Platforms, Applications and Emerging Research Trends
Hatcher, W. G., & Yu, W. (2018) · IEEE Access · doi:10.1109/access.2018.2830661 · Cited by 654
A Survey on Dialogue Systems: Recent Advances and New Frontiers
Chen, H., Liu, X., Yin, D., & Tang, J. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3166054.3166058 · Cited by 579
A Literature Survey of Recent Advances in Chatbots
Caldarini, G., Jaf, S., & McGarry, K. (2022) · Information · doi:10.3390/info13010041 · Cited by 470
(2020) · arXiv (Cornell University) · Cited by 461
Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.121 · Cited by 445
Deal or No Deal? End-to-End Learning of Negotiation Dialogues
Lewis, M., Yarats, D., Dauphin, Y., Parikh, D., & Batra, D. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1259 · Cited by 347
Deep Reinforcement Learning for Sequence-to-Sequence Models
Keneshloo, Y., Shi, T., Ramakrishnan, N., & Reddy, C. K. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2929141 · Cited by 230
Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E., Japkowicz, N., & Viktor, H. (2022) · arXiv · doi:10.48550/arxiv.2210.07321 · Cited by 196
“Deep reinforcement learning (RL) has been used with neural networks to learn policy gradient methods that reward text characteristics associated with high-quality text generation [111].”
Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E. N., Japkowicz, N., & Viktor, H. L. (2023) · IEEE Access · doi:10.1109/access.2023.3294090 · Cited by 196
Learning to Write with Cooperative Discriminators
Holtzman, A., Buys, J., Forbes, M., Bosselut, A., Golub, D., & Choi, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1152 · Cited by 192
Survey on reinforcement learning for language processing
Uc-Cetina, V., Navarro-Guerrero, N., Martin-Gonzalez, A., Weber, C., & Wermter, S. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10205-5 · Cited by 177
Conversational Agents: Goals, Technologies, Vision and Challenges
Allouch, M., Azaria, A., & Azoulay, R. (2021) · Sensors · doi:10.3390/s21248448 · Cited by 168
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · arXiv · doi:10.48550/arxiv.1801.07243 · Cited by 165
AI-based conversational agents: A scoping review from technologies to future directions
Kusal, S., Patil, S., Choudrie, J., Kotecha, K., Mishra, S., & Abraham, A. (2022) · IEEE Access · doi:10.1109/access.2022.3201144 · Cited by 140
Mulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132
Schöbel, S., Schmitt, A., Benner, D., Saqr, M., Janson, A., & Leimeister, J. M. (2024) · Information Systems Frontiers · doi:10.1007/s10796-023-10375-9 · Cited by 127
Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., & Picard, R. (2019) · arXiv · doi:10.48550/arxiv.1907.00456 · Cited by 126
Iteratively Questioning and Answering for Interpretable Legal Judgment Prediction
Zhong, H., Wang, Y., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5479 · Cited by 113
Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Nguyen, K., Daumé III, H., & Boyd-Graber, J. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1153 · Cited by 110
Beyond Dyadic Interactions: Considering Chatbots as Community Members
Seering, J., Luria, M., Kaufman, G., & Hammer, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300680 · Cited by 97
Neural Approaches to Conversational AI
Gao, J., Galley, M., & Li, L. (2018) · Proceedings of ACL 2018, Tutorial Abstracts · doi:10.18653/v1/p18-5002 · Cited by 73
Self-citationRobust Text-to-SQL Generation with Execution-Guided Decoding
Wang, C., Tatwawadi, K., Brockschmidt, M., Huang, P.-S., Mao, Y., Polozov, O., & Singh, R. (2018) · arXiv · Cited by 64
Liang, Y., Meng, F., Zhang, Y., Chen, Y., Xu, J., & Zhou, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17575 · Cited by 63
Ramamurthy, R., Ammanabrolu, P., Brantley, K., Hessel, J., Sifa, R., Bauckhage, C., Hajishirzi, H., & Choi, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.01241 · Cited by 56
Effects of Communication Directionality and AI Agent Differences in Human-AI Interaction
Ashktorab, Z., Dugan, C., Johnson, J., Pan, Q., Zhang, W., Kumaravel, S., & Campbell, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445256 · Cited by 50
Learning towards conversational AI: A survey
Fu, T., Gao, S., Zhao, X., Wen, J.-R., & Yan, R. (2022) · AI Open · doi:10.1016/j.aiopen.2022.02.001 · Cited by 46
Reinforced Dynamic Reasoning for Conversational Question Generation
Pan, B., Li, H., Yao, Z., Cai, D., & Sun, H. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1203 · Cited by 40
A Distributional Approach to Controlled Text Generation
Khalifa, M., Elsahar, H., & Dymetman, M. (2020) · arXiv · doi:10.48550/arxiv.2012.11635 · Cited by 39
Putting the Horse before the Cart: A Generator-Evaluator Framework for Question Generation from Text
Kumar, V., Ramakrishnan, G., & Li, Y.-F. (2019) · Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/k19-1076 · Cited by 46
Memory-Augmented Dialogue Management for Task-Oriented Dialogue Systems
Zhang, Z., Huang, M., Zhao, Z., Ji, F., Chen, H., & Zhu, X. (2019) · ACM Transactions on Information Systems · doi:10.1145/3317612 · Cited by 37
Memory-augmented Dialogue Management for Task-oriented Dialogue Systems
Zhang, Z., Huang, M., Zhao, Z., Ji, F., Chen, H., & Zhu, X. (2018) · arXiv · doi:10.48550/arxiv.1805.00150 · Cited by 36
Lifelong and Continual Learning Dialogue Systems: Learning during Conversation
Liu, B., & Mazumder, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i17.17768 · Cited by 34
Improvised Theatre Alongside Artificial Intelligences
Mathewson, K., & Mirowski, P. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v13i1.12926 · Cited by 31
Natural Language Generation Using Sequential Models: A Survey
Pandey, A. K., & Roy, S. S. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11281-6 · Cited by 29
Reinforcement learning and bandits for speech and language processing: Tutorial, review and outlook
Lin, B. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122254 · Cited by 28
Yi, S., Goel, R., Khatri, C., Cervone, A., Chung, T., Hedayatnia, B., Venkatesh, A., Gabriel, R., & Hakkani-Tur, D. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8608 · Cited by 26
Qi, P., Zhang, Y., & Manning, C. D. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.3 · Cited by 23
Caution Ahead: Numerical Reasoning and Look‐Ahead Bias in AI Models
LEVY, B. (2026) · Journal of Accounting Research · doi:10.1111/1475-679x.70058 · Cited by 20
Refocusing on Relevance: Personalization in NLG
Dudy, S., Bedrick, S., & Webber, B. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.421 · Cited by 17
What's to Know? Uncertainty as a Guide to Asking Goal-Oriented Questions
Abbasnejad, E., Wu, Q., Shi, Q., & van den Hengel, A. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00428 · Cited by 16
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9
Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
Fan, X., Lyu, Y., Liang, P. P., Salakhutdinov, R., & Morency, L.-P. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.758 · Cited by 3
Metadata
- Authors
- Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., & Gao, J.
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Pages
- 1192-1202
- OpenAlex
- W2963167310
- Cited by
- 1,204
- References
- 7
- FWCI
- 87.13
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
