Learning from Dialogue after Deployment: Feed Yourself, Chatbot!
Open accessThe DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Abstract
The majority of conversations a dialogue agent sees over its lifetime occur after it has already been trained and deployed, leaving a vast store of potential training signal untapped.In this work, we propose the self-feeding chatbot, a dialogue agent with the ability to extract new training examples from the conversations it participates in.As our agent engages in conversation, it also estimates user satisfaction in its responses.When the conversation appears to be going well, the user's responses become new training examples to imitate.When the agent believes it has made a mistake, it asks for feedback; learning to predict the feedback that will be given improves the chatbot's dialogue abilities further.On the PERSONACHAT chitchat dataset with over 131k training examples, we find that learning from dialogue with a selffeeding chatbot significantly improves performance, regardless of the amount of traditional supervision.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Most of these approaches are limited in their task application domain.”
From User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal (2025) · cites this paper as Hancock et al., 2019
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
(2017) · Cited by 11,625
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · cited 1×
“Tokens are embedded with fastText (Bojanowski et al., 2017) 300-dimensional embeddings.”
BackgroundKruger, J., & Dunning, D. (1999) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.77.6.1121 · Cited by 7,731
An Overview of Multi-Task Learning in Deep Neural Networks
Ruder, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.05098 · Cited by 2,265
Toward an architecture for never-ending language learning
Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka, E., & Mitchell, T. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7519 · Cited by 2,000 · cited 2×
“, 2013) and never-ending (language) learning (Carlson et al., 2010) are related to the topics dis-”
BackgroundPersonalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177 · cited 11×
“Our experiments use the PERSONACHAT distribution that was released as a part of the ConvAI2 (Burtsev et al., 2018) challenge.”
BackgroundMethodInfluentialSelf-citationA Persona-Based Neural Conversation Model
Li, J., Galley, M., Brockett, C., Spithourakis, G., Gao, J., & Dolan, B. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1094 · Cited by 898
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3157794.3157797 · Cited by 777
Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset
Rashkin, H., Smith, E. M., Li, M., & Boureau, Y.-L. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1534 · Cited by 771
A stochastic model of human-machine interaction for learning dialog strategies
Levin, E., Pieraccini, R., & Eckert, W. (2000) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/89.817450 · Cited by 516
Snorkel: rapid training data creation with weak supervision
Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., & Ré, C. (2020) · The VLDB Journal · doi:10.1007/s00778-019-00552-1 · Cited by 510
SCHATZMANN, J., WEILHAMMER, K., STUTTLE, M., & YOUNG, S. (2006) · The Knowledge Engineering Review · doi:10.1017/s0269888906000944 · Cited by 336 · cited 1×
“…et al., 2018; Hixon et al., 2015; Pappu and Rudnicky, 2013), and more commonly in reinforcement learning settings, where the feedback is a scalar rather than the dialogue messages themselves (Levin et al., 2000; Schatzmann et al., 2006; Rieser and Lemon, 2011; Liu et al., 2018; Hong et al., 2019).”
BackgroundParlAI: A Dialog Research Software Platform
Miller, A., Feng, W., Batra, D., Bordes, A., Fisch, A., Lu, J., Parikh, D., & Weston, J. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d17-2014 · Cited by 315
Self-citation(2013) · Cited by 290
Learning to Extract Relations from the Web using Minimal Supervision
Bunescu, R., & Mooney, R. J. (2007) · Cited by 261
Training Millions of Personalized Dialogue Agents
Mazaré, P.-E., Humeau, S., Raison, M., & Bordes, A. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1298 · Cited by 226
Self-citationA Hierarchical Multi-Task Approach for Learning Embeddings from Semantic Tasks
Sanh, V., Wolf, T., & Ruder, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016949 · Cited by 203
A Deep Reinforcement Learning Chatbot
Serban, I. V., Sankar, C., Germain, M., Zhang, S., Lin, Z., Subramanian, S., Kim, T., Pieper, M., Chandar, S., Ke, N. R., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.02349 · Cited by 185
Who are the Turkers? Worker Demographics in Amazon Mechanical Turk
Ross, J. A., Zaldivar, A., Irani, L. C., & Tomlinson, B. (2009) · Sound Ideas (University of Puget Sound) · Cited by 177
A Survey of Available Corpora for Building Data-Driven Dialogue Systems
Serban, I. V., Lowe, R., Henderson, P. B., Charlin, L., & Pineau, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.05742 · Cited by 168
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · arXiv · doi:10.48550/arxiv.1801.07243 · Cited by 165
Self-citationA Dual Approach to Scalable Verification of Deep Networks
Krishnamurthy, Dvijotham, Stanforth, R., Gowal, S., Mann, T., & Kohli, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.06567 · Cited by 162
Engaging Image Captioning via Personality
Shuster, K., Humeau, S., Hu, H., Bordes, A., & Weston, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01280 · Cited by 158
Self-citationRao, S., & Daumé III, H. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1255 · Cited by 152
Liu, B., Tür, G., Hakkani-Tür, D., Shah, P., & Heck, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1187 · Cited by 148 · cited 4×
“requested, existing methods typically require accompanying scalar rewards or adherence to particular templates or structure to ensure that the feedback is usable by the model (Rieser and Lemon, 2011; Zhang et al., 2017; Liu et al., 2018).”
BackgroundMethodInfluentialReinforcement Learning for Adaptive Dialogue Systems
Rieser, V., & Lemon, O. (2011) · Springer eBooks · doi:10.1007/978-3-642-24942-6 · Cited by 117
Instructive feedback: Review of parameters and effects
Werts, M. G., Wolery, M. R., Holcombe, A., & Gast, D. L. (1995) · Journal of Behavioral Education · doi:10.1007/bf02110214 · Cited by 113
Training Classifiers with Natural Language Explanations
Hancock, B., Varma, P., Wang, S., Bringmann, M., Liang, P., & Ré, C. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1175 · Cited by 109
Self-citationLearning Knowledge Graphs for Question Answering through Conversational Dialog
Hixon, B., Clark, P., & Hajishirzi, H. (2015) · Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.3115/v1/n15-1086 · Cited by 105
Image-Chat: Engaging Grounded Conversations
Shuster, K., Humeau, S., Bordes, A., & Weston, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.219 · Cited by 88
Self-citationDialog-based Language Learning
Weston, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.06045 · Cited by 76
Self-citationLearning through Dialogue Interactions by Asking Questions
Li, J., Miller, A., Chopra, S., Ranzato, M., & Weston, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.04936 · Cited by 74
Self-citationLearning Personalized End-to-End Goal-Oriented Dialog
Luo, L., Huang, W., Zeng, Q., Nie, Z., & Sun, X. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016794 · Cited by 73
LEARNING TO GENERATE ARTIFICIAL FOVEA TRAJECTORIES FOR TARGET DETECTION
Schmidhuber, J., & Huber, R. (1991) · International Journal of Neural Systems · doi:10.1142/s012906579100011x · Cited by 66
Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation Models
Luan, Y., Brockett, C., Dolan, B., Gao, J., & Galley, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.07388 · Cited by 66
I Know the Feeling: Learning to Converse with EmpathyPreprint
Rashkin, H., Smith, E. M., Li, M., & Boureau, Y.-L. (2018) · arXiv (Cornell University) · Cited by 50
End-to-end optimization of goal-driven and visually grounded dialogue systemsPreprint
Strub, F., de Vries, H., Mary, J., Piot, B., Courville, A. C., & Pietquin, O. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.05423 · Cited by 50
Towards a Continuous Knowledge Learning Engine for Chatbots
Mazumder, S., Ma, N., & Liu, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06024 · Cited by 23
Engaging Image Chat: Modeling Personality in Grounded Dialogue.Preprint
Shuster, K., Humeau, S., Bordes, A., & Weston, J. (2018) · arXiv (Cornell University) · Cited by 20
Self-citationBeyond Turing: Intelligent Agents Centered on the UserPreprint
Eskénazi, M., Mehri, S., Razumovskaia, E., & Zhao, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.06613 · Cited by 15
Bootstrapping Conversational Agents with Weak Supervision
Mallinar, N., Shah, A., Ugrani, R., Gupta, A. K., Gurusankar, M., Ho, T. K., Liao, Q. V., Zhang, Y., Bellamy, R., Yates, R., et al. (2019) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v33i01.33019528 · Cited by 14
Interactional Feedback and the Impact of Attitude and Motivation on Noticing L2 Form
Bassiri, M. A. (2011) · English Language and Literature Studies · doi:10.5539/ells.v1n2p61 · Cited by 13
Predicting Tasks in Goal-Oriented Spoken Dialog Systems using Semantic Knowledge Bases
Pappu, A., & Rudnicky, A. I. (2013) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · Cited by 12
Listen, Interact and Talk: Learning to Speak via InteractionPreprint
Zhang, H., Yu, H., & Xu, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09906 · Cited by 11
Learning Deep Hidden Nonlinear Dynamics from Aggregate DataPreprint
Wang, Y., Dai, B., Kong, L., Erfani, S., Bailey, J., & Zha, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.08237 · Cited by 10
Hashimoto, C., & Sassano, M. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3185992 · Cited by 9
Interactive Language Acquisition with One-shot Visual Concept Learning through a Conversational Game
Zhang, H., Yu, H., & Xu, W. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1243 · Cited by 9
Towards an Engine for Lifelong Interactive Knowledge Learning in Human-Machine ConversationsPreprint
Mazumder, S., Ma, N., & Liu, B. (2018) · arXiv (Cornell University) · Cited by 8
Interactive Language Acquisition with One-shot Visual Concept Learning through a Conversational GamePreprint
Zhang, H., Yu, H., & Xu, W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.00462 · Cited by 4
Learning Personalized End-to-End Goal-Oriented DialogPreprint
Luo, L., Huang, W., Zeng, Q., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.04604 · Cited by 4
(2000) · CrossRef Listing of Deleted DOIs · doi:10.1162/153244302760185243
Research works printed but not linked: 3
- T. Hong, O. Kwon, and Y. Kim. 2019. An end-to- end trainable task-oriented dialog system with hu- man feedback. In Association for the Advancement of Artificial Intelligence (AAAI).
- J. Li, A. H. Miller, S. Chopra, M. Ranzato, and J. We- ston. 2017a. Dialogue learning with human-in-the- loop. In International Conference on Learning Rep- resentations (ICLR).
- S. Tong and D. Koller. 2001. Support vector ma- chine active learning with applications to text clas- sification. Journal of machine learning research, 2(0):45-66.
Cited by
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
“Dialog based learning (Weston, 2016; Li et al., 2016; Hancock et al., 2019) develops techniques to learn from interactive natural language feedback in dialog.”
BackgroundSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
“The PersonaChat datasets [31, 228] have accelerated research into persona consistency [68, 91, 126, 208, 219, 224, 232].”
BackgroundChatbots: History, technology, and applications
Adamopoulou, E., & Moussiades, L. (2020) · Machine Learning with Applications · doi:10.1016/j.mlwa.2020.100006 · Cited by 1,090
A survey of human-in-the-loop for machine learning
Wu, X., Xiao, L., Sun, Y., Zhang, J., Ma, T., & He, L. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.05.014 · Cited by 800
Recipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784
Self-citation
Metadata
- Authors
- Hancock, B., Bordes, A., Mazare, P.-E., & Weston, J.
- Affiliations
- Stanford University (US)
- Meta (United Kingdom) (GB)
- ORCID
- Hancock, Braden
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 3667-3684
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2910458567
- Cited by
- 161
- References
- 53
- Citations by year
- 2026: 6 · 2025: 14 · 2024: 13 · 2023: 21 · 2022: 23 · 2021: 41 · 2020: 33 · 2019: 10
- FWCI
- 10.24
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- imitation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.