CoChat
Sign Up
PreprintCited by 3Speech and dialogue systemsAI in Service InteractionsIntelligent Tutoring Systems and Adaptive Learning

Leveraging Implicit Feedback from Deployment Data in Dialogue

Open access
Pang, Richard Yuanzhe, Roller, Stephen, Cho, Kyunghyun, He, He, Weston, Jason·2023·arXiv
DOI 10.48550/arxiv.2307.14117arXiv 2307.14117
Open this paper in CoChat — free accountOpen PDF

Abstract

We study improving social conversational agents by learning from natural dialogue between users and a deployed model, without extra annotations. To implicitly measure the quality of a machine-generated utterance, we leverage signals like user response length, sentiment and reaction of the future human utterances in the collected dialogue episodes. Our experiments use the publicly released deployment data from BlenderBot (Xu et al., 2023). Human evaluation indicates improvements in our new models over baseline responses; however, we find that some proxy signals can lead to more generations with undesirable properties as well. For example, optimizing for conversation length can lead to more controversial or unfriendly generations compared to the baseline, whereas optimizing for positive sentiment or reaction can decrease these behaviors.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.14117

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs can be easily led into veering off-topic.

    “This seems like a serious challenge as LLMs can be easily led into veering off-topic (Pang et al., 2023).”

    From NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023) · cites this paper as Pang et al., 2023

    The Behavior analysis section reports that some implicit-feedback signals produced more off-topic responses than the baseline, particularly signals involving conversation length. This supports the concern that dialogue models can veer off-topic, but the paper does not establish that LLMs in general are easily led to do so; its finding is specific to the signals it tested. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Prior feedback harvesting approaches are limited in their task application domain.

    “Most of these approaches are limited in their task application domain.”

    From User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal (2025) · cites this paper as Pang et al., 2023

    The Introduction and §3.1 describe harvesting feedback from human–bot conversations, and Related Work discusses other dialogue methods. The passages found when searching for task-specific approaches and domain limitations do not say that most prior feedback-harvesting approaches are limited in their task application domain. (LLM, conf=0.70)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 41

  1. Oluwatosin Adewumi, Rickard Brännvall, Nosheen Abid, Maryam Pahlavan, Sana Sabah Sabry, Foteini Liwicki, and Marcus Liwicki. 2022. Småprat: Di- alogpt for natural language generation of swedish dialogue by transfer learning. In 5th Northern Lights Deep Learning Conference (NLDL), Tromsø, Norway, January 10-12, 2022, volume 3. Septentrio Academic Publishing.
  2. Leonard Adolphs, Tianyu Gao, Jing Xu, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. 2023. The CRINGE loss: Learning what language not to model. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8854-8874, Toronto, Canada. Association for Computational Linguistics. · DOI
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. · arXiv
  4. Yejin Bang, Nayeon Lee, Etsuko Ishii, Andrea Madotto, and Pascale Fung. 2021. Assessing political pru- dence of open-domain chatbots. In Proceedings of the 22nd Annual Meeting of the Special Inter- est Group on Discourse and Dialogue, pages 548- 555, Singapore and Online. Association for Compu- tational Linguistics.
  5. Jose Camacho-Collados, Kiamehr Rezaee, Talayeh Riahi, Asahi Ushio, Daniel Loureiro, Dimosthe- nis Antypas, Joanne Boisson, Luis Espinosa-Anke, Fangyu Liu, Eugenio Martínez-Cámara, et al. 2022. TweetNLP: Cutting-Edge Natural Language Process- ing for Social Media. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing: System Demonstrations, Abu Dhabi, U.A.E. Association for Computational Lin- guistics.
  6. Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. ACM SIGKDD Explo- rations Newsletter, 19(2):25-35.
  7. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2020. The second conversational intel- ligence challenge (ConvAI2). In The NeurIPS'18 Competition: From Machine Learning to Intelligent Conversations, pages 187-208. Springer.
  8. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's guide to testing sta- tistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383-1392, Melbourne, Aus- tralia. Association for Computational Linguistics. · DOI · Paper page
  9. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feedback. In Thirty-seventh Conference on Neural Information Processing Systems.
  10. Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neu- ral approaches to conversational ai. In The 41st In- ternational ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1371- 1374.
  11. Sarik Ghazarian, Behnam Hedayatnia, Di Jin, Sijia Liu, Nanyun Peng, Yang Liu, and Dilek Hakkani-Tur. 2023. MERCY: Multiple response ranking concur- rently in realistic open-domain conversational sys- tems. In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dia- logue, pages 615-631, Prague, Czechia. Association for Computational Linguistics. · DOI
  12. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. ChatGPT outperforms crowd-workers for text- annotation tasks. arXiv preprint arXiv:2303.15056. · arXiv
  13. Fenfei Guo, Angeliki Metallinou, Chandra Khatri, Anirudh Raju, Anu Venkatesh, and Ashwin Ram. 2018. Topic-based evaluation for conversational bots. arXiv preprint arXiv:1801.03622. · arXiv
  14. Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 3667- 3684, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
  15. Robert P. Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Joe Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp, Xiaoding Lu, Thomas Rialan, and William Beauchamp. 2023. Rewarding chatbots for real-world engagement with millions of users. arXiv preprint arXiv:2303.06135. · arXiv
  16. Da Ju, Jing Xu, Y-Lan Boureau, and Jason Weston. 2022. Learning from data in the mixed adversarial non- adversarial case: Finding the helpers and ignoring the trolls. arXiv preprint arXiv:2208.03295. · arXiv
  17. Chandra Khatri, Behnam Hedayatnia, Anu Venkatesh, Jeff Nunn, Yi Pan, Qing Liu, Han Song, Anna Got- tardi, Sanjeev Kwatra, Sanju Pancholi, et al. 2018. Advancing the state of the art in open domain dia- log systems through the Alexa prize. arXiv preprint arXiv:1812.10757. · arXiv
  18. Ilia Kulikov, Jason Lee, and Kyunghyun Cho. 2019. Multi-turn beam search for neural dialogue modeling. arXiv preprint arXiv:1906.00141. · arXiv
  19. Jaewook Lee, Seongsik Park, Seong-Heum Park, Hongjin Kim, and Harksoo Kim. 2023. A framework for vision-language warm-up tasks in multimodal dialogue models. In Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, pages 2789-2799, Singapore. Associa- tion for Computational Linguistics.
  20. Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pas- cale Fung, Mohammad Shoeybi, and Bryan Catan- zaro. 2022. Factuality enhanced language models for open-ended text generation. In Advances in Neural Information Processing Systems.
  21. Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2017a. Dialogue learning with human-in-the-loop. In Inter- national Conference on Learning Representations.
  22. Jiwei Li, Will Monroe, and Dan Jurafsky. 2017b. Learn- ing to decode for future success. arXiv preprint arXiv:1701.06549. · arXiv
  23. Shikib Mehri and Maxine Eskenazi. 2020. Unsuper- vised evaluation of interactive dialog with DialoGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 225-235, 1st virtual meeting. Association for Computational Linguistics.
  24. Heather L O'Brien and Elaine G Toms. 2008. What is user engagement? A conceptual framework for defining user engagement with technology. Journal of the American society for Information Science and Technology, 59(6):938-955.
  25. Richard Yuanzhe Pang, He He, and Kyunghyun Cho. 2022. Amortized noisy channel neural machine trans- lation. In Proceedings of the 15th International Con- ference on Natural Language Generation, pages 131- 143, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics.
  26. Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur Parikh, and He He. 2023. Reward gaming in conditional text generation. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4746-4763, Toronto, Canada. Association for Computational Linguistics.
  27. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Sys- tems.
  28. Abigail See and Christopher Manning. 2021. Under- standing and predicting user dissatisfaction in a neu- ral generative chatbot. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Dis- course and Dialogue, pages 1-12, Singapore and Online. Association for Computational Linguistics.
  29. Igor Shalyminov, Ondřej Dušek, and Oliver Lemon. 2018. Neural response ranking for social conversa- tion: A data-efficient approach. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd Interna- tional Workshop on Search-Oriented Conversational AI, pages 1-8, Brussels, Belgium. Association for Computational Linguistics. · DOI
  30. Weiyan Shi, Emily Dinan, Kurt Shuster, Jason We- ston, and Jing Xu. 2022. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels. arXiv preprint arXiv:2210.15893. · arXiv
  31. Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022a. Language models that seek for knowl- edge: Modular search & generation for dialogue and prompt completion. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 373-393, Abu Dhabi, United Arab Emirates. Associ- ation for Computational Linguistics. · DOI
  32. Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, et al. 2022b. Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188. · arXiv
  33. Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characteriz- ing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471.
  34. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008- 3021.
  35. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial artificial artificial intel- ligence: Crowd workers widely use large language models for text production tasks. arXiv preprint arXiv:2306.07899. · arXiv
  36. Frank Wilcoxon. 1992. Individual comparisons by rank- ing methods. In Breakthroughs in Statistics: Method- ology and Distribution, pages 196-202. Springer.
  37. Jing Xu, Da Ju, Joshua Lane, Mojtaba Komeili, Eric Michael Smith, Megan Ung, Morteza Behrooz, William Ngan, Rashel Moritz, Sainbayar Sukhbaatar, et al. 2023. Improving open language models by learning from organic interactions. arXiv preprint arXiv:2306.04707. · arXiv
  38. Weizhe Yuan, Kyunghyun Cho, and Jason Weston. 2023. System-level natural language feedback. arXiv preprint arXiv:2306.13588. · arXiv
  39. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. arXiv preprint arXiv:2401.10020. · arXiv
  40. Yury Zemlyanskiy and Fei Sha. 2018. Aiming to know you better perhaps makes me a more engaging dia- logue partner. In Proceedings of the 22nd Confer- ence on Computational Natural Language Learning, pages 551-561, Brussels, Belgium. Association for Computational Linguistics. · DOI
  41. Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online. As- sociation for Computational Linguistics. · DOI · Paper page

Cited by

Sorted by citations

Authors
Pang, R. Y., Roller, S., Cho, K., He, H., & Weston, J.
Affiliations
New York University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2307.14117
arXiv
2307.14117
OpenAlex
W4385327996
Cited by
1
References
0
Citations by year
2023: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
implicit feedback, dialogue generation, sentiment, user response

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review