The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.
Rewarding Chatbots for Real-World Engagement with Millions of Users
Open accessAbstract
The emergence of pretrained large language models has led to the deployment of a range of social chatbots for chitchat. Although these chatbots demonstrate language ability and fluency, they are not guaranteed to be engaging and can struggle to retain users. This work investigates the development of social chatbots that prioritize user engagement to enhance retention, specifically examining the use of human feedback to efficiently develop highly engaging chatbots. The proposed approach uses automatic pseudo-labels collected from user interactions to train a reward model that can be used to reject low-scoring sample responses generated by the chatbot model at inference time. Intuitive evaluation metrics, such as mean conversation length (MCL), are introduced as proxies to measure the level of engagement of deployed chatbots. A/B testing on groups of 10,000 new daily chatbot users on the Chai Research platform shows that this approach increases the MCL by up to 70%, which translates to a more than 30% increase in user retention for a GPT-J 6B model. Future work aims to use the reward model to realise a data fly-wheel, where the latest user conversations can be used to alternately fine-tune the language model and the reward model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai, M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.01815 · Cited by 1,102 · cited 1×
“Human feedback has also been used in several other ways including constrained optimization (Achiam et al., 2017), control codes (Keskar et al., 2019) and expert iteration (Anthony et al., 2017; Silver et al., 2017).”
Method
Flagged first, then by citations
Research works printed but not linked: 41
- Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. 2017. Constrained policy optimization. CoRR, abs/1705.10528. · arXiv
- Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. Towards a human-like open- domain chatbot. CoRR, abs/2001.09977. · arXiv
- Thomas Anthony, Zheng Tian, and David Barber. 2017. Thinking fast and slow with deep learning and tree search. CoRR, abs/1705.08439. · arXiv
- Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernan- dez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. 2021. A general language assistant as a laboratory for alignment. CoRR, abs/2112.00861. · arXiv
- Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhen Guo, Zhibin Liu, and Xinchao Xu. 2020. PLATO-2: towards building an open-domain chatbot via curriculum learning. CoRR, abs/2006.16779. · arXiv
- Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Re- cent advances and new frontiers. SIGKDD Explor. Newsl., 19(2):25-35. · DOI · Paper page
- Ritvik Choudhary and Daisuke Kawahara. 2022. Grounding in social media: An approach to build- ing a chit-chat dialogue model. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pages 9-15, Hybrid: Seattle, Washing- ton + Online. Association for Computational Lin- guistics. · DOI
- Silje Christensen, Simen Johnsrud, Massimiliano Ruocco, and Heri Ramampiaro. 2018. Context- aware sequence-to-sequence models for conversa- tional systems. CoRR, abs/1805.08455. · arXiv
- Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep re- inforcement learning from human preferences. Ad- vances in neural information processing systems, 30.
- Heriberto Cuayáhuitl, Donghyeon Lee, Seonghan Ryu, Yongjin Cho, Sungja Choi, Satish Reddy Indurthi, Seunghak Yu, Hyungtak Choi, Inchul Hwang, and Ji- hie Kim. 2019. Ensemble-based deep reinforcement learning for chatbots. CoRR, abs/1908.10422. · arXiv
- Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language mod- els: A simple approach to controlled text generation. CoRR, abs/1912.02164. · arXiv
- Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, and Mark Cieliebak. 2020. Spot the bot: A robust and efficient framework for the evaluation of conversational dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3971-3984, Online. Association for Computa- tional Linguistics. · DOI
- Manyu Dhyani and Rajiv Kumar. 2021. An intelligent chatbot using deep learning with bidirectional rnn and attention model. Materials Today: Proceedings, 34:817-824. 3rd International Conference on Sci- ence and Engineering in Materials. · DOI
- Sarah E. Finch and Jinho D. Choi. 2020. Towards uni- fied dialogue system evaluation: A comprehensive analysis of current evaluation protocols. In Proceed- ings of the 21th Annual Meeting of the Special Inter- est Group on Discourse and Dialogue, pages 236- 245, 1st virtual meeting. Association for Computa- tional Linguistics.
- Iason Gabriel. 2020. Artificial intelligence, values and alignment. CoRR, abs/2001.09768. · arXiv
- Ryuichiro Higashinaka, Kenji Imamura, Toyomi Me- guro, Chiaki Miyazaki, Nozomi Kobayashi, Hi- roaki Sugiyama, Toru Hirano, Toshiro Makino, and Yoshihiro Matsuo. 2014. Towards an open-domain conversational system fully based on natural lan- guage processing. In Proceedings of COLING 2014, the 25th International Conference on Compu- tational Linguistics: Technical Papers, pages 928- 939, Dublin, Ireland. Dublin City University and As- sociation for Computational Linguistics.
- Adebayo Kolawole John, Luigi Di Caro, Livio Robaldo, and Guido Boella. 2017. Legalbot: A deep learning-based conversational agent in the legal do- main. In Natural Language Processing and Infor- mation Systems, pages 267-273, Cham. Springer In- ternational Publishing.
- Nitish Shirish Keskar, Bryan McCann, Lav R. Varsh- ney, Caiming Xiong, and Richard Socher. 2019. CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05858. · arXiv
- Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction. CoRR, abs/1811.07871. · arXiv
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting ob- jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 110-119, San Diego, California. Association for Computational Linguistics. · DOI
- Margaret Li, Jason Weston, and Stephen Roller. 2019. ACUTE-EVAL: improved dialogue evaluation with optimized questions and multi-turn comparisons. CoRR, abs/1909.03087. · arXiv
- Jorge J Moré. 2006. The levenberg-marquardt algo- rithm: implementation and theory. In Numerical Analysis: Proceedings of the Biennial Conference Held at Dundee, June 28-July 1, 1977, pages 105- 116. Springer.
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint arXiv:2203.02155. · arXiv
- Alexandros Papangelis, Paweł Budzianowski, Bing Liu, Elnaz Nouri, Abhinav Rastogi, and Yun-Nung Chen, editors. 2021. Proceedings of the 3rd Work- shop on Natural Language Processing for Conver- sational AI. Association for Computational Linguis- tics, Online.
- Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. CoRR, abs/2003.08271. · arXiv
- Nicole M. Radziwill and Morgan C. Benton. 2017. Evaluating quality of chatbots and intelligent conver- sational agents. CoRR, abs/1704.04579. · arXiv
- Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason We- ston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu- tational Linguistics: Main Volume, pages 300-325, Online. Association for Computational Linguistics. · DOI · Paper page
- Tanik Saikh, Sudip Kumar Naskar, Asif Ekbal, and Sivaji Bandyopadhyay. 2018. Textual entailment us- ing machine translation evaluation metrics. In Com- putational Linguistics and Intelligent Text Process- ing, pages 317-328, Cham. Springer International Publishing.
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. CoRR, abs/1707.06347. · arXiv
- Stuart M. Shieber. 1994. Lessons from a restricted tur- ing test. Commun. ACM, 37(6):70-78. · DOI
- Heung-Yeung Shum, Xiaodong He, and Di Li. 2018. From eliza to xiaoice: Challenges and opportunities with social chatbots. CoRR, abs/1801.01957. · arXiv
- Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive gen- eration of conversational responses. In Proceedings of the 2015 Conference of the North American Chap- ter of the Association for Computational Linguis- tics: Human Language Technologies, pages 196- 205, Denver, Colorado. Association for Computa- tional Linguistics. · DOI
- Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008- 3021.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30.
- Joseph Weizenbaum. 1966. Eliza-a computer pro- gram for the study of natural language communi- cation between man and machine. Commun. ACM, 9(1):36-45. · DOI · Paper page
- Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. CoRR, abs/2010.07079. · arXiv
- Rui Yan, Juntao Li, Zhou Yu, et al. 2022. Deep learn- ing for dialogue systems: Chit-chat and beyond. Foundations and Trends® in Information Retrieval, 15(5):417-589.
- Ruolan Yang, Zitong Li, Haifeng Tang, and Kenny Zhu. 2022. ChatMatch: Evaluating chatbots by au- tonomous chat tournaments. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 7579-7590, Dublin, Ireland. Association for Com- putational Linguistics. · DOI
- Zaib, Quan Z. Sheng, and Wei Emma Zhang. 2020. A short survey of pre-trained language mod- els for conversational ai-a new age in nlp. In Proceedings of the Australasian Computer Science Week Multiconference, ACSW '20, New York, NY, USA. Association for Computing Machinery. · DOI
- Xinyan Zhao, Bin He, Yasheng Wang, Yitong Li, Fei Mi, Yajiao Liu, Xin Jiang, Qun Liu, and Huan- huan Chen. 2022. UniDS: A unified dialogue sys- tem for chit-chat and task-oriented dialogues. In Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering, pages 13-22, Dublin, Ireland. Association for Computational Linguistics. · DOI
- Zhenyi Zhu. 2022. A simple survey of pre-trained lan- guage models. Preprints.org 202208.0238.
Cited by
Sorted by citations
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candes, E., & Hashimoto, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1025 · Cited by 76
Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025) · arXiv · doi:10.48550/arxiv.2501.19393 · Cited by 12
“First, we need to find the magnitude squared of 𝐛 , which is ‖ ‖ 2 .”
LatentEvolve: Self-Evolving Test-Time Scaling in Latent Space
Zhang, G., Meng, F., Wan, G., Li, Z., Wang, K., Yin, Z., Bai, L., & Yan, S. (2025) · arXiv · doi:10.48550/arxiv.2509.24771
“Likewise, in “sampling-and-voting” scaling methods (Brown et al., 2024; Irvine et al., 2023), prior successes in selecting the correct answer do not inform or refine future selection strategies.”
Metadata
- Authors
- Irvine, R., Boubert, D., Raina, V., Liusie, A., Zhu, Z., Mudupalli, V., Korshuk, A., Liu, Z., Cremer, F., Assassi, V., Beauchamp, C.-C., Lu, X., Rialan, T., & Beauchamp, W.
- Affiliations
- University of Cambridge (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2303.06135
- OpenAlex
- W4324109433
- Cited by
- 6
- References
- 1
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 2 · 2023: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- social chatbots, user engagement, user retention, reward modeling, pseudo-labeling, human feedback, rejection sampling, A/B testing, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
