PreprintCited by 0AI-based Problem Solving and PlanningReinforcement Learning in RoboticsMachine Learning and Algorithms
Retrospective Learning from Interactions
Open accessChen, Zizhao, Gul, Mustafa Omer, Chen, Yiwei, Geng, Gloria, Wu, Anne, Artzi, Yoav·2024·arXiv
DOI 10.48550/arxiv.2410.13852arXiv 2410.13852
Abstract
Multi-turn interactions between large language models (LLMs) and users naturally include implicit feedback signals. If an LLM responds in an unexpected way to an instruction, the user is likely to signal it by rephrasing the request, expressing frustration, or pivoting to an alternative task. Such signals are task-independent and occupy a relatively constrained subspace of language, allowing the LLM to identify them even if it fails on the actual task. We introduce ReSpect, a method to learn from such signals in past interactions via retrospection without additional annotations. We deploy ReSpect in a new multimodal interaction scenario, where humans instruct a multimodal LLM to solve an abstract reasoning task with a combinatorial solution space. Through thousands of interactions with humans, we show how ReSpect gradually improves task completion rate from 31% to 82%, all without any external annotation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
DataCite · 10.48550/arxiv.2410.13852
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior feedback harvesting approaches are limited in their task application domain.
“Most of these approaches are limited in their task application domain.”
From User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal (2025) · cites this paper as Chen et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Research works printed but not linked: 46
- Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ah- met Üstün, and Sara Hooker. 2024. Back to basics: Revisiting REINFORCE-style optimization for learn- ing from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12248-12267, Bangkok, Thailand. Association for Computational Linguistics. · DOI
- Abdullah Almaatouq, Joshua Becker, James P. Houghton, Nicolas Paton, Duncan J. Watts, and Mark E. Whiting. 2021. Empirica: A virtual lab for high-throughput macro-level experiments. Behavior Research Methods, 53(5):2158-2171. · DOI
- Cem Anil, Guodong Zhang, Yuhuai Wu, and Roger Grosse. 2021. Learning to Give Checkable An- swers with Prover-Verifier Games. Preprint, arXiv:2108.12099. · DOI · Paper page
- Yoav Artzi and Luke Zettlemoyer. 2011. Bootstrapping semantic parsers from conversations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 421-432, Edinburgh, Scotland, UK. Association for Computational Lin- guistics.
- Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christo- pher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott John- ston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bow- man, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022. Constitutional AI: Harmlessness from AI Feedback. Preprint, arXiv:2212.08073. · DOI
- Collin Burns, Pavel Izmailov, Jan Hendrik Kirch- ner, Bowen Baker, Leo Gao, Leopold Aschenbren- ner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. 2023. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. Preprint, arXiv:2312.09390. · arXiv
- Zihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei, Xi- aocheng Feng, Wanxiang Che, Min Li, and Libo Qin. 2024. Comt: A novel benchmark for chain of multi-modal thought on large vision-language mod- els. arXiv preprint arXiv:2412.12932. · arXiv
- Herbert H Clark and Deanna Wilkes-Gibbs. 1986. Re- ferring as a collaborative process. Cognition, 22(1):1- 39.
- Shachar Don-Yehiya, Leshem Choshen, and Omri Abend. 2024. Learning from Naturally Occurring Feedback. Preprint, arXiv:2407.10944. · arXiv
- Anna Effenberger, Rhia Singh, Eva Yan, Alane Suhr, and Yoav Artzi. 2021. Analysis of language change in collaborative instruction following. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2803-2811, Punta Cana, Do- minican Republic. Association for Computational Linguistics. · DOI
- Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. KTO: Model Alignment as Prospect Theoretic Optimiza- tion. Preprint, arXiv:2402.01306. · DOI · Paper page
- Jean E Fox Tree. 1999. Listening in on monologues and dialogues. Discourse processes, 27(1):35-53.
- Ge Gao, Hung-Ting Chen, Yoav Artzi, and Eunsol Choi. 2023. Continually improving extractive QA via hu- man feedback. In Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing, pages 406-423, Singapore. Association for Computational Linguistics. · DOI
- Ge Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro, and Dipendra Misra. 2024. Aligning LLM Agents by Learning Latent Preference from User Ed- its. Preprint, arXiv:2404.15269. · DOI · Paper page
- Noah D Goodman and Michael C Frank. 2016. Prag- matic language interpretation as probabilistic infer- ence. Trends in cognitive sciences, 20(11):818-829.
- Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 3667- 3684, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
- Robert Hawkins, Minae Kwon, Dorsa Sadigh, and Noah Goodman. 2020a. Continual Adaptation for Effi- cient Machine Communication. In Proceedings of the 24th Conference on Computational Natural Lan- guage Learning, pages 408-419, Online. Association for Computational Linguistics. · DOI
- Robert D. Hawkins, Michael C. Frank, and Noah D. Goodman. 2020b. Characterizing the Dynamics of Learning in Repeated Reference Games. Cognitive Science, 44(6):e12845. · DOI
- William S. Horton and Richard J. Gerrig. 2002. Speak- ers' experiences and audience design: Knowing when and knowing how to adjust utterances to addressees. Journal of Memory and Language, 47(4):589-606. · DOI
- Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
- Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert Hawkins, and Yoav Artzi. 2022. Abstract Visual Reasoning with Tangram Shapes. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 582-601, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI
- Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. 2024. Prover- Verifier Games improve legibility of LLM outputs. Preprint, arXiv:2407.13692. · DOI · Paper page
- Noriyuki Kojima, Alane Suhr, and Yoav Artzi. 2021. Continual Learning for Grounded Instruction Gen- eration by Observing Human Following Behavior. Transactions of the Association for Computational Linguistics, 9:1303-1319. · DOI
- Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. 2024. Training language models to self- correct via reinforcement learning. arXiv preprint arXiv:2409.12917. · arXiv
- Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. 2024. What matters when building vision-language models? Preprint, arXiv:2405.02246. · DOI
- Harold J Leavitt and Ronald AH Mueller. 1951. Some effects of feedback on communication. Human rela- tions, 4(4):401-410.
- Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2017. Di- alogue learning with human-in-the-loop. In Interna- tional Conference on Learning Representations.
- Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxi- ang Gu, and Tianyi Zhou. 2024. Selective reflection- tuning: Student-selected data recycling for LLM instruction-tuning. In Findings of the Association for Computational Linguistics ACL 2024, pages 16189- 16211, Bangkok, Thailand and virtual meeting. As- sociation for Computational Linguistics.
- Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin A Raf- fel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Ad- vances in Neural Information Processing Systems, 35:1950-1965.
- Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh, and Peter Clark. 2023. Self-refine: Itera- tive refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Sys- tems.
- Dipendra Misra, John Langford, and Yoav Artzi. 2017. Mapping instructions and visual observations to ac- tions with reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 1004-1015, Copen- hagen, Denmark. Association for Computational Lin- guistics. · DOI
- Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica. 2018. Ray: A distributed framework for emerging AI applications. In Proceed- ings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI'18, pages 561-577, USA. USENIX Association.
- Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019. When does label smoothing help? Advances in neural information processing systems, 32.
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730-27744.
- Richard Yuanzhe Pang, Stephen Roller, Kyunghyun Cho, He He, and Jason Weston. 2023. Leveraging Implicit Feedback from Deployment Data in Dia- logue. Preprint, arXiv:2307.14117. · DOI · Paper page
- Dominic Petrak, Nafise Moosavi, Ye Tian, Nikolai Rozanov, and Iryna Gurevych. 2023. Learning from free-text human feedback -collect new datasets or extend existing ones? In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 16259-16279, Singapore. Association for Computational Linguistics. · DOI
- Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. 2024. Recursive introspection: Teaching LLM agents how to self-improve. In ICML 2024 Workshop on Foundation Models in the Wild.
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your lan- guage model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728- 53741.
- Seymour Rosenberg and Bertram D. Cohen. 1964. Speakers' and Listeners' Processes in a Word- Communication Task. Science, 145(3637):1201- 1203. · DOI
- Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2023. Training Language Mod- els with Language Feedback at Scale. Preprint, arXiv:2303.16755. · arXiv
- Michael F Schober and Herbert H Clark. 1989. Under- standing by addressees and overhearers. Cognitive Psychology, 21(2):211-232. · DOI
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal Policy Optimization Algorithms. Preprint, arXiv:1707.06347. · DOI · Paper page
- Alane Suhr and Yoav Artzi. 2023. Continual learn- ing for instruction following from realtime feedback. In Thirty-seventh Conference on Neural Information Processing Systems.
- Theodore R. Sumers, Mark K. Ho, Robert D. Hawkins, Karthik Narasimhan, and Thomas L. Griffiths. 2021. Learning Rewards From Linguistic Feedback. Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 35(7):6002-6010. · DOI
- Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Ja- son E Weston. 2024. Self-rewarding language mod- els. In Forty-first International Conference on Ma- chine Learning.
- Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. 2024. Quiet-star: Language models can teach them- selves to think before speaking. arXiv preprint arXiv:2403.09629. · arXiv
- Authors
- Chen, Z., Gul, M. O., Chen, Y., Geng, G., Wu, A., & Artzi, Y.
- Affiliations
- Cornell University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 24580-24606
- ISSN
- 0736-587X (ISSN-L)
- arXiv
- 2410.13852
- OpenAlex
- W4412945125
- Cited by
- 0
- References
- 0
- FWCI
- 0.00
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1750499, 1818253, 2005572, 2311521)
- National Aeronautics and Space Administration
- University of Texas at Austin (1818253)
- Open Philanthropy Project
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- interaction-based learning, learning from interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
