CoChat
Sign Up
PreprintCited by 1Natural Language Processing TechniquesTopic ModelingMachine Learning and Algorithms

Let Me Teach You: Pedagogical Foundations of Feedback for Language Models

Open access
Borges, Beatriz, Tandon, Niket, Käser, Tanja, Bosselut, Antoine·2023·arXiv
DOI 10.48550/arxiv.2307.00279arXiv 2307.00279
Open this paper in CoChat — free accountOpen PDF

Abstract

Natural Language Feedback (NLF) is an increasingly popular mechanism for aligning Large Language Models (LLMs) to human preferences. Despite the diversity of the information it can convey, NLF methods are often hand-designed and arbitrary, with little systematic grounding. At the same time, research in learning sciences has long established several effective feedback models. In this opinion piece, we compile ideas from pedagogy to introduce FELT, a feedback framework for LLMs that outlines various characteristics of the feedback space, and a feedback content taxonomy based on these variables, providing a general mapping of the feedback space. In addition to streamlining NLF designs, FELT also brings out new, unexplored directions for research in NLF. We make our taxonomy available to the community, providing guides and examples for mapping our categorizations to future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.00279

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Concepts from learning sciences can be limited to fully explain user feedback from the real-world LLM-human setting, as only half of the participants (humans) can be characterized.

    “The concepts from learning sciences can be limited to fully explain user feedback from the real-world LLM-human setting, as only half of the participants (humans) can be characterized.”

    From User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal (2025) · cites this paper as Borges et al., 2023

    The Limitations section says the framework’s pedagogical grounding has not been fully validated for LLMs and that failures may reveal differences from human learning. This supports the possibility that learning-sciences concepts do not fully explain LLM feedback, but not the claim that only the human half of an LLM–human interaction can be characterized: the Task section explicitly characterizes the LLM as the learner. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

  • Feedback in Written Instruction

    Kulhavy, R. W. (1977) · Review of Educational Research · doi:10.3102/00346543047002211 · Cited by 571 · cited 1× · see citing works

    “Kulhavy and Stock (1989) Refer to a previous definition of feedback, whereby feedback is considered "any of the numerous procedures that are used to tell a learner if an instructional response is right or wrong" (Kulhavy, 1977).”
    Background

Flagged first, then by citations

Research works printed but not linked: 74

  1. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson El- hage, Zac Hatfield-Dodds, Danny Hernandez, Jack- son Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam Mc- Candlish, Chris Olah, and Jared Kaplan. 2021. A general language assistant as a laboratory for align- ment. arXiv preprint arXiv:2112.00861. · arXiv
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. · arXiv
  3. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christo- pher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott John- ston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. · arXiv
  4. Robert L. Bangert-Drowns, Chen-Lin C. Kulik, James A. Kulik, and MaryTeresa Morgan. 1991. The Instruc- tional Effect of Feedback in Test-like Events. In Review of Educational Research, volume 61, pages 213-238. Sage Publications, Inc. · DOI · Paper page
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901.
  6. Deborah L. Butler and Philip H. Winne. 1995. Feedback and Self-Regulated Learning: A Theoretical Synthe- sis. In Review of Educational Research, volume 65, pages 245-281. Sage Publications, Inc. · DOI · Paper page
  7. David Carless and David Boud. 2018. The develop- ment of student feedback literacy: enabling uptake of feedback. In Assessment & Evaluation in Higher Education, volume 43, pages 1315-1325. Routledge. · DOI · Paper page
  8. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arxiv:2301.07597. · arXiv
  9. Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 3667- 3684, Florence, Italy. Association for Computational Linguistics. · DOI · Paper page
  10. John Hattie and Helen Timperley. 2007. The Power of Feedback. In Review of Educational Research, volume 77, pages 81-112. · DOI · Paper page
  11. Jiwoo Hong, Noah Lee, and James Thorne. 2024. ORPO: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691. · arXiv
  12. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  13. Avraham N. Kluger and Angelo DeNisi. 1996. The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. In Psychological Bul- letin, volume 119, pages 254-284. · DOI · Paper page
  14. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Alexandrovich Glushkov, Arnav Varma Dan- tuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Julian Mattick. 2023. Ope- nAssistant Conversations -democratizing large lan- guage model alignment. In Thirty-seventh Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track.
  15. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. 2023. Pre- training language models with human preferences. In Proceedings of the 40th International Conference on Machine Learning. JMLR.
  16. Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. 2018. Can neural machine translation be improved with user feedback? In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 92-105, New Orleans -Louisiana. Association for Computational Linguistics. · DOI · Paper page
  17. Raymond W. Kulhavy. 1977. Feedback in Written In- struction. In Review of Educational Research, vol- ume 47, pages 211-232. Sage Publications, Inc. · DOI · Paper page
  18. Alina Leidinger, Robert van Rooij, and Ekaterina Shutova. 2023. The language of prompting: What linguistic properties make a prompt successful? In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pages 9210-9232, Singapore. Association for Computational Linguistics. · DOI
  19. Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2017. Di- alogue learning with human-in-the-loop. In Interna- tional Conference on Learning Representations.
  20. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri- son Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let's verify step by step. In The Twelfth Inter- national Conference on Learning Representations.
  21. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chan- dra Bhagavatula, and Yejin Choi. 2024. The unlock- ing spell on base LLMs: Rethinking alignment via in-context learning. In The Twelfth International Conference on Learning Representations.
  22. Anastasiya A. Lipnevich, David A. G. Berg, and Jef- frey K. Smith. 2016. Toward a Model of Student Response to Feedback. In Handbook of Human and Social Conditions in Assessment. Routledge.
  23. Anastasiya A. Lipnevich and Ernesto Panadero. 2021. A Review of Feedback Models and Theories: Descrip- tions, Definitions, and Conclusions. In Frontiers in Education, volume 6. · DOI · Paper page
  24. Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023. Chain of hindsight aligns language models with feed- back. volume abs/2302.02676. · DOI · Paper page
  25. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7052-7063, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics. · DOI · Paper page
  26. Ximing Lu, Faeze Brahman, Peter West, Jaehun Jung, Khyathi Chandu, Abhilasha Ravichander, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Lin, Skyler Hallinan, Lian- hui Qin, Xiang Ren, Sean Welleck, and Yejin Choi. 2023. Inference-time policy adapters (IPA): Tailoring extreme-scale LMs without fine-tuning. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6863-6883, Singapore. Association for Computational Linguis- tics. · DOI
  27. Aman Madaan, Niket Tandon, Peter Clark, and Yim- ing Yang. 2022. Memory-assisted prompt editing to improve GPT-3 after deployment. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2833-2861, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI · Paper page
  28. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh, and Peter Clark. 2023. Self-refine: Itera- tive refinement with self-feedback. In Thirty-seventh Conference on Neural Information Processing Sys- tems.
  29. Santosh Mathan and K. Koedinger. 2005. Fostering the intelligent novice: Learning from errors with metacognitive tutoring. In Educational Psychologist, volume 40, pages 257 -265.
  30. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. SimPO: Simple preference optimiza- tion with a reference-free reward. arXiv preprint arXiv:2405.14734. · arXiv
  31. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell- Gillingham, Geoffrey Irving, and Nat McAleese. 2022. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147. · arXiv
  32. Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. 2024. Fine-grained hallucina- tion detection and editing for language models. In First Conference on Language Modeling.
  33. Shikhar Murty, Christopher Manning, Scott Lundberg, and Marco Tulio Ribeiro. 2022. Fixing model bugs with natural language patches. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 11600-11613, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics. · DOI
  34. Keerthiram Murugesan, Sarathkrishna Swaminathan, Soham Dan, Subhajit Chaudhury, Chulaka Gu- nasekara, Maxwell Crouse, Diwakar Mahajan, Ibrahim Abdelaziz, Achille Fokoue, Pavan Kapa- nipathi, Salim Roukos, and Alexander Gray. 2023. MISMATCH: Fine-grained evaluation of machine- generated text with mismatch error types. In Find- ings of the Association for Computational Linguis- tics: ACL 2023, pages 4485-4503, Toronto, Canada. Association for Computational Linguistics. · DOI
  35. Susanne Narciss. 2008. Feedback Strategies for Inter- active Learning Tasks. In Handbook of research on educational communications and technology, pages 125-144. Routledge.
  36. Susanne Narciss and Katja Huth. 2004. How to de- sign informative tutoring feedback for multi-media learning. volume Instructional design for multimedia learning, pages 181-195.
  37. David J. Nicol and Debra Macfarlane-Dick. 2006. For- mative assessment and self-regulated learning: a model and seven principles of good feedback practice. In Studies in Higher Education, volume 31, pages 199-218. Routledge. · DOI · Paper page
  38. OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
  39. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2024. Training language models to follow instructions with human feedback. In Proceedings of the 36th Interna- tional Conference on Neural Information Processing Systems, Red Hook, NY, USA.
  40. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2022. The effects of reward misspecification: Map- ping and mitigating misaligned models. In Interna- tional Conference on Learning Representations.
  41. Ernesto Panadero and Anastasiya A. Lipnevich. 2022. A review of feedback models and typologies: To- wards an integrative model of feedback elements. In Educational Research Review, volume 35. · DOI · Paper page
  42. Richard Yuanzhe Pang, Stephen Roller, Kyunghyun Cho, He He, and Jason Weston. 2024. Leveraging implicit feedback from deployment data in dialogue. In Proceedings of the 18th Conference of the Euro- pean Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 60-75, St. Julian's, Malta. Association for Computational Linguistics.
  43. Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beat- riz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2024. REFINER: Reasoning feedback on intermediate representations. In Proceedings of the 18th Conference of the European Chapter of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1100-1126, St. Julian's, Malta. Association for Computational Linguistics.
  44. Jakub Prochazka, Martin Ovcari, and Michal Durinik. 2020. Sandwich feedback: The empirical evidence of its effectiveness. In Learning and Motivation, volume 71. · DOI
  45. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Sys- tems.
  46. Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2023. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion. In The Eleventh International Conference on Learning Representations.
  47. Arkalgud Ramaprasad. 1983. On the Definition of Feed- back. In Behavioral Science, volume 28, pages 4-13. · DOI · Paper page
  48. D. Royce Sadler. 1989. Formative assessment and the design of instructional systems. In Instructional Sci- ence, volume 18, pages 119-144. · DOI · Paper page
  49. Amir Saeidi, Shivanshu Verma, Aswin RRV, and Chitta Baral. 2024. Triple preference optimization: Achiev- ing better alignment with less data in a single step optimization. arXiv preprint arXiv:2405.16681. · arXiv
  50. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802. · arXiv
  51. Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2022. Training language models with language feed- back. arXiv preprint arXiv:2204.14146. · arXiv
  52. Timo Schick, Jane A. Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebas- tian Riedel. 2023. PEER: A collaborative language model. In The Eleventh International Conference on Learning Representations.
  53. Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. 2024. When life gives you lemons, make cherryade: Converting feedback from bad re- sponses into good labels. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3066-3082, Mexico City, Mexico. Association for Computational Linguistics. · DOI
  54. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. Re- flexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems.
  55. Ziang Song, Tianle Cai, Jason D. Lee, and Weijie J Su. 2023. Reward collapse in aligning large lan- guage models: A prompt-aware approach to prefer- ence rankings. In ICML 2023 Workshop The Many Facets of Preference-Based Learning.
  56. Moritz Stephan, Alexander Khazatsky, Eric Mitchell, Annie S Chen, Sheryl Hsu, Archit Sharma, and Chelsea Finn. 2024. RLVF: Learning from verbal feedback without overgeneralization. arXiv preprint arXiv:2402.10893. · arXiv
  57. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. In Proceedings of the 34th International Conference on Neural In- formation Processing Systems, NIPS '20, Red Hook, NY, USA.
  58. Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. 2022. Learning to repair: Repairing model out- put errors after deployment using a dynamic memory of feedback. In Findings of the Association for Com- putational Linguistics: NAACL 2022, pages 339-352, Seattle, United States. Association for Computational Linguistics. · DOI
  59. Yikun Wang, Rui Zheng, Liang Ding, Qi Zhang, Dahua Lin, and Dacheng Tao. 2024. Uncertainty aware learning for language model alignment. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 11087-11099, Bangkok, Thailand. As- sociation for Computational Linguistics. · DOI
  60. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484-13508, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  61. Yuqing Wang and Yun Zhao. 2024. Metacognitive prompting improves understanding in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pages 1914-1926, Mexico City, Mexico. Association for Computational Linguistics. · DOI
  62. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2024. Chain-of-thought prompt- ing elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA.
  63. Jason Weston. 2016. Dialog-based language learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, page 829-837, Red Hook, NY, USA.
  64. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Sti- ennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862. · arXiv
  65. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine- grained human feedback gives better rewards for lan- guage model training. In Thirty-seventh Conference on Neural Information Processing Systems.
  66. Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. 2023. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 13557-13572, Toronto, Canada. Association for Computational Linguistics. · DOI
  67. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems.
  68. Xunjian Yin, Baizhou Huang, and Xiaojun Wan. 2023. ALCUNA: Large language models meet new knowl- edge. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1397-1414, Singapore. Association for Com- putational Linguistics. · DOI
  69. Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gun- soo Han, Daniel Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo. 2024. TLCR: Token-level continuous reward for fine-grained reinforcement learning from human feedback. In Findings of the Association for Computational Linguistics ACL 2024, pages 14969- 14981, Bangkok, Thailand and virtual meeting. As- sociation for Computational Linguistics. · DOI
  70. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Ja- son E Weston. 2024. Self-rewarding language mod- els. In Forty-first International Conference on Ma- chine Learning.
  71. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. Textgrad: Automatic "differentiation" via text. arXiv preprint arXiv: 2406.07496. · arXiv
  72. Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024. Is in-context learn- ing sufficient for instruction following in llms? arXiv preprint arXiv:2405.19874. · arXiv
  73. Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. Context-faithful prompting for large language models. In Findings of the As- sociation for Computational Linguistics: EMNLP 2023, pages 14544-14556, Singapore. Association for Computational Linguistics. · DOI
  74. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2020. Fine-tuning lan- guage models from human preferences. arXiv preprint arXiv:1909.08593. · arXiv

Cited by

Sorted by citations

Authors
Borges, B., Tandon, N., Käser, T., & Bosselut, A.
Affiliations
  • École Polytechnique Fédérale de Lausanne (CH)
  • Allen Institute for Artificial Intelligence (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
12082-12104
DOI
10.48550/arxiv.2307.00279
arXiv
2307.00279
OpenAlex
W4404782792
Cited by
0
References
1
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
  • National Science Foundation
  • Allen Institute for Artificial Intelligence
  • École Polytechnique Fédérale de Lausanne
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (215390)
  • Innosuisse - Schweizerische Agentur für Innovationsförderung (PFFS-21-29)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
natural language feedback, large language models, preference alignment, human-in-the-loop learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review