CoChat
Sign Up
PreprintCited by 44Natural Language Processing TechniquesTopic ModelingSpeech and dialogue systems

Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation

Open access
Fernandes, Patrick, Madaan, Aman, Liu, Emmy, Farinhas, António, Martins, Pedro Henrique, Bertsch, Amanda, de Souza, José G. C., Zhou, Shuyan +3 more·2023·arXiv
DOI 10.48550/arxiv.2305.00955arXiv 2305.00955
Open this paper in CoChat — free accountOpen PDF

Abstract

Many recent advances in natural language generation have been fueled by training large language models on internet-scale data. However, this paradigm can lead to models that generate toxic, inaccurate, and unhelpful content, and automatic evaluation metrics often fail to identify these behaviors. As models become more capable, human feedback is an invaluable signal for evaluating and improving models. This survey aims to provide an overview of the recent research that has leveraged human feedback to improve natural language generation. First, we introduce an encompassing formalization of feedback, and identify and organize existing research into a taxonomy following this formalization. Next, we discuss how feedback can be described by its format and objective, and cover the two approaches proposed to use feedback (either for training or decoding): directly using the feedback or training feedback models. We also discuss existing datasets for human-feedback data collection, and concerns surrounding feedback collection. Finally, we provide an overview of the nascent field of AI feedback, which exploits large language models to make judgments based on a set of principles and minimize the need for human intervention.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.00955

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • While RLHF substantially enhances the performance of LLMs and aligns them with human preferences, it is not without limitations, including text hallucination, bias and toxicity, and generation of harmful text when probed.

    “While RLHF substantially enhances the performance of LLMs and aligns them with human preferences, it is not without its limitations.These include, but are not limited to, issues such as text hallucination [115], bias and toxicity [37,48,58], and the generation of harmful text when probed [142,189].”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [48]

    The cited source is among the citations attached to “bias and toxicity,” not those attached to hallucination or harmful text when probed. In §6.2.3 (page unavailable), the survey reports that tuning with human feedback can shift outputs toward particular political views and that annotators may disagree about what is toxic. This supports bias as a limitation of human-feedback methods, but does not establish that RLHF-trained models generate toxic text as the citing sentence implies. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Recent literature surveys have focussed on integrating human feedback specifically for LMs, but are limited to feedback without providing a comprehensive survey of complete RLHF pipelines and alternatives.

    “Recent literature surveys have also focussed on integrating (human) feedback specifically for LMs [47,188], but are limited to feedback without providing a comprehensive survey of complete RLHF pipelines and alternatives.”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [47]

    The Abstract and Introduction describe a survey centered on feedback for natural language generation, supporting the characterization of its focus. But §5.2.1 discusses stages of RLHF, and the Conclusion explicitly says the survey covers numerous alternatives to RLHF. The claim overstates the extent to which those topics are absent; the copy does not establish whether its coverage amounts to a comprehensive survey of complete pipelines. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Mitigating hallucinations can be achieved by constructing a reward model based on human feedback and using reinforcement learning algorithms that encourage the target model to acknowledge its uncertainty, especially when correctness cannot be guaranteed.

    “Another approach is constructing a reward model based on human feedback, and using reinforcement learning algorithms with a reward model that encourages the target model to acknowledge its uncertainty, especially in scenarios where correctness cannot be guaranteed [149, 150].”

    From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [150]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 111

  1. Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014a. Power to the people: The role of humans in interactive ma- chine learning. Ai Magazine, 35(4):105-120.
  2. Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014b. Power to the people: The role of humans in interactive ma- chine learning. AI Magazine, 35(4):105-120.
  3. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. CoRR, abs/1606.06565. · arXiv
  4. Chantal Amrhein and Rico Sennrich. 2022. Iden- tifying weaknesses in machine translation met- rics through minimum Bayes risk decoding: A case study for COMET. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1125-1141, Online only. Associ- ation for Computational Linguistics.
  5. Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. 2022. Director: Generator- Classifiers For Supervised Language Model- ing. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing.
  6. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova Das- Sarma, et al. 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. · arXiv
  7. Karl Johan Åström and Richard M Murray. 2021. Feedback systems: an introduction for scientists and engineers. Princeton university press.
  8. Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team per- formance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1-16.
  9. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA. Associa- tion for Computing Machinery.
  10. Pushpak Bhattacharyya, Rajen Chatterjee, Markus Freitag, Diptesh Kanojia, Matteo Negri, and Marco Turchi. 2022. Findings of the WMT 2022 shared task on automatic post-editing. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 109-117, Abu Dhabi, United Arab Emirates (Hybrid). As- sociation for Computational Linguistics.
  11. Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019. Better rewards yield better sum- maries: Learning to summarise without refer- ences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3110-3120, Hong Kong, China. Association for Computational Linguistics.
  12. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29.
  13. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Ni- ladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Don- ahue, Moussa Doumbouya, Esin Durmus, Ste- fano Ermon, John Etchemendy, Kawin Etha- yarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tat- sunori Hashimoto, Peter Henderson, John He- witt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karam- cheti, Geoff Keeling, Fereshte Khani, O. Khat- tab, Pang Wei Koh, Mark S. Krass, Ranjay Kr- ishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Mun- yikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel J. Orr, Isabel Pa- padimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghu- nathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Para- suram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Ya- sunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the opportunities and risks of foundation models. ArXiv, abs/2108.07258. · arXiv
  14. Nick Bostrom. 2014. Superintelligence: Paths, Dangers, Strategies, 1st edition. Oxford Uni- versity Press, Inc., USA.
  15. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing auto- matic metrics in natural language evalaution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643-653, Mel- bourne, Australia. Association for Computa- tional Linguistics.
  16. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large lan- guage models to self-debug. arXiv preprint arXiv:2304.05128. · arXiv
  17. Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith. 2021. All that's 'human' is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Preprint Papers), pages 7282-7296, Online. Association for Computational Linguistics.
  18. Gonçalo M. Correia and André F. T. Martins. 2019. A simple and effective approach to automatic post-editing with transfer learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3050-3056, Florence, Italy. Association for Computational Linguistics.
  19. Michael Denkowski, Chris Dyer, and Alon Lavie. 2014. Learning from post-editing: Online model adaptation for statistical machine trans- lation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 395-404, Gothenburg, Sweden. Association for Computa- tional Linguistics.
  20. Yi Ding, Jacob You, Tonja-Katrin Machulla, Jen- nifer Jacobs, Pradeep Sen, and Tobias Höllerer. 2022. Impact of annotator demographics on sentiment dataset labeling. Proc. ACM Hum.-Comput. Interact., 6(CSCW2).
  21. Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groen- eveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled cor- pus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286-1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  22. Ahmed Elgohary, Christopher Meek, Matthew Richardson, Adam Fourney, Gonzalo Ramos, and Ahmed Hassan Awadallah. 2021. Nl- edit: Correcting semantic parse errors through natural language interaction. arXiv preprint arXiv:2103.14540. · arXiv
  23. Patrick Fernandes, António Farinhas, Ricardo Rei, José De Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality- aware decoding for neural machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1396-1412, Seattle, United States. Association for Computational Linguis- tics.
  24. J.L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378-382.
  25. Markus Freitag, George Foster, David Grang- ier, Viresh Ratnakar, Qijun Tan, and Wolf- gang Macherey. 2021. Experts, errors, and context: A large-scale study of human evalu- ation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460-1474.
  26. Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022a. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics, 10:811-825.
  27. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi- kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022b. Results of WMT22 metrics shared task: Stop using BLEU -neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Associa- tion for Computational Linguistics.
  28. Yang Gao, Christian M. Meyer, and Iryna Gurevych. 2018. APRIL: Interactively learning to summarise by combining active preference learning and reinforcement learn- ing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4120-4130, Brussels, Bel- gium. Association for Computational Linguis- tics.
  29. Sebastian Gehrmann, Abhik Bhattacharjee, Abi- naya Mahendiran, Alex Wang, Alexandros Pa- pangelis, Aman Madaan, Angelina McMillan- Major, Anna Shvets, Ashish Upadhyay, Bing- sheng Yao, Bryan Wilie, Chandra Bhagavat- ula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir R. Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Se- doc, Juraj Juraska, Kaustubh D. Dhole, Khy- athi Raghavi Chandu, Leonardo F. R. Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Kale, Moussa Kamal Eddine, Nico Daheim, Nis- hant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qinqin Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahri- yar, Ronald Cardenas, Saad Mahamood, Sa- lomey Osei, Samuel Cahyawijaya, Sanja vSta- jner, Sébastien Montella, Shailza, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin P. Adewumi, Vikas Raunak, Vipul Ra- heja, Vitaly Nikolaev, Vivian Tsai, Yacine Jer- nite, Yi Xu, Yisi Sang, Yixin Liu, and Yu- fang Hou. 2022a. Gemv2: Multilingual nlg benchmarking in a single line of code. In Conference on Empirical Methods in Natural Language Processing.
  30. Sebastian Gehrmann, Elizabeth Clark, and Thibault Sellam. 2022b. Repairing the cracked foun- dation: A survey of obstacles in evaluation practices for generated text. arXiv preprint arXiv:2202.06935. · arXiv
  31. Bhavya Ghai, Q Vera Liao, Yunfeng Zhang, Rachel Bellamy, and Klaus Mueller. 2021. Explainable active learning (xal) toward ai explanations as interfaces for machine teachers. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3):1-28.
  32. Dan Gillick and Yang Liu. 2010. Non-expert evaluation of summarization systems is risky. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 148-151, Los Angeles. Association for Com- putational Linguistics.
  33. Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022. Improving align- ment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375. · arXiv
  34. Taisiya Glushkova, Chrysoula Zerva, Ricardo Rei, and André F. T. Martins. 2021. Uncertainty- aware machine translation evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2021. Association for Computational Linguistics.
  35. Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In CHI Conference on Human Factors in Computing Systems. ACM.
  36. Yvette Graham, Timothy Baldwin, Alistair Mof- fat, and Justin Zobel. 2013. Continuous mea- surement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33-41, Sofia, Bulgaria. Asso- ciation for Computational Linguistics.
  37. Braden Hancock, Antoine Bordes, Pierre- Emmanuel Mazare, and Jason Weston. 2019. Learning from Dialogue after Deployment: Feed Yourself, Chatbot! In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3667-3684.
  38. Dan Hendrycks, Collin Burns, Steven Basart, An- drew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020. Aligning ai with shared hu- man values. arXiv preprint arXiv:2008.02275. · arXiv
  39. Lars Hornuf and Daniel Vrankar. 2022. Hourly wages in crowdworking: A meta-analysis. Business & Information Systems Engineering, 64(5):553-573.
  40. Maurice Jakesch, Advait Bhat, Daniel Buschek, Lior Zalmanson, and Mor Naaman. 2023. Co- writing with opinionated language models af- fects users' views. ArXiv, abs/2302.00560. · arXiv
  41. Natasha Jaques, Asma Ghandeharioun, Judy Han- wen Shen, Craig Ferguson, Àgata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Pi- card. 2019. Way off-policy batch deep reinforce- ment learning of implicit human preferences in dialog. CoRR, abs/1907.00456. · arXiv
  42. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021a. Alignment of language agents. arXiv preprint arXiv:2103.14659. · arXiv
  43. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021b. Alignment of language agents. CoRR, abs/2103.14659. · arXiv
  44. W. Bradley Knox and Peter Stone. 2013. Learn- ing non-myopically from human-generated re- ward. In Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI '13, page 191-202, New York, NY, USA. Asso- ciation for Computing Machinery.
  45. Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. 2018. Can neural machine translation be improved with user feedback? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 92-105, New Orleans -Louisiana. As- sociation for Computational Linguistics.
  46. Shankar Kumar and William Byrne. 2002. Min- imum bayes-risk word alignments of bilin- gual texts. In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing -Volume 10, EMNLP '02, page 140-147, USA. Association for Computa- tional Linguistics.
  47. Florian Lehmann, Niklas Markert, Hai Dang, and Daniel Buschek. 2022. Suggestion lists vs. con- tinuous generation: Interaction design for writ- ing with generative models on mobile devices affect text length, wording and perceived au- thorship. Proceedings of Mensch und Computer 2022.
  48. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction. ArXiv, abs/1811.07871. · arXiv
  49. Jiwei Li, Alexander H Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2017. Dialogue Learning With Human-in-the- Loop. In International Conference on Learning Representations.
  50. Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie Cheung, and Siva Reddy. 2022. Using interac- tive feedback to improve the accuracy and ex- plainability of question answering systems post- deployment. In Findings of the Association for Computational Linguistics: ACL 2022. Associ- ation for Computational Linguistics.
  51. Bing Liu, Gokhan Tür, Dilek Hakkani-Tür, Pararth Shah, and Larry Heck. 2018. Dialogue learn- ing with human teaching and feedback in end- to-end trainable task-oriented dialogue systems. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages Preprint 2060-2069, New Orleans, Louisiana. Associa- tion for Computational Linguistics.
  52. Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023. Languages are Rewards: Hindsight Fine- tuning using Human Feedback. arXiv preprint arXiv:2302.02676. · arXiv
  53. Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2890-2903, Dublin, Ireland. As- sociation for Computational Linguistics.
  54. Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022. Memory-assisted prompt editing to improve GPT-3 after deploy- ment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2833-2861, Abu Dhabi, United Arab Emirates. Association for Compu- tational Linguistics.
  55. Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, and Eduard Hovy. 2021. Think about it! improving defeasible reasoning by first modeling the question sce- nario. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6291-6310, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  56. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevalu- ating the evaluation of automatic machine trans- lation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984-4997, Online. Association for Computational Linguis- tics.
  57. Nikhil Mehta and Dan Goldwasser. 2019. Improv- ing natural language interaction with robots us- ing advice. arXiv preprint arXiv:1905.04655. · arXiv
  58. Richard Ngo. 2022. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626. · arXiv
  59. Duy-Hung Nguyen, Nguyen Viet Dung Nghiem, Bao-Sinh Nguyen, Dung Tien Tien Le, Sha- hab Sabahi, Minh-Tien Nguyen, and Hung Le. 2022. Make the most of prior data: A solution for interactive text summarization with prefer- ence feedback. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1919-1930, Seattle, United States. Asso- ciation for Computational Linguistics.
  60. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opin- ions on natural language inference data? arXiv preprint arXiv:2010.03532. · arXiv
  61. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. A conversational paradigm for program synthesis. arXiv e-prints, pages arXiv-2203.
  62. Santanu Pal, Sudip Kumar Naskar, Mihaela Vela, and Josef van Genabith. 2016. A neural net- work based approach to automatic post-editing. Preprint In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 281-286, Berlin, Germany. Association for Computational Linguistics.
  63. Tatiana Passali, Alexios Gidiotis, Efstathios Chatzikyriakidis, and Grigorios Tsoumakas. 2021. Towards human-centered summa- rization: A case study on financial news. In Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing, pages 21-27, On- line. Association for Computational Linguistics.
  64. Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904. · arXiv
  65. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. CoRR, abs/1705.04304. · arXiv
  66. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Lidén, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with exter- nal knowledge and automated feedback. ArXiv, abs/2302.12813. · arXiv
  67. Maxime Peyrard, Teresa Botschen, and Iryna Gurevych. 2017. Learning to score system sum- maries for better content selection evaluation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 74-84, Copenhagen, Denmark. Association for Computational Lin- guistics.
  68. Maxime Peyrard and Iryna Gurevych. 2018. Ob- jective function learning to match human judge- ments for optimization-based summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 654-660, New Orleans, Louisiana. Association for Computational Linguistics.
  69. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020a. COMET: A neural frame- work for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages Preprint 2685-2702, Online. Association for Computa- tional Linguistics.
  70. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020b. Unbabel's participation in the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 911-920, Online. Association for Compu- tational Linguistics.
  71. Machel Reid and Graham Neubig. 2022. Learn- ing to model editing processes. arXiv preprint arXiv:2205.12374. · arXiv
  72. Christopher Riesbeck. 1981. Failure-driven re- minding for incremental learning. In IJCAI, pages 115-120.
  73. Arturo Rosenblueth, Norbert Wiener, and Julian Bigelow. 1943. Behavior, purpose and teleology. Philosophy of science, 10(1):18-24.
  74. Annabel Rothschild, Justin Booker, Christa Davoll, Jessica Hill, Venise Ivey, Carl DiSalvo, Ben Ry- dal Shapiro, and Betsy DiSalvo. 2022. Towards fair and pro-social employment of digital piece- workers for sourcing machine learning training data. In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pages 1-9.
  75. Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2022. A survey of evaluation metrics used for nlg systems. ACM Comput. Surv., 55(2).
  76. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators.
  77. Roger C Schank. 1983. Dynamic memory: A theory of reminding and learning in computers and people. cambridge university press.
  78. Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izac- ard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022. Peer: A collaborative language model. ArXiv, abs/2208.11663. · arXiv
  79. Natalie Schluter. 2017. The limits of auto- matic summarisation according to ROUGE. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 41-45, Valencia, Spain. Associa- tion for Computational Linguistics.
  80. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. · arXiv
  81. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Associa- tion for Computational Linguistics.
  82. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine trans- lation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683-1692, Berlin, Germany. Association for Computational Linguistics.
  83. Emily Sheng, Kai-Wei Chang, Premkumar Natara- jan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in lan- guage generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3407-3412, Hong Kong, China. Association for Computational Linguistics.
  84. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? improv- ing data quality and data mining using multi- ple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '08, page 614-622, New York, NY, USA. Asso- ciation for Computing Machinery.
  85. Weiyan Shi, Yu Li, Saurav Sahay, and Zhou Yu. 2021. Refine and imitate: Reducing rep- etition and inconsistency in persuasion dia- logues via reinforcement learning and human demonstration. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3478-3492, Punta Cana, Dominican Re- public. Association for Computational Linguis- tics.
  86. Boaz Shmueli, Jan Fell, Soumya Ray, and Lun-Wei Ku. 2021. Beyond fair pay: Ethical implica- tions of NLP crowdsourcing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3758-3769, Online. Association for Com- putational Linguistics.
  87. Michel Simard, Nicola Ueffing, Pierre Isabelle, and Roland Kuhn. 2007. Rule-based transla- tion with statistical phrase-based post-editing. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 203-206.
  88. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. 2008. Cheap and fast -but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254-263, Honolulu, Hawaii. Association for Computa- tional Linguistics.
  89. Teerachart Soratana, Yili Liu, and X Jessie Yang. 2022. Effects of payment rate and country's income level on attitude toward acrowdsourc- ing task. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol- ume 66, pages 2220-2224. SAGE Publications Sage CA: Los Angeles, CA.
  90. Miriah Steiger, Timir J Bharucha, Sukrit Venkata- giri, Martin J. Riedl, and Matthew Lease. 2021. The psychological well-being of content mod- erators: The emotional labor of commercial moderation and avenues for improving support. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21, New York, NY, USA. Association for Comput- ing Machinery.
  91. Simone Stumpf, Vidya Rajaram, Lida Li, Mar- garet Burnett, Thomas Dietterich, Erin Sulli- van, Russell Drummond, and Jonathan Her- locker. 2007. Toward harnessing user feedback for machine learning. In Proceedings of the 12th international conference on Intelligent user interfaces, pages 82-91.
  92. Zeerak Talat, Hagen Blix, Josef Valvoda, Maya In- dira Ganesh, Ryan Cotterell, and Adina Williams. 2022. On the machine learning of ethical judgments from natural language. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 769-779, Seattle, United States. Association for Computational Linguis- tics.
  93. Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. 2020. Teach- ing pre-trained models to systematically rea- son over implicit knowledge. arXiv preprint arXiv:2006.06609, 4(6). · arXiv
  94. Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. 2022. Learning to repair: Repair- ing model output errors after deployment using a dynamic memory of feedback. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 339-352, Seattle, United States. Association for Computational Linguis- tics.
  95. Andrea L. Thomaz and Cynthia Breazeal. 2008. Teachable robots: Understanding human teach- ing behavior to build more effective robot learn- ers. Artificial Intelligence, 172(6):716-737.
  96. Craig Thomson and Ehud Reiter. 2021. Genera- tion challenges: Results of the accuracy evalu- ation shared task. In Proceedings of the 14th International Conference on Natural Language Generation, pages 240-248, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  97. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenber- gen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. 2022b. Lamda: Language models for dialog applications. CoRR, abs/2201.08239. · arXiv
  98. Carlos Toxtli, Siddharth Suri, and Saiph Sav- age. 2021. Quantifying the invisible labor in crowd work. Proceedings of the ACM on human-computer interaction, 5(CSCW2):1-26.
  99. Alexander Matt Turner, Aseem Saxena, and Prasad Tadepalli. 2022. Formalizing the problem of side effect regularization. In NeurIPS ML Safety Workshop.
  100. Sean Welleck, Ximing Lu, Peter West, Faeze Brah- man, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053. · arXiv
  101. Jason Weston, Sumit Chopra, and Antoine Bor- des. 2014. Memory networks. arXiv preprint arXiv:1410.3916. · arXiv
  102. Jason E Weston. 2016. Dialog-based language learning. Advances in Neural Information Processing Systems.
  103. Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connection- ist reinforcement learning. Mach. Learn., 8(3-4):229-256.
  104. Bin Wu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2018. Query suggestion with feed- back memory network. In Proceedings of the 2018 World Wide Web Conference, pages 1563- 1571.
  105. Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein. 2022. Re3: Generating longer sto- ries with recursive reprompting and revision. In Conference on Empirical Methods in Natural Language Processing.
  106. Michihiro Yasunaga and Percy Liang. 2020. Graph- based, self-supervised program repair from di- agnostic feedback. 37th Int. Conf. Mach. Learn. ICML 2020, PartF168147-14:10730-10739.
  107. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language mod- els with human feedback without tears. arXiv preprint arXiv:2304.05302. · arXiv
  108. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. · arXiv
  109. Shuyan Zhou, Uri Alon, Sumit Agarwal, and Gra- ham Neubig. 2023b. Codebertscore: Evaluating code generation with pretrained models of code. arXiv preprint arXiv:2302.05527. · arXiv
  110. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human pref- erences. CoRR, abs/1909.08593. · arXiv
  111. Markus Zopf. 2018. Estimating summary qual- ity with pairwise preferences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1687-1696, New Orleans, Louisiana. Association for Com- putational Linguistics.

Cited by

Sorted by citations

Authors
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., & Martins, A. F. T.
Affiliations
  • Instituto de Telecomunicações (PT)
  • Carnegie Mellon University (US)
Corresponding author
Fernandes, Patrick
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
11
Pages
1643-1668
ISSN
2307-387X (ISSN-L)
DOI
10.48550/arxiv.2305.00955
arXiv
2305.00955
OpenAlex
W4390035079
Cited by
44
References
0
Citations by year
2026: 7 · 2025: 13 · 2024: 17 · 2023: 7
FWCI
4.14
Type
journal article
Language
en
Open access
Licence
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
  • European Commission (101070631)
  • HORIZON EUROPE Framework Programme
  • Defence Science and Technology Group
  • HORIZON EUROPE European Research Council (101088763)
  • NextGenerationEU
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
human feedback, natural language generation, feedback-based training, feedback model, AI-generated feedback, large language models, feedback collection

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review