CoChat
Sign Up
PreprintCited by 2Natural Language Processing TechniquesTopic ModelingAdversarial Robustness in Machine Learning

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Reward Gaming in Conditional Text Generation

Open access
Pang, Richard Yuanzhe, Padmakumar, Vishakh, Sellam, Thibault, Parikh, Ankur P., He, He·2022·arXiv
DOI 10.48550/arxiv.2211.08714arXiv 2211.08714
Open this paper in CoChat — free accountOpen PDF

Abstract

To align conditional text generation model outputs with desired behaviors, there has been an increasing focus on training the model using reinforcement learning (RL) with reward functions learned from human annotations. Under this framework, we identify three common cases where high rewards are incorrectly assigned to undesirable patterns: noise-induced spurious correlation, naturally occurring spurious correlation, and covariate shift. We show that even though learned metrics achieve high performance on the distribution of the data used to train the reward function, the undesirable patterns may be amplified during RL training of the text generation model. While there has been discussion about reward gaming in the RL or safety community, in this discussion piece, we would like to highlight reward gaming in the natural language generation (NLG) community using concrete conditional text generation examples and discuss potential fixes and areas for future work.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2211.08714

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • RLHF-PPO has shown promising results but suffers from sensitivity to hyperparameters, the need for training additional models, and potential misalignment of the RM.

    “While RLHF-PPO has shown promising results, it suffers from sensitivity to hyperparameters, the need for training additional models, and potential misalignment of the RM [138,149,174,197,220].”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [138]

    Section 3 (Background; page unavailable) says the KL coefficient is hard to tune, and the Introduction and Conclusion describe learned rewards assigning high scores to undesirable generations. The paper says PPO is widely used and can prevent reward collapse, but its own experiments use REINFORCE. It does not establish that RLHF-PPO requires training additional models or report promising RLHF-PPO results. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 74

  1. Leonard Adolphs, Tianyu Gao, Jing Xu, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. 2022. The cringe loss: Learning what language not to model. arXiv preprint arXiv:2211.05826. · arXiv
  2. Farhad Akhbardeh, Arkady Arkhangorodsky, Mag- dalena Biesialska, Ondřej Bojar, Rajen Chatter- jee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Fe- dermann, Markus Freitag, Yvette Graham, Ro- man Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Au- guste Tapo, Marco Turchi, Valentin Vydrin, and Mar- cos Zampieri. 2021. Findings of the 2021 conference on machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, pages 1-88, Online. Association for Computational Linguis- tics.
  3. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. · arXiv
  4. Chantal Amrhein and Rico Sennrich. 2022. Identifying weaknesses in machine translation metrics through minimum Bayes risk decoding: A case study for COMET. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1125-1141, Online only. Association for Computational Linguistics.
  5. Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Cheung. 2022. Why exposure bias matters: An imitation learning perspective of error accumu- lation in language generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 700-710, Dublin, Ireland. Association for Computational Linguistics. · DOI
  6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. · arXiv
  7. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073. · arXiv
  8. Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mor- datch. 2020. Emergent tool use from multi-agent autocurricula. In International Conference on Learn- ing Representations.
  9. Loïc Barrault, Magdalena Biesialska, Ondřej Bo- jar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Had- dow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshi- aki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 conference on machine translation (WMT20). In Proceedings of the Fifth Conference on Machine Translation, pages 1-55, Online. Association for Computational Linguis- tics.
  10. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Ad- vances in Neural Information Processing Systems, pages 1171-1179.
  11. Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In Pro- ceedings of the Second Conference on Machine Trans- lation, pages 489-513, Copenhagen, Denmark. Asso- ciation for Computational Linguistics. · DOI
  12. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 metrics shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Pa- pers, pages 199-231, Berlin, Germany. Association for Computational Linguistics. · DOI
  13. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2014. Report on the 11th IWSLT evaluation campaign. In Pro- ceedings of the International Workshop on Spoken Language Translation, volume 57, Hanoi, Vietnam.
  14. Ting-Rui Chiang and Yun-Nung Chen. 2021. Relating neural text degeneration to exposure bias. In Pro- ceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 228-239, Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  15. Tom Everitt, Marcus Hutter, Ramana Kumar, and Victo- ria Krakovna. 2021. Reward tampering problems and solutions in reinforcement learning: A causal influ- ence diagram perspective. Synthese, 198(27):6435- 6467.
  16. Xiang Fan, Yiwei Lyu, Paul Pu Liang, Ruslan Salakhut- dinov, and Louis-Philippe Morency. 2022. Nano: Nested human-in-the-loop reward learning for few- shot language model control. arXiv preprint arXiv:2211.05750. · arXiv
  17. Markus Freitag, George Foster, David Grangier, and Colin Cherry. 2020. Human-paraphrased references improve neural machine translation. In Proceed- ings of the Fifth Conference on Machine Translation, pages 1183-1192, Online. Association for Computa- tional Linguistics.
  18. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transac- tions of the Association for Computational Linguis- tics, 9:1460-1474. · DOI · Paper page
  19. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceed- ings of the Sixth Conference on Machine Translation, pages 733-774, Online. Association for Computa- tional Linguistics.
  20. Leo Gao, John Schulman, and Jacob Hilton. 2022. Scal- ing laws for reward model overoptimization. arXiv preprint arXiv:2210.10760. · arXiv
  21. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in Neural Information Processing Systems, 27.
  22. Chulaka Gunasekara, Guy Feigenblat, Benjamin Sz- najder, Ranit Aharonov, and Sachindra Joshi. 2021. Using question answering rewards to improve ab- stractive summarization. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2021, pages 518-526, Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  23. Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. 2018. Long text generation via adver- sarial training with leaked information. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). · DOI
  24. Ferenc Huszár. 2015. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? arXiv preprint arXiv:1511.05101. · arXiv
  25. Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. 2018. Reward learn- ing from human preferences and demonstrations in atari. Advances in Neural Information Processing Systems, 31.
  26. Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2019. Way off-policy batch deep reinforcement learning of im- plicit human preferences in dialog. arXiv preprint arXiv:1907.00456. · arXiv
  27. Zachary Kenton, Tom Everitt, Laura Weidinger, Ia- son Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of language agents. arXiv preprint arXiv:2103.14659. · arXiv
  28. Muhammad Khalifa, Hady Elsahar, and Marc Dymet- man. 2021. A distributional approach to controlled text generation. In International Conference on Learning Representations.
  29. Łukasz Kidziński, Sharada Prasanna Mohanty, Carmichael F Ong, Zhewei Huang, Shuchang Zhou, Anton Pechenko, Adam Stelmaszczyk, Piotr Jarosik, Mikhail Pavlov, Sergey Kolesnikov, et al. 2018. Learning to run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments. In The NIPS'17 Competition: Building Intelligent Systems, pages 121-153. Springer.
  30. Samuel Kiegeland and Julia Kreutzer. 2021. Revisiting the weaknesses of reinforcement learning for neu- ral machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1673-1681, Online. Association for Computational Linguistics. · DOI
  31. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. · arXiv
  32. Jack Koch, Lauro Langosco, Jacob Pfau, James Le, and Lee Sharkey. 2022. Objective robustness in deep reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning, Pro- ceedings of Machine Learning Research. PMLR.
  33. Ryosuke Kohita, Akifumi Wachi, Yang Zhao, and Ryuki Tachibana. 2020. Q-learning with language model for edit-based unsupervised summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 470-484, Online. Association for Computa- tional Linguistics. · DOI
  34. Tomasz Korbak, Ethan Perez, and Christopher L Buckley. 2022. RL with KL penalties is bet- ter viewed as bayesian inference. arXiv preprint arXiv:2205.11275. · arXiv
  35. Ilia Kulikov, Maksim Eremeev, and Kyunghyun Cho. 2022. Characterizing and addressing the issue of oversmoothing in neural autoregressive sequence modeling. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1115-1124, Online only. Association for Computational Linguistics.
  36. Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. 2020. Conservative Q-learning for offline reinforcement learning. Advances in Neural Informa- tion Processing Systems, 33:1179-1191.
  37. Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michał Zaj ąc, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. 2020. Google research foot- ball: A novel reinforcement learning environment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34(4), pages 4501-4510.
  38. Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bent- ley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al. 2020. The surprising creativity of dig- ital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities. Artificial life, 26(2):274-306.
  39. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  40. Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to- end learning of negotiation dialogues. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443-2453, Copenhagen, Denmark. Association for Computa- tional Linguistics. · DOI · Paper page
  41. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv:1907.11692. · arXiv
  42. Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good perfor- mance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671-688, Belgium, Brussels. Association for Com- putational Linguistics. · DOI
  43. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT sys- tems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90, Flo- rence, Italy. Association for Computational Linguis- tics. · DOI
  44. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Con- ference on Machine Translation, pages 688-725, On- line. Association for Computational Linguistics.
  45. Yuki Nakatani, Tomoyuki Kajiwara, and Takashi Ni- nomiya. 2022. Comparing BERT-based reward func- tions for deep reinforcement learning in machine translation. In Proceedings of the 9th Workshop on Asian Translation, pages 37-43, Gyeongju, Republic of Korea. International Conference on Computational Linguistics.
  46. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI
  47. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2022. The effects of reward misspecification: Map- ping and mitigating misaligned models. In Interna- tional Conference on Learning Representations.
  48. Richard Yuanzhe Pang and He He. 2021. Text gener- ation by learning from demonstrations. In Interna- tional Conference on Learning Representations.
  49. Richard Yuanzhe Pang, He He, and Kyunghyun Cho. 2022. Amortized noisy channel neural machine trans- lation. In Proceedings of the 15th International Con- ference on Natural Language Generation, pages 131- 143, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics.
  50. Richard Yuanzhe Pang, Adam Lelkes, Vinh Tran, and Cong Yu. 2021. AgreeSum: Agreement-oriented multi-document summarization. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 3377-3391, Online. Association for Computational Linguistics. · DOI
  51. Ramakanth Pasunuru and Mohit Bansal. 2018. Multi- reward reinforced summarization with saliency and entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 2 (Short Papers), pages 646-653, New Orleans, Louisiana. Association for Computa- tional Linguistics. · DOI
  52. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kada- vath, et al. 2022. Discovering language model behav- iors with model-written evaluations. arXiv preprint arXiv:2212.09251. · arXiv
  53. Pouya Pezeshkpour, Sarthak Jain, Sameer Singh, and Byron Wallace. 2022. Combining feature and in- stance attribution to detect artifacts. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1934-1946, Dublin, Ireland. Association for Computational Linguistics. · DOI
  54. Barbara Plank. 2016. What to do about non-standard (or non-canonical) language in NLP. arXiv preprint arXiv:1608.07836. · arXiv
  55. Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2023. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion. In International Conference on Learning Rep- resentations.
  56. Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level train- ing with recurrent neural networks. In International Conference on Learning Representations.
  57. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548. · arXiv
  58. Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1678, Florence, Italy. Asso- ciation for Computational Linguistics. · DOI · Paper page
  59. Florian Schmidt. 2019. Generalization in generation: A closer look at exposure bias. In Proceedings of the 3rd Workshop on Neural Generation and Trans- lation, pages 157-167, Hong Kong. Association for Computational Linguistics. · DOI
  60. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint arXiv:1707.06347. · arXiv
  61. Thibault Sellam, Amy Pu, Hyung Won Chung, Sebas- tian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur Parikh. 2020. Learning to evaluate translation beyond English: BLEURT submissions to the WMT metrics 2020 shared task. In Proceed- ings of the Fifth Conference on Machine Translation, pages 921-927, Online. Association for Computa- tional Linguistics.
  62. Joar Skalse, Nikolaus HR Howe, Dmitrii Krashenin- nikov, and David Krueger. 2022. Defining and characterizing reward hacking. arXiv preprint arXiv:2209.13085. · arXiv
  63. Felix Stahlberg and Bill Byrne. 2019. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3356- 3362, Hong Kong, China. Association for Computa- tional Linguistics. · DOI · Paper page
  64. Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015. Results of the WMT15 metrics shared task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256-273, Lisbon, Portugal. Association for Computational Lin- guistics. · DOI · Paper page
  65. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008- 3021.
  66. Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approxima- tion. Advances in Neural Information Processing Systems, 12.
  67. Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. 2019. Is deep reinforcement learning re- ally superhuman on Atari? leveling the playing field. arXiv preprint arXiv:1908.04683. · arXiv
  68. Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In Pro- ceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 183-196, Online. Association for Computa- tional Linguistics. · DOI · Paper page
  69. Quan Vuong, Aviral Kumar, Sergey Levine, and Yevgen Chebotar. 2022. Dual generator offline reinforcement learning. arXiv preprint arXiv:2211.01471. · arXiv
  70. Sean Welleck, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho. 2020a. Consistency of a recurrent language model with respect to incomplete decoding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5553-5568, Online. Association for Computational Linguistics. · DOI
  71. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. 2020b. Neural text generation with unlikelihood training. In International Conference on Learning Representa- tions.
  72. Sarah Wiegreffe and Ana Marasovic. 2021. Teach me to explain: A review of datasets for explainable natural language processing. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
  73. Ronald J Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-256.
  74. Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Sti- ennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Pang, R. Y., Padmakumar, V., Sellam, T., Parikh, A. P., & He, H.
Affiliations
  • New York University (US)
  • Google DeepMind (United Kingdom) (GB)
Corresponding author
Pang, Richard Yuanzhe
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
4746-4763
ISSN
0736-587X (ISSN-L)
DOI
10.48550/arxiv.2211.08714
arXiv
2211.08714
OpenAlex
W4385569890
Cited by
2
References
1
Citations by year
2026: 1 · 2025: 1
FWCI
0.35
Type
conference paper
Language
en
Open access
Licence
Locations
  • Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Samsung Advanced Institute of Technology
  • Samsung
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
conditional text generation, reinforcement learning, reward function, spurious correlations, covariate shift, natural language generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review