CoChat
Sign Up
PreprintCited by 134Speech and dialogue systemsIntelligent Tutoring Systems and Adaptive LearningNatural Language Processing Techniques

Improving alignment of dialogue agents via targeted human judgements

Open access
Glaese, Amelia, McAleese, Nat, Trębacz, Maja, Aslanides, John, Firoiu, Vlad, Ewalds, Timo, Rauh, Maribeth, Weidinger, Laura +26 more·2022·arXiv
DOI 10.48550/arxiv.2209.14375arXiv 2209.14375
Open this paper in CoChat — free accountOpen PDF

Abstract

We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into natural language rules the agent should follow, and ask raters about each rule separately. We demonstrate that this breakdown enables us to collect more targeted human judgements of agent behaviour and allows for more efficient rule-conditional reward models. Second, our agent provides evidence from sources supporting factual claims when collecting preference judgements over model statements. For factual questions, evidence provided by Sparrow supports the sampled response 78% of the time. Sparrow is preferred more often than baselines while being more resilient to adversarial probing by humans, violating our rules only 8% of the time when probed. Finally, we conduct extensive analyses showing that though our model learns to follow our rules it can exhibit distributional biases.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2209.14375

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Fine-tuning a model from human feedback at scale using preference data is a potent technique that can shape responses to reflect user norms, filter toxic language, improve factual accuracy, and mitigate fabrication.

    “Fine-tuning a model from human feedback at scale, using preference data from paid raters or drawn from a large and willing user base, is an especially potent technique.It has the potential not only to shape a model's responses to better reflect user norms (for better or worse), but also to filter out toxic language, improve factual accuracy, and mitigate the tendency to fabricate information.”

    From Talking about Large Language Models (2024) · cites this paper as [14]

    Sections 2.3 (p. 7) and 2.7 (p. 11) describe collecting human response preferences and fine-tuning on preferred responses. The paper also reports reduced rule violations (p. 3), evidence of improved correctness (section 3.3, p. 18), and an aim to reduce hallucinations (section 2.4, p. 9). It supports the technique’s potential, but does not establish the alternative of drawing preference data from a large, willing user base; its evaluations use paid annotators (section 3.1, p. 14). (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • RLHF with LLMs has recently achieved significant success in sequence-to-sequence tasks such as summarization, providing reliable answers with citations, and creating Helpful, Harmless and Honest dialogue agents aligned with broad human values.

    “However, while the combination of RL and human feedback has been extensively studied [30,82], it is only recently that RLHF with LLMs has achieved significant success in sequence-to-sequence tasks such as Summarization [177,194,222], providing reliable answers with citations to queries [53,123], creating Helpful, Harmless and Honest dialogue agents aligned with broad human values [12,136].”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [53]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • RL finetuning decreases the diversity of samples produced by a model (mode collapse).

    “RL finetuning decreases the diversity of samples produced by a model (Khalifa et al., 2021;Perez et al., 2022a;Glaese et al., 2022;Go et al., 2023) (a phenomenon known as “mode collapse”).”

    From Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023) · cites this paper as Glaese et al., 2022

    Section 2.8 (p. 13) identifies collapse to a single high-reward generation as a risk of RL and says the authors use a KL penalty to prevent it. This supports the mode-collapse concern, but the paper does not report that RL fine-tuning decreased the diversity of samples produced by their model. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • In current LLM systems, calibration emerges from post-training: RLHF shapes tone, while constitutional AI translates high-level principles into conversational behaviours including when to refuse or redirect.

    “In current LLM systems, calibration emerges from post-training: RLHF shapes tone [15], while constitutional AI translates high-level principles into conversational behaviours including when to refuse or redirect [49, 50].”

    From Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaboration (2026) · cites this paper as [50]

    Section 2.1 (p. 4) says the authors turn high-level goals into detailed rules for dialogue behaviour. The rules include declining medical advice and suggesting a doctor instead (Appendix F.1, p. 49). This supports the refusal-and-redirection example, but the paper describes Sparrow’s rule-based approach, not constitutional AI in current LLM systems generally. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Training a single model with policy and value head is much more computationally efficient than training separate policy and value functions.

    “For RLHF, we train a single model with policy and value head, rather than separate policy and value functions.This is much more computationally efficient, and follows other recent work that still achieves impressive results (Glaese et al., 2022).”

    From Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023) · cites this paper as Glaese et al., 2022

    Appendix E (p. 45) says the authors share parameters between the policy and value functions to reduce memory usage, and describes a separate value-function MLP using the shared transformer representation. Section 2.8 (p. 13) likewise describes a shared trunk with separate heads. The copy does not establish that this is much more computationally efficient than training separate policy and value functions; it identifies a memory-saving benefit instead. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 93

  1. M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.
  2. A. Abdolmaleki, S. Huang, L. Hasenclever, M. Neunert, F. Song, M. Zambelli, M. Martins, N. Heess, R. Hadsell, and M. Riedmiller. A distributional view on multi-objective policy optimization. In International Conference on Machine Learning , pages 11-22. PMLR, 2020.
  3. L. Adolphs, K. Shuster, J. Urbanek, A. Szlam, and J. Weston. Reason first, then respond: Modular generation for knowledge-infused dialogue. arXiv preprint arXiv:2111.05204 , 2021. URL https: //arxiv.org/abs/2111.05204 . · arXiv
  4. A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, J. Kernion, K. Ndousse, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, and J. Kaplan. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021. URL https://arxiv.org/abs/2112. 00861 . · arXiv
  5. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022. URL https://arxiv.org/abs/2204.05862 . · arXiv
  6. M. Banko, B. MacKeen, and L. Ray. A unified taxonomy of harmful content. In Proceedings of the Fourth Workshop on Online Abuse and Harms , pages 125-137, Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.alw-1.16. URL https://aclanthology. org/2020.alw-1.16 . · DOI
  7. E. M. Bender, T. Gebru, A. McMillan-Major, and M. Mitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages 610-623, 2021.
  8. A. Birhane, V. Prabhakaran, M. Diaz, I. Gabriel, M. C. Elish, S. Mohamed, and W. S. Isaac. Power to the people? opportunities and challenges for participatory ai. Proceedings of the ACM conference on Equity and Access in Algorithms, Mechanisms, and Optimization , 2022.
  9. S. L. Blodgett, S. Barocas, H. Daumé III, and H. Wallach. Language (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050 , 2020. URL https://arxiv.org/abs/ 2005.14050 . · arXiv
  10. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. van den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy, J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan, J. W. Rae, E. Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning , pages 2206-2240. PMLR, 2022.
  11. J. Boyd-Graber, B. Satinoff, H. He, and H. Daumé III. Besting the Quiz Master: Crowdsourcing incremental classification games. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning , pages 1290-1301, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URL https://aclanthology.org/D12-1118 .
  12. L. Breitfeller, E. Ahn, D. Jurgens, and Y. Tsvetkov. Finding microaggressions in the wild: A case for locating elusive phenomena in social media posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 1664-1674, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1176. URL https: //aclanthology.org/D19-1176 . · DOI
  13. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf .
  14. P. Christiano, B. Shlegeris, and D. Amodei. Supervising strong learners by amplifying weak experts. arXiv preprint arXiv:1810.08575 , 2018. URL https://arxiv.org/abs/1810.08575 . · arXiv
  15. A. Creswell, M. Shanahan, and I. Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712 , 2022. URL https://arxiv. org/abs/2205.09712 . · arXiv
  16. B. Dang, M. J. Riedl, and M. Lease. But who protects the moderators? the case of crowdsourced image moderation. arXiv preprint arXiv:1804.10999 , 2018. URL https://arxiv.org/abs/ 1804.10999 . · arXiv
  17. E. Denton, A. Hanna, R. Amironesei, A. Smart, H. Nicole, and M. K. Scheuerman. Bringing the people back in: Contesting benchmark machine learning datasets. arXiv preprint arXiv:2007.07399 , 2020. URL https://arxiv.org/abs/2007.07399 . · arXiv
  18. E. Dinan, S. Humeau, B. Chintagunta, and J. Weston. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4537-4546, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1461. URL https://aclanthology. org/D19-1461 . · DOI · Paper page
  19. D. Dohan, W. Xu, A. Lewkowycz, J. Austin, D. Bieber, R. Gontijo Lopes, Y. Wu, H. Michalewski, R. A. Saurous, J. Sohl-dickstein, K. Murphy, and C. Sutton. Language model cascades. arXiv preprint arXiv:2207.10342 , 2022. URL https://arxiv.org/abs/2207.10342 . · arXiv
  20. N. Dziri, S. Milton, M. Yu, O. Zaiane, and S. Reddy. On the origin of hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5271-5285, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.387. URL https://aclanthology.org/2022.naacl-main. 387 . · DOI · Paper page
  21. R. El-Yaniv and Y. Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research , 11(5), 2010.
  22. O. Evans, O. Cotton-Barratt, L. Finnveden, A. Bales, A. Balwit, P. Wills, L. Righetti, and W. Saunders. Truthful ai: Developing and governing ai that does not lie. arXiv preprint arXiv:2110.06674 , 2021. · arXiv
  23. A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli. Eli5: Long form question answering. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 3558-3567, 2019.
  24. T. Gao, A. Fisch, and D. Chen. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723 , 2020. · arXiv
  25. Y. Geifman and R. El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems , 30, 2017.
  26. Y. Geifman and R. El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International Conference on Machine Learning , pages 2151-2159. PMLR, 2019.
  27. S. Gershman. How to never be wrong. Psychonomic Bulletin & Review , 26:13-28, 2019.
  28. M. Gor, K. Webster, and J. Boyd-Graber. Toward deconfounding the effect of entity demographics for question answering accuracy. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5457-5473, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.444. URL https://aclanthology.org/2021.emnlp-main.444 . · DOI
  29. M. Goulden, M. A. Mason, and K. Frasch. Keeping women in the science pipeline. The Annals of the American Academy of Political and Social Science , 638:141-162, 2011. ISSN 00027162, 15523349. URL http://www.jstor.org/stable/41328583 .
  30. H. P. Grice. Logic and conversation. In Speech acts , pages 41-58. Brill, 1975.
  31. S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. Deep learning with limited numerical precision. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 , ICML'15, page 1737-1746. JMLR.org, 2015.
  32. D. Hadfield-Menell, S. J. Russell, P. Abbeel, and A. Dragan. Cooperative inverse reinforcement learning. Advances in neural information processing systems , 29:3909-3917, 2016.
  33. A. Halfaker and R. S. Geiger. Ores: Lowering barriers with participatory machine learning in wikipedia. Proceedings of the ACM Conference on Human-Computer Interaction , 4(CSCW2):1-37, 2020.
  34. D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR) , 2021a.
  35. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR) , 2021b.
  36. G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural computation , 14(8):1771-1800, 2002.
  37. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training computeoptimal large language models. arXiv preprint arXiv:2203.15556 , 2022. URL https://arxiv. org/abs/2203.15556 . · arXiv
  38. A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 , 2019. URL https://arxiv.org/abs/1904.09751 . · arXiv
  39. G. Irving and A. Askell. AI safety needs social scientists. Distill , 2019. doi: 10.23915/distill.00014. URL https://doi.org/10.23915/distill.00014 . https://distill.pub/2019/safety-needssocial-scientists. · DOI
  40. G. Irving, P. Christiano, and D. Amodei. AI safety via debate. arXiv preprint arXiv:1805.00899 , 2018. URL https://arxiv.org/abs/1805.00899 . · arXiv
  41. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1147. URL https: //aclanthology.org/P17-1147 . · DOI · Paper page
  42. A. Kamath, R. Jia, and P. Liang. Selective question answering under domain shift. arXiv preprint arXiv:2006.09462 , 2020. URL https://arxiv.org/abs/2006.09462 . · arXiv
  43. D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014. URL https://arxiv.org/abs/1412.6980 . · arXiv
  44. N. Kotonya, A. Vlachos, M. Yazdani, L. Mathias, and M. Saeidi. Policy compliance detection via expression tree inference. arXiv preprint arXiv:2205.12259 , 2022. URL https://arxiv.org/ abs/2205.12259 . · arXiv
  45. Z. Kunda. The case for motivated reasoning. psychological bulletin , 108.3:480, 1990.
  46. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453-466, 2019.
  47. A. Lazaridou, E. Gribovskaya, W. Stokowiec, and N. Grigorev. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115 , 2022. URL https://arxiv.org/abs/2203.05115 . · arXiv
  48. M. K. Lee, D. Kusbit, A. Kahng, J. T. Kim, X. Yuan, A. Chan, D. See, R. Noothigattu, S. Lee, A. Psomas, and A. D. Procaccia. WeBuildAI: Participatory framework for algorithmic governance. Proceedings of the ACM Conference on Human-Computer Interaction , 3(CSCW):1-35, 2019. URL https://doi. org/10.1145/3359283 . · DOI
  49. J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871 , 2018. URL https: //arxiv.org/abs/1811.07871 . · arXiv
  50. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems , 33:9459-9474, 2020.
  51. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14858 , 2022. URL https: //arxiv.org/abs/2206.14858 . · arXiv
  52. S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.229. URL https://aclanthology.org/2022. acl-long.229 . · DOI · Paper page
  53. A. Liška, T. Kočiský, E. Gribovskaya, T. Terzi, E. Sezener, D. Agrawal, C. de Masson d'Autume, T. Scholtes, M. Zaheer, S. Young, E. Gilsenan-McMahon, S. Austin, P. Blunsom, and A. Lazaridou. StreamingQA: A benchmark for adaptation to new knowledge over time in question answering models. In International Conference on Machine Learning , pages 13604-13622. PMLR, 2022.
  54. K. McGuffie and A. Newhouse. The radicalization risks of GPT-3 and advanced neural language models. arXiv preprint arXiv:2009.06807 , 2020. URL https://arxiv.org/abs/2009.06807 . · arXiv
  55. J. Menick, M. Trebacz, V. Mikulik, J. Aslanides, F. Song, M. Chadwick, M. Glaese, S. Young, L. CampbellGillingham, G. Irving, and N. McAleese. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147 , 2022. URL https://arxiv.org/abs/2203.11147 . · arXiv
  56. V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Harley, T. P. Lillicrap, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 , ICML'16, page 1928-1937. JMLR.org, 2016.
  57. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 , 2021. URL https://arxiv.org/abs/2112.09332 . · arXiv
  58. [4]Open Ended Learning Team, A. Stooke, A. Mahajan, C. Barros, C. Deck, J. Bauer, J. Sygnowski, M. Trebacz, M. Jaderberg, M. Mathieu, N. McAleese, N. Bradley-Schmieg, N. Wong, N. Porcel, R. Raileanu, S. Hughes-Fitt, V. Dalibard, and W. M. Czarnecki. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808 , 2021. URL https://arxiv.org/ abs/2107.12808 . · arXiv
  59. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022. URL https://arxiv.org/abs/2203.02155 . · arXiv
  60. A. Parrish, A. Chen, N. Nangia, V. Padmakumar, J. Phang, J. Thompson, P. M. Htut, and S. R. Bowman. Bbq: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193 , 2021. URL https://arxiv.org/abs/2110.08193 . · arXiv
  61. A. Parrish, H. Trivedi, E. Perez, A. Chen, N. Nangia, J. Phang, and S. R. Bowman. Single-turn debate does not help humans answer hard reading-comprehension questions. arXiv preprint arXiv:2204.05212 , 2022. URL https://arxiv.org/abs/2204.05212 . · arXiv
  62. E. Perez, S. Karamcheti, R. Fergus, J. Weston, D. Kiela, and K. Cho. Finding generalizable evidence by learning to convince Q&A models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2402-2411, Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1244. URL https://aclanthology.org/ D19-1244 . · DOI
  63. E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.03286 , 2022. · arXiv
  64. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 , 2021. URL https://arxiv.org/abs/2112.11446 . · arXiv
  65. G. Ras, N. Xie, M. van Gerven, and D. Doran. Explainable deep learning: A field guide for the uninitiated. Journal of Artificial Intelligence Research , 73:329-397, 2022.
  66. M. Rauh, J. Mellor, J. Uesato, P.-S. Huang, J. Welbl, L. Weidinger, S. Dathathri, A. Glaese, G. Irving, I. Gabriel, W. Isaac, and L. A. Hendricks. Characteristics of harmful text: Towards rigorous benchmarking of language models. arXiv preprint arXiv:2206.08325 , 2022. URL https://arxiv. org/abs/2206.08325 . · arXiv
  67. R. Rudinger, J. Naradowsky, B. Leonard, and B. Van Durme. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301 , 2018. URL https://arxiv.org/abs/1804.09301 . · arXiv
  68. S. Russell. Human Compatible . Penguin, 2020.
  69. M. Saeidi, M. Yazdani, and A. Vlachos. Cross-policy compliance detection via question answering. arXiv preprint arXiv:2109.03731 , 2021. URL https://arxiv.org/abs/2109.03731 . · arXiv
  70. W. Saunders, C. Yeh, J. Wu, S. Bills, L. Ouyang, J. Ward, and J. Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802 , 2022. URL https://arxiv.org/ abs/2206.05802 . · arXiv
  71. N. Shazeer and M. Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In Proceedings of the 35th International Conference on Machine Learning , pages 4603-4611. PMLR, 2018.
  72. M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019. URL http://arxiv. org/abs/1909.08053 . cite arxiv:1909.08053. · arXiv
  73. K. Shuster, M. Komeili, L. Adolphs, S. Roller, A. Szlam, and J. Weston. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. arXiv preprint arXiv:2203.13224 , 2022a. URL https://arxiv.org/abs/2203.13224 . · arXiv
  74. K. Shuster, J. Xu, M. Komeili, D. Ju, E. M. Smith, S. Roller, M. Ung, M. Chen, K. Arora, J. Lane, M. Behrooz, W. Ngan, S. Poff, N. Goyal, A. Szlam, Y.-L. Boureau, M. Kambadur, and J. Weston. BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188 , 2022b. URL https://arxiv.org/abs/2208.0318 . · arXiv
  75. M. Sloane, E. Moss, O. Awomolo, and L. Forlano. Participation is not a design fix for machine learning. arXiv preprint arXiv:2007.02423 , 2020. URL https://arxiv.org/abs/2007.02423 . · arXiv
  76. H. F. Song, A. Abdolmaleki, J. T. Springenberg, A. Clark, H. Soyer, J. W. Rae, S. Noury, A. Ahuja, S. Liu, D. Tirumala, N. Heess, D. Belov, M. Riedmiller, and M. M. Botvinick. V-mpo: On-policy maximum a posteriori policy optimization for discrete and continuous control. arXiv preprint arXiv:1909.12238 , 2019. URL https://arxiv.org/abs/1909.12238 . · arXiv
  77. M. Steiger, T. J. Bharucha, S. Venkatagiri, M. J. Riedl, and M. Lease. The psychological well-being of content moderators: the emotional labor of commercial moderation and avenues for improving support. In Proceedings of the 2021 CHI conference on human factors in computing systems , pages 1-14, 2021.
  78. N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.
  79. R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction . The MIT Press, second edition, 2018. URL http://incompleteideas.net/book/the-book-2nd.html .
  80. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, V. Zhao, Y. Zhou, C.-C. Chang, I. Krivokon, W. Rusch, M. Pickett, P. Srinivasan, L. Man, K. Meier-Hellstern, M. Ringel Morris, T. Doshi, R. Delos Santos, T. Duke, J. Soraker, B. Zevenbergen, V. Prabhakaran, M. Diaz, B. Hutchinson, K. Olson, A. Molina, E. Hoffman-John, J. Lee, L. Aroyo, R. Rajakumar, A. Butryna, M. Lamm, V. Kuzmina, J. Fenton, A. Cohen, R. Bernstein, R. Kurzweil, B. Aguera-Arcas, C. Cui, M. Croak, E. Chi, and Q. Le. LaMDA: Language models for dialog applications. arXiv preprint arXiv:2201.08239 , 2022. URL https://arxiv.org/abs/2201.08239 . · arXiv
  81. N. Tomasev, K. R. McKee, J. Kay, and S. Mohamed. Fairness for unobserved characteristics: Insights from technological impacts on queer communities. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages 254-265, 2021.
  82. B. Vidgen, A. Harris, D. Nguyen, R. Tromble, S. Hale, and H. Margetts. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online , pages 80-93, Florence, Italy, Aug. 2019. Association for Computational Linguistics. doi: 10.18653/v1/ W19-3509. URL https://aclanthology.org/W19-3509 . · DOI
  83. O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wunsch, K. McKinney, O. Smith, T. Schaul, T. P. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nat. , 575(7782): 350-354, 2019.
  84. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021. URL https: //arxiv.org/abs/2109.01652 . · arXiv
  85. L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, Z. Kenton, S. Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane, J. Haas, L. Laura Rimell, L. A. Hendricks, W. Isaac, S. Legassick, G. Irving, and I. Gabriel. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 , 2021. URL https: //arxiv.org/abs/2112.04359 . · arXiv
  86. J. Welbl, A. Glaese, J. Uesato, S. Dathathri, J. Mellor, L. A. Hendricks, K. Anderson, P. Kohli, B. Coppin, and P.-S. Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 2447-2469, Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-emnlp.210. URL https://aclanthology.org/2021.findings-emnlp.210 . · DOI · Paper page
  87. J. Wu, L. Ouyang, D. M. Ziegler, N. Stiennon, R. Lowe, J. Leike, and P. Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862 , 2021. URL https: //arxiv.org/abs/2109.10862 . · arXiv
  88. A. Xu, E. Pathak, E. Wallace, S. Gururangan, M. Sap, and D. Klein. Detoxifying language models risks marginalizing minority voices. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2390-2397, Online, June 2021a. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main. 190. URL https://aclanthology.org/2021.naacl-main.190 . · DOI
  89. J. Xu, D. Ju, M. Li, Y.-L. Boureau, J. Weston, and E. Dinan. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2950-2968, 2021b.
  90. H. Zamani, J. R. Trippas, J. Dalton, and F. Radlinski. Conversational information seeking. arXiv preprint arXiv:2201.08808 , 2022. URL https://arxiv.org/abs/2201.08808 . · arXiv
  91. Z. Zhang and L. Luo. Hate speech detection: A solved problem? the challenging case of long tail on twitter. Semantic Web , 10(5):925-945, 2019.
  92. J. Zhao, T. Wang, M. Yatskar, V. Ordonez, and K.-W. Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876 , 2018. URL https://arxiv. org/abs/1804.06876 . · arXiv
  93. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019. URL https://arxiv.org/abs/1909.08593 . · arXiv

Cited by

Sorted by citations

Authors
Glaese, A., McAleese, N., Trębacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., Campbell-Gillingham, L., Uesato, J., Huang, P.-S., Comanescu, R., Yang, F., See, A., Dathathri, S., Greig, R., Chen, C., Fritz, D., Elias, J. S., Green, R., Mokrá, S., Fernando, N., Wu, B., Foley, R., Young, S., Gabriel, I., Isaac, W., Mellor, J., Hassabis, D., Kavukcuoglu, K., Hendricks, L. A., & Irving, G.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2209.14375
arXiv
2209.14375
OpenAlex
W4298181573
Cited by
134
References
0
Citations by year
2026: 4 · 2025: 21 · 2024: 44 · 2023: 65
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
dialogue agents, alignment, reinforcement learning from human feedback, helpfulness, harm reduction, distributional bias, sparrows

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review