CoChat
Sign Up
PreprintCited by 29Machine Learning and AlgorithmsInnovative Teaching and Learning MethodsArtificial Intelligence Applications

Pretraining Language Models with Human Preferences

Open access
Korbak, Tomasz, Shi, Kejian, Chen, Angelica, Bhalerao, Rasika, Buckley, Christopher L., Phang, Jason, Bowman, Samuel R., Perez, Ethan·2023·arXiv
DOI 10.48550/arxiv.2302.08582arXiv 2302.08582
Open this paper in CoChat — free accountOpen PDF

Abstract

Language models (LMs) are pretrained to imitate internet text, including content that would violate human preferences if generated by an LM: falsehoods, offensive comments, personally identifiable information, low-quality or buggy code, and more. Here, we explore alternative objectives for pretraining LMs in a way that also guides them to generate text aligned with human preferences. We benchmark five objectives for pretraining with human feedback across three tasks and study how they affect the trade-off between alignment and capabilities of pretrained LMs. We find a Pareto-optimal and simple approach among those we explored: conditional training, or learning distribution over tokens conditional on their human preference scores given by a reward model. Conditional training reduces the rate of undesirable content by up to an order of magnitude, both when generating without a prompt and with an adversarially-chosen prompt. Moreover, conditional training maintains the downstream task performance of standard LM pretraining, both before and after task-specific finetuning. Pretraining with human feedback results in much better preference satisfaction than standard LM pretraining followed by finetuning with feedback, i.e., learning and then unlearning undesirable behavior. Our results suggest that we should move beyond imitation learning when pretraining LMs and incorporate human preferences from the start of training.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2302.08582

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Despite screening, there may be low agreement among annotators themselves, or even between researchers and annotators.

    “However, despite screening, there may be low agreement among the annotators themselves, or even between researchers and annotators [85].”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [85]

  • Korbak et al. (2023) argue that it can be more effective to use human feedback during pretraining by using a reward model to filter, weight, or annotate pretraining data.

    “Korbak et al. (2023) argue that it can be more effective to use human feedback during pretraining by using a reward model to filter, weight, or annotate pretraining data.”

    From Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023) · cites this paper as Korbak et al., 2023

  • SELF-RAG uses reflection tokens to decide the need for retrieval and to self-evaluate generation quality.

    “Other works use general control tokens to guide LM generation (Lu et al., 2022; Korbak et al., 2023), while SELF-RAG uses reflection tokens to decide the need for retrieval and to self-evaluate generation quality.”

    From CareerX: A Retrieval-Augmented Generation Framework for Personalized AI-Driven Career Guidance (2023) · cites this paper as Korbak et al., 2023

Cites

Still being checked.

Cited by

Sorted by citations

Related

Authors
Korbak, T., Shi, K., Chen, A., Bhalerao, R., Buckley, C. L., Phang, J., Bowman, S. R., & Perez, E.
Affiliations
  • University of Sussex (GB)
  • New York University (US)
  • Northeastern University (US)
  • Anthropological Survey of India (IN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2302.08582
arXiv
2302.08582
OpenAlex
W4321392329
Cited by
23
References
0
Citations by year
2025: 1 · 2024: 13 · 2023: 9
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Faculty Digital Archive (New York University Florence) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2046556, 1922658)
  • Open Philanthropy Project
  • Leverhulme Trust
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
  • Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
language model pretraining, human preferences, preference alignment, reward models, human feedback, downstream task performance, fine-tuning, imitation learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review