The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling
Abstract
This paper concerns the problem of aligning samples from large language models to human preferences using best-of-$n$ sampling, where we draw $n$ samples, rank them, and return the best one. We consider two fundamental problems. First: what is the relationship between best-of-$n$ and approaches to alignment that train LLMs to output samples with a high expected reward (e.g., RLHF or DPO)? To answer this, we embed both the best-of-$n$ distribution and the sampling distributions learned by alignment procedures in a common class of tiltings of the base LLM distribution. We then show that, within this class, best-of-$n$ is essentially optimal in terms of the trade-off between win-rate against the base model vs KL distance from the base model. That is, best-of-$n$ is the best choice of alignment distribution if the goal is to maximize win rate. However, best-of-$n$ requires drawing $n$ samples for each inference, a substantial cost. To avoid this, the second problem we consider is how to fine-tune a LLM to mimic the best-of-$n$ sampling distribution. We derive BoNBoN Alignment to achieve this by exploiting the special structure of the best-of-$n$ distribution. Experiments show that BoNBoN alignment yields substantial improvements in producing a model that is preferred to the base policy while minimally affecting off-target aspects.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In more general tasks without verification tools, we can conduct techniques like majority vote, self-consistency, and best-of-n to choose the most promising one from candidate responses”
From Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial? (2025) · cites this paper as Gui et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., & Kim, Y. J. (2024) · arXiv · doi:10.48550/arxiv.2401.08417 · Cited by 123
RRHF: Rank Responses to Align Language Models with Human Feedback without tears
Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., & Huang, F. (2023) · arXiv · doi:10.48550/arxiv.2304.05302 · Cited by 41
KTO: Model Alignment as Prospect Theoretic Optimization
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., & Kiela, D. (2024) · arXiv · doi:10.48550/arxiv.2402.01306 · Cited by 29
Efficient RLHF: Reducing the Memory Usage of PPO
Santacroce, M., Lu, Y., Yu, H., Li, Y., & Shen, Y. (2023) · arXiv · doi:10.48550/arxiv.2309.00754 · Cited by 4
Cited by
Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?
Li, W., Lin, Y., Xia, M., & Jin, C. (2025) · arXiv · doi:10.48550/arxiv.2502.00674 · Cited by 62
BOND: Aligning LLMs with Best-of-N Distillation
Sessa, P. G., Dadashi, R., Hussenot, L., Ferret, J., Vieillard, N., Ramé, A., Shariari, B., Perrin, S., Friesen, A., Cideron, G., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.14622 · Cited by 1
From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs
Wang, H., Liang, W., Fu, Z., Zheng, N., Zhang, Y., Tong, Y., Zhu, T., Jiang, H., Li, C., Wu, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2509.23196
Metadata
- Authors
- Gui, L., Gârbacea, C., & Veitch, V.
- Venue
- Advances in Neural Information Processing Systems 37
- Published
- 2024
- Pages
- 2851-2885
- Cited by
- 4
- References
- 5
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.