The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.
Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging
Abstract
While Reinforcement Learning from Human Feedback (RLHF) aligns Large Language Models (LLMs) with general, aggregate human preferences, it is suboptimal for learning diverse, individual perspectives. In this work, we study Reinforcement Learning from Personalized Human Feedback (RLPHF) problem, wherein LLMs are aligned to multiple (sometimes conflicting) preferences by modeling alignment as a Multi-Objective Reinforcement Learning (MORL) problem. Compared to strong single-objective baselines, we show that we can achieve personalized alignment by decomposing preferences into multiple dimensions. These dimensions are defined based on personalizations that are declared as desirable by the user. In this work, we show that they can be efficiently trained independently in a distributed manner and combined effectively post-hoc through parameter merging. The code is available at https://github.com/joeljang/RLPHF.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Personalized Soups [Jang et al., 2023] and Reward Soups [Rame et al., 2024] address the challenge of RL from personalized human feedback by first training multiple policy models with distinct preferences independently and then merging their parameters post-hoc during inference.”
From A survey of personalized large language models: progress and future directions (2025) · cites this paper as Jang et al., 2023
In the citing paper
“Both methods allow for dynamic weighting of the networks based on user preferences, enhancing model alignment and reducing reward misspecification.”
From A survey of personalized large language models: progress and future directions (2025) · cites this paper as Jang et al., 2023
In the citing paper
“Another strategy for MORL is to consider ad-hoc combinations of multiple trained policy LLMs during the decoding phase to achieve personalization.”
From A survey of personalized large language models: progress and future directions (2025) · cites this paper as Jang et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · arXiv · doi:10.48550/arxiv.1801.07243 · Cited by 165
Cited by
From Matching to Generation: A Survey on Generative Information Retrieval
Li, X., Jin, J., Zhou, Y., Zhang, Y., Zhang, P., Zhu, Y., & Dou, Z. (2025) · ACM Transactions on Information Systems · doi:10.1145/3722552 · Cited by 172
A survey of personalized large language models: progress and future directions
Liu, J., Qiu, Z., Li, Z., Dai, Q., Yu, W., Zhu, J., Hu, M., Yang, M., Chua, T.-S., & King, I. (2025) · arXiv · doi:10.48550/arxiv.2502.11528 · Cited by 146
Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries
Malaviya, C., Chang, J. C., Roth, D., Iyyer, M., Yatskar, M., & Lo, K. (2024) · arXiv · doi:10.48550/arxiv.2411.07237 · Cited by 4
On the Way to LLM Personalization: Learning to Remember User Conversations
Magister, L. C., Metcalf, K., Zhang, Y., & ter Hoeve, M. (2024) · arXiv · doi:10.48550/arxiv.2411.13405 · Cited by 6
“RLHF (Stiennon et al., 2020; Jang et al., 2023; Cheng et al., 2023; Park et al., 2024; Poddar et al., 2024; Li et al., 2024b) and low-rank PEFT-based meth-ods (Tan et al., 2024; Zhuang et al., 2024) alleviate these issues by directly encoding user information in model parameters, however, they are…”
Metadata
- Authors
- Jang, J., Kim, S., Lin, B. Y., Wang, Y., Hessel, J., Zettlemoyer, L., Hajishirzi, H., Choi, Y., & Ammanabrolu, P.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- arXiv
- 2310.11564
- References
- 3
- Type
- Preprint
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. https://app.cochat.ai/papers/10.48550/arxiv.2310.11564
- The paper
- Jang, J., Kim, S., Lin, B. Y., Wang, Y., Hessel, J., Zettlemoyer, L., Hajishirzi, H., Choi, Y., & Ammanabrolu, P. (2023). Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging. arXiv. https://doi.org/10.48550/arxiv.2310.11564
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
