The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF
Open accessAbstract
Model alignment with human preferences is an essential step in making Large Language Models (LLMs) helpful and consistent with human values.It typically consists of supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) stages.However, RLHF faces inherent limitations stemming from a complex training setup and its tendency to align the model with implicit values that end users cannot control at run-time.Moreover, reward models in RLHF stage commonly rely on single-dimensional feedback as opposed to explicit, multifaceted signals that indicate attributes such as helpfulness, humor, and toxicity.To address these limitations, we propose STEERLM, a supervised finetuning method that empowers end-users to control responses during inference.STEERLM conditions responses to conform to an explicitly defined multi-dimensional set of attributes, thereby empowering a steerable AI capable of generating helpful and high-quality responses while maintaining customizability.Experiments show that STEERLM trained on open source datasets generates responses that are preferred by human and automatic evaluators to many state-of-the-art baselines trained with RLHF while being much easier to train.Try STEERLM at https://huggingface.co/ nvidia/SteerLM-llama2-13B
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Overall, alignment is an active area of research, with approaches ranging from data-efficient alignment [55] to alternatives to RLHF [25] to aligning cross-modal representations [84].”
From Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey) (2024) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058
Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset
Rashkin, H., Smith, E. M., Li, M., & Boureau, Y.-L. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1534 · Cited by 1,020
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init AttentionPreprint
Zhang, R., Han, J., Liu, C., Gao, P., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., & Qiao, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.16199 · Cited by 957
Decision Transformer: Reinforcement Learning via Sequence ModelingPreprint
Chen, L.-L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., & Mordatch, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01345 · Cited by 849
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612
RAFT: Reward rAnked FineTuning for Generative Foundation Model AlignmentPreprint
Dong, H., Xiong, W. H., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., & Zhang, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.06767 · Cited by 537
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293
LIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.11206 · Cited by 275
WebGPT: Browser-assisted question-answering with human feedback
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09332 · Cited by 272
Human-level play in the game of Diplomacy by combining language models with strategic reasoning
Meta Fundamental AI Research Diplomacy Team (FAIR)†, Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., et al. (2022) · Science · doi:10.1126/science.ade9097 · Cited by 241
OpenAssistant Conversations - Democratizing Large Language Model Alignment
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07327 · Cited by 218
Peng, B., Li, C., He, P., Galley, M., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2304.03277 · Cited by 183
NeMo: a toolkit for building AI applications using Neural ModulesPreprint
Kuchaiev, O., Li, J., Nguyen, H., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09577 · Cited by 173
Self-citationRed Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130
WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., & Jiang, D. (2023) · arXiv · doi:10.48550/arxiv.2306.08568 · Cited by 120
Offline RL for Natural Language Generation with Implicit Language Q LearningPreprint
Snell, C., Kostrikov, I., Su, Y., Yang, M., & Levine, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.11871 · Cited by 104
ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models
Jentzsch, S. F., & Kersting, K. (2023) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/2023.wassa-1.29 · Cited by 86
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
Dubois, Y., Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., & Hashimoto, T. B. (2023) · arXiv · doi:10.48550/arxiv.2305.14387 · Cited by 78
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., & Hashimoto, T. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1308 · Cited by 76
Neuroaesthetics and art's diversity and universality
Nadal, M., & Chatterjee, A. K. (2018) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1487 · Cited by 72
Controlling Style in Generated DialoguePreprint
Smith, E. M., Gonzalez-Rico, D., Dinan, E., & Boureau, Y.-L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.10855 · Cited by 53
RRHF: Rank Responses to Align Language Models with Human Feedback without tears
Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., & Huang, F. (2023) · arXiv · doi:10.48550/arxiv.2304.05302 · Cited by 41
Ji, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
Extracting and Inferring Personal Attributes from Dialogue
Wang, Z., Zhou, X., Koncel-Kedziorski, R., Marin, A., & Xia, F. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.6 · Cited by 9
Self-citationMeasuring and signing fairness as performance under multiple stakeholder distributionsPreprint
Lopez-Paz, D., Bouchacourt, D., Sagun, L., & Usunier, N. (2022) · arXiv.org · doi:10.48550/arxiv.2207.09960 · Cited by 9
Cited by
Sorted by citations
Kenthapadi, K., Sameki, M., & Taly, A. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671467 · Cited by 36
Metadata
- Authors
- Dong, Y., Wang, Z., Sreedhar, M., Wu, X., & Kuchaiev, O.
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Published
- 2023
- Pages
- 11275-11288
- OpenAlex
- W4389520260
- Cited by
- 15
- References
- 34
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 4
- FWCI
- 2.66
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
