Training Language Models to Follow Instructions with Human Feedback
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Existing literature (Ouyang et al., 2022) does not fully describe the details of these models, such as the pretraining corpus, model sizes, and model biases.”
From Self-Refine: Iterative Refinement with Self-Feedback (2023) · cites this paper as Ouyang et al., 2022
Experimental — accuracy not yet measuredIn the citing paper
“Specifically, providers mitigate race and gender bias through reinforcement learning from human feedback (RLHF), adversarial stress-testing known as red teaming, and hard-coded refusals governed by the provider's own content policies [6, 43, 71].”
From All Public Voices Are Equal, But Are Some More Equal Than Others to LLMs? (2026) · cites this paper as [71]
In the citing paper
“Fine-tuning a model from human feedback at scale, using preference data from paid raters or drawn from a large and willing user base, is an especially potent technique.It has the potential not only to shape a model's responses to better reflect user norms (for better or worse), but also to filter out toxic language, improve factual accuracy, and mitigate the tendency to fabricate information.”
From Talking about Large Language Models (2024) · cites this paper as [26]
In the citing paper
“These include general competencies such as in-context learning (ICL) [13], chain-of-thought (CoT) [17] reasoning, and instruction following [25], as well as task-specialized skills like mathematical reasoning [30].”
From A Survey of Large Language Models (2026) · cites this paper as [25]
In the citing paper
“When external reward models or ground-truth answers are available, rejection sampling can also be applied to select higher-quality responses from multiple candidates [25].”
From A Survey of Large Language Models (2026) · cites this paper as [25]
41 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Cited by
Items 4,301–4,340 of 4,340
ELASTIC-V: Reconstructing Spatiotemporal Fidelity via Saliency Compression and Elastic Encoding
Liu, T., Wang, G., & Liu, Y. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3836253
Mitigating Hallucinations in Large Multimodal Reasoning Models by Regulating Parametric Prior Drift
Qu, X., Wu, Y., Xie, Z., Liu, D., & Wei, W. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3835890
Human and machine intelligence: Insights from children
Spelke, E. S., & Chandra, K. (2026) · AI Magazine · doi:10.1002/aaai.70118
MIRD: Multimodal Implicit Reasoning Distillation for Efficient Sarcasm Detection
Liu, R., Yu, T., Bai, L., Sun, S., Xue, J., & Guo, Y. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3835865
Tu, G., Jing, R., Sun, S., Wu, Y., Lang, Q., Liu, Y., Jin, B., & Xu, R. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3838612
Onti, W. H., Mahmood, M. S., & Mridha, M. F. (2026) · International Journal of Energy Research · doi:10.1155/er/8756057
GraphSense: Resource-Aware Graph Memory for Efficient Sequential Agents
Li, L., Sun, L., & Yuan, Z. (2026) · doi:10.1145/3841452.3841496
Generating Multiple-Choice Questions and Diagrams for Multimodal Machine Comprehension
Ding, W., Zhang, Y., Wang, Y.-X., Jatowt, A., Sun, Z., & Yang, Z. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3835864
Kong, J., Niu, Y., Xu, T., Liu, Q., & Yang, X. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3838636
Med-OR: Curriculum Reinforcement Learning for Open-Ended Medical Visual Reasoning
Rui, S., Chen, K., Ma, W., & Wang, X. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3835993
Öç, U. (2026) · Applied Measurement in Education · doi:10.1080/08957347.2026.2758589
Learning When to Invoke the Head: Routing-Aware Early Exit for MoE Language Models
Park, S. (2026) · Electronics · doi:10.3390/electronics15215159
Beyond Unimodal Reliance: Multimodal Synergy for Training-Free Test-Time Adaptation
Jiang, G., Ding, C. H., Dong, S., Ma, Z., Wang, Q., He, Y., & Gong, Y. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3835136
Yamada, K. (2026) · AI and Ethics · doi:10.1007/s43681-026-01493-6
TaME4D: Tailoring Multisensory Effects for Improving 4D Movie Watching Experiences
Chung, M., Nam, S., Park, J., & Yoo, Y. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3835393
Red-Teaming African-Language LLMs: Safety Alignment Failures Across Low-Resource LanguagesPreprint
Nyong, G. N., Okeyo, G. O., & Ndibwile, J. D. (2026) · Research Square · doi:10.21203/rs.3.rs-11242446/v1
Traj-MCPO: Trajectory-wise Marginal Consistent Policy Optimization
Zhang, S., Zhang, Q., Zhang, H., Zhou, L., Zang, X., He, Z., He, Z., Zhang, C., Cao, T., Liang, K., et al. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3835489
Self-citationState-Dependent Lagrange Multipliers for State-Wise Safety in Constrained Reinforcement Learning
Seo, M., & Choi, K. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-23010-2_14
Camara, L. G. S. (2024) · RCMOS - Revista Científica Multidisciplinar O Saber · doi:10.51473/rcmos.v1i2.2024.3142
MBench: Rebuilding Human-Aligned Zero-Shot Evaluation for Motion-Language Models
Ma, W., 宣恩允, Li, T., Yi, M., Liu, S., & Fan, S. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3835795
Adapting Knowledge and Human Resources to Digital Entrepreneurship in Tourism
Avdımıotıs, S., & Konstantinidis, I. (2026) · Merits · doi:10.3390/merits6040035
Huang, X., Xiang, P., Wang, J., Zhai, X., Liu, J., & Han, H. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3834945
LoopEdit: Learning Data-driven Hierarchical Planners for Short-Video Assembly
Gupta, S., Dasgupta, I., Petrangeli, S., & Subramanian, J. (2026) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3767308.3834917
Wang, Y., Lu, X., Guo, W., & Zhang, H. (2026) · Informatics · doi:10.3390/informatics13110188
Marinopoulou, A., & CHORIANOPOULOU, M. K. (2026) · ARHE · doi:10.19090/arhe.2026.45.315-343
Shah, V. (2025) · Sciences du Nord Nature Science and Technology · doi:10.71238/snnst.v2i02.179
Subjectivity and Integrity of Knowledge: Two Ethical Challenges of Artificial Intelligence
Gribkov, A. A. (2026) · Философия и культура · doi:10.7256/2454-0757.2026.11.83071
PromptLeaks: Assessing Data Leakage in Guardrails and Automated Jailbreak Tools
Sauwens, K., Knuts, S., hamme, T. V., & Desmet, L. (2026) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3847352.3848118
Rahman, R., & Sibert, C. (2026) · Frontiers in Computer Science · doi:10.3389/fcomp.2026.1932390
Untitled work
Ovcharov, V. (2026) · arXiv
“Standard RLHF [17] and DPO [20] rely on human-generated preference pairs.”
Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs
Tie, G., Yuan, Z., Zhao, Z., Hu, C., Gu, T., Zhang, R., Zhang, S., Wu, J., Tu, X., Jin, M., et al. (2025) · arXiv · doi:10.48550/arxiv.2510.16062 · Cited by 16
“However, Reflexion-v2 6 increases its effectiveness, yielding improvements of +7.33% on HotpotQA and +7.00% on AQUA.”
InfluentialSelf-citationTTRL: Test-Time Reinforcement Learning
Zuo, Y., Zhang, K., Sheng, L., Qu, S., Cui, G., Zhu, X., Li, H., Zhang, Y., Long, X., Hua, E., et al. (2025) · arXiv · Cited by 7
“Reinforcement Learning (RL) [44] plays a critical role in enhancing the instruction-following capabilities of Large Language Models (LLMs), particularly through approaches like Reinforcement Learning from Human Feedback (RLHF) [31].”
LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation
Fakhoury, S., Naik, A., Sakkas, G., Chakraborty, S., & Lahiri, S. K. (2024) · arXiv · doi:10.48550/arxiv.2404.10100
“5 architecture and Instruct-GPT [11] and can be used for a variety of natural language tasks.”
MethodInfluentialMAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems
Ye, R., Tang, S., Ge, R., Du, Y., Yin, Z., Chen, S., & Shao, J. (2025) · arXiv · doi:10.48550/arxiv.2503.03686
Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editing
Wu, Y., Ding, L., Shen, L., & Tao, D. (2025) · arXiv · doi:10.48550/arxiv.2511.12661
Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Models
Sel, B., Al-Tawaha, A., Khattar, V., Jia, R., & Jin, M. (2023) · arXiv · doi:10.48550/arxiv.2308.10379 · Cited by 15
Universal and Transferable Adversarial Attacks on Aligned Language Models
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15043 · Cited by 197
Prompt Injection Attack to Tool Selection in LLM Agents
Shi, J., Yuan, Z., Tie, G., Zhou, P., Gong, N. Z., & Sun, L. (2025) · arXiv · doi:10.48550/arxiv.2504.19793 · Cited by 2
AgentScope: A Flexible yet Robust Multi-Agent Platform
Gao, D., Li, Z., Pan, X., Kuang, W., Ma, Z., Qian, B., Wei, F., Zhang, W., Xie, Y., Chen, D., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.14034 · Cited by 11
“With the advancement of Large Language Models (LLMs) (Ouyang et al., 2022; OpenAI, 2023; Touvron et al., 2023a,b), multi-agent applications have made great process in both research and industrial communities, including software engineering (Hong et al., 2023), society simulation (Park et al.,…”
Method
Metadata
- Authors
- Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R.
- Venue
- Advances in Neural Information Processing Systems 35
- Published
- 2022
- Pages
- 27730-27744
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133211372
- Cited by
- 3,117
- References
- 0
- Citations by year
- 2026: 1421 · 2025: 544 · 2024: 852 · 2023: 297 · 2022: 3
- FWCI
- 234.27
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- neural information processing systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- instruction following, language models, human feedback, reinforcement learning from human feedback, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.