Training Language Models with Language Feedback at Scale
Open accessAbstract
Pretrained language models often generate outputs that are not in line with human preferences, such as harmful text or factually incorrect summaries. Recent work approaches the above issues by learning from a simple form of human feedback: comparisons between pairs of model-generated outputs. However, comparison feedback only conveys limited information about human preferences. In this paper, we introduce Imitation learning from Language Feedback (ILF), a new approach that utilizes more informative language feedback. ILF consists of three steps that are applied iteratively: first, conditioning the language model on the input, an initial LM output, and feedback to generate refinements. Second, selecting the refinement incorporating the most feedback. Third, finetuning the language model to maximize the likelihood of the chosen refinement given the input. We show theoretically that ILF can be viewed as Bayesian Inference, similar to Reinforcement Learning from human feedback. We evaluate ILF's effectiveness on a carefully-controlled toy task and a realistic summarization task. Our experiments demonstrate that large language models accurately incorporate feedback and that finetuning with ILF scales well with the dataset size, even outperforming finetuning on human summaries. Learning from both language and comparison feedback outperforms learning from each alone, achieving human-level summarization performance.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Language feedback is more effective than preference-based feedback in the context of summarization systems.
“Recently, [163] has shown that language feedback is more effective than preference-based feedback in the context of summarization systems.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [163]
In §H.2 (page unavailable), the preference-based OPT-RM method achieved a 45.1 ± 1.9% win rate against human summaries, versus 36.0 ± 1.8% for ILF trained on language-feedback refinements. Section 4.3.2 also says OPT-RM outperformed the finetuning methods; the strongest result came from combining language and binary feedback, not from language feedback alone. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredAutomated metrics like BLEU and ROUGE often do not correlate with human judgment, making them unreliable for evaluation and training.
“Automated metrics like BLEU and ROUGE [99] often do not correlate with human judgment [103, 164, 166, 177], making them unreliable for evaluation and training.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [164]
In “Scoring Refinements with InstructRM” (page unavailable), the paper reports that embedding similarity did not outperform random selection against human rankings on its dataset. That supports a narrower concern about an automated scoring method, but the paper does not establish that BLEU and ROUGE often fail to correlate with human judgment or are generally unreliable for evaluation and training. (LLM, conf=0.83)
Experimental — accuracy not yet measuredCollecting pairwise feedback might still be difficult for near similar responses and may result in much time spent by labelers even on single input.
“However, collecting pairwise feedback might still be difficult for near similar responses and may result in much time spent by the labelers even on single input [163].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [163]
The Dataset Analysis section (page unavailable) reports that annotators spent 61.5 ± 5.3 seconds on average on a binary comparison, supporting the claim that one comparison can take time. The annotation instructions allow either choice when summaries have no distinguishing feature, but the paper does not say that comparing near-similar responses is difficult or that such similarity increases labeling time. (LLM, conf=0.84)
Experimental — accuracy not yet measuredLabelers take only 3x times to provide language feedback compared to preference-based feedback, despite providing much granular information.
“Although task-dependent, [163] further find that labelers take only 3x times to provide language feedback compared to preference-based feedback, despite providing much granular information.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [163]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 41
- Andreas, J., Klein, D., and Levine, S. Modular multi- task reinforcement learning with policy sketches. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learn- ing, volume 70 of Proceedings of Machine Learn- ing Research, pp. 166-175. PMLR, 06-11 Aug 2017. URL https://proceedings.mlr.press/v70/ andreas17a.html.
- Austin, J., Odena, A., Nye, M. I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https: //arxiv.org/abs/2108.07732. · arXiv
- Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. · arXiv
- Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020. URL https://arxiv.org/pdf/ 2005.14165.pdf.
- Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., and Blunsom, P. e-snli: Natural language inference with natural language explanations. Advances in Neural In- formation Processing Systems, 31, 2018. URL https: //arxiv.org/pdf/1812.01193.pdf.
- Chen, A., Scheurer, J., Korbak, T., Campos, J. A., Chan, J. S., Bowman, S. R., Cho, K., and Perez, E. Improv- ing code generation by training with natural language feedback. arXiv preprint arXiv:2303.16749, 2023. · arXiv
- Dohan, D., Xu, W., Lewkowycz, A., Austin, J., Bieber, D., Lopes, R. G., Wu, Y., Michalewski, H., Saurous, R. A., Sohl-Dickstein, J., et al. Language model cascades. arXiv preprint arXiv:2207.10342, 2022. · arXiv
- Fidler, S. et al. Teaching machines to describe images with natural language feedback. Advances in Neural Information Processing Systems, 30, 2017.
- Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization, 2022. URL https: //arxiv.org/abs/2210.10760.
- Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. Realtoxicityprompts: Eval- uating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462, 2020. URL https://aclanthology.org/2020. findings-emnlp.301.pdf. · arXiv
- Hancock, B., Bordes, A., Mazare, P.-E., and Weston, J. Learning from dialogue after deployment: Feed yourself, chatbot! arXiv preprint arXiv:1901.05415, 2019. · arXiv
- Hase, P. and Bansal, M. When can models learn from explanations? a formal framework for understand- ing the roles of explanation data. arXiv preprint arXiv:2102.02201, 2021. URL https://arxiv. org/pdf/2102.02201.pdf. · arXiv
- Hermann, K. M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
- Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. URL https:// arxiv.org/pdf/1904.09751.pdf. · arXiv
- Korbak, T., Perez, E., and Buckley, C. L. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275, 2022. · arXiv
- Lampinen, A. K., Dasgupta, I., Chan, S. C., Matthewson, K., Tessler, M. H., Creswell, A., McClelland, J. L., Wang, J. X., and Hill, F. Can language models learn from ex- planations in context? arXiv preprint arXiv:2204.02329, 2022. · arXiv
- Li, J., Miller, A. H., Chopra, S., Ranzato, M., and Weston, J. Dialogue learning with human-in-the-loop. arXiv preprint arXiv:1611.09823, 2016. · arXiv
- Li, Z., Sharma, P., Lu, X. H., Cheung, J. C., and Reddy, S. Using interactive feedback to improve the accuracy and explainability of question answering systems post- deployment. arXiv preprint arXiv:2204.03025, 2022. · arXiv
- Lin, J., Fried, D., Klein, D., and Dragan, A. Infer- ring rewards from language in context. arXiv preprint arXiv:2204.02515, 2022. · arXiv
- Liu, Y., Deb, B., Teruel, M., Halfaker, A., Radev, D., and Awadallah, A. H. On improving summarization fac- tual consistency from natural language feedback. arXiv preprint arXiv:2212.09968, 2022. · arXiv
- Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.08786, 2021. · arXiv
- Matiana, S., Smith, J., Teehan, R., Castricato, L., Biderman, S., Gao, L., and Frazier, S. Cut the carp: Fishing for zero- shot story evaluation. arXiv preprint arXiv:2110.03111, 2021. · arXiv
- Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saun- ders, W., et al. WebGPT: Browser-assisted question- answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. URL https://arxiv. org/pdf/2112.09332.pdf. · arXiv
- Nguyen, K. X., Misra, D., Schapire, R., Dudík, M., and Shafto, P. Interactive learning from activity description. In International Conference on Machine Learning, pp. 8096-8108. PMLR, 2021.
- Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Preprint, 2022. URL https://cdn.openai.com/papers/ Training_language_models_to_follow_ instructions_with_human_feedback.pdf.
- Perez, E., Kiela, D., and Cho, K. True few-shot learning with language models. Advances in Neural Information Processing Systems, 34:11054-11070, 2021.
- Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoff- mann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. Scaling language models: Meth- ods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. URL https:// arxiv.org/pdf/2112.11446.pdf. · arXiv
- Rupprecht, C., Laina, I., Navab, N., Hager, G. D., and Tombari, F. Guide me: Interacting with deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8551-8561, 2018.
- Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022. · arXiv
- Scheurer, J., Campos, J. A., Chan, J. S., Chen, A., Cho, K., and Perez, E. Training language models with lan- guage feedback. In The First Workshop on Learning with Natural Language Supervision at ACL, 2022.
- Schick, T., Dwivedi-Yu, J., Jiang, Z., Petroni, F., Lewis, P., Izacard, G., You, Q., Nalmpantis, C., Grave, E., and Riedel, S. Peer: A collaborative language model. arXiv preprint arXiv:2208.11663, 2022. · arXiv
- Shi, W., Dinan, E., Shuster, K., Weston, J., and Xu, J. When life gives you lemons, make cherryade: Converting feed- back from bad responses into good labels. arXiv preprint arXiv:2210.15893, 2022. · arXiv
- Stacey, J., Belinkov, Y., and Rei, M. Supervising Model Attention with Human Explanations for Robust Natural Language Inference. arXiv preprint arXiv:2104.08142, 2021. URL https://arxiv.org/pdf/2104. 08142.pdf. · arXiv
- Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020. URL https://arxiv.org/ pdf/2009.01325.pdf.
- Sumers, T. R., Ho, M. K., Hawkins, R. D., Narasimhan, K., and Griffiths, T. L. Learning rewards from linguistic feedback. feedback, 1(2):3, 2021.
- Tam, A. C., Rabinowitz, N. C., Lampinen, A. K., Roy, N. A., Chan, S. C., Strouse, D., Wang, J. X., Banino, A., and Hill, F. Semantic exploration from language ab- stractions and pretrained representations. arXiv preprint arXiv:2204.05080, 2022. · arXiv
- Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elic- its reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. · arXiv
- Weston, J. E. Dialog-based language learning. Advances in Neural Information Processing Systems, 29, 2016.
- Xu, J., Ung, M., Komeili, M., Arora, K., Boureau, Y.-L., and Weston, J. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. arXiv preprint arXiv:2208.03270, 2022. · arXiv
- Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. · arXiv
- Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Rad- ford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. URL https: //arxiv.org/pdf/1909.08593.pdf. · arXiv
Cited by
Sorted by citations
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970 · see citing works
Towards conversational diagnostic artificial intelligence
Tu, T., Schaekermann, M., Palepu, A., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Cheng, Y., et al. (2025) · Nature · doi:10.1038/s41586-025-08866-7 · Cited by 366 · see citing works
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · see citing works
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
Self-citationPan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00660 · Cited by 86 · see citing works
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., Deshpande, A., & Castro da Silva, B. (2026) · ACM Computing Surveys · doi:10.1145/3743127 · Cited by 92 · see citing works
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Fernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P. H., Bertsch, A., de Souza, J. G. C., Zhou, S., Wu, T., Neubig, G., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00626 · Cited by 57 · see citing works
New Frontiers in Information Systems Theorizing: Human-gAI Collaboration
Järvenpää, S. L., & Klein, S. P. (2024) · Journal of the Association for Information Systems · doi:10.17705/1jais.00868 · Cited by 29 · see citing works
Goal-Driven Explainable Clustering via Language Descriptions
Wang, Z., Shang, J., & Zhong, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.657 · Cited by 31 · see citing works
Chen, Y., Sikka, K., Cogswell, M., Ji, H., & Divakaran, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01350 · Cited by 29 · see citing works
Obaidoon, S., & Wei, H. (2024) · Future in Educational Research · doi:10.1002/fer3.39 · Cited by 23 · see citing works
Aligning Large Language Models through Synthetic Feedback
Kim, S., Bae, S., Shin, J., Kang, S., Kwak, D., Yoo, K. M., & Seo, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.844 · Cited by 34 · see citing works
This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models
García-Ferrero, I., Altuna, B., Álvez, J., González-Dios, I., & Rigau, G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.531 · Cited by 28 · see citing works
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17 · see citing works
“…reward function or feedback function and then trains a goal-conditioned policy on these outputs (similar to (Andrychowicz et al., 2017)); and ILF (Scheurer et al., 2023), which uses natural language human feedback to prompt the model to produce better outputs than its original inputs, and then…”
BackgroundLearning from Mistakes via Cooperative Study Assistant for Large Language Models
Wang, D., & Li, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.659 · Cited by 10 · see citing works
VideoCon: Robust Video-Language Alignment via Contrast Captions
Bansal, H., Bitton, Y., Szpektor, I., Chang, K.-W., & Grover, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01321 · Cited by 8 · see citing works
DUnE: Dataset for Unified Editing
Akyürek, A. F., Pan, E., Kuwanto, G., & Wijaya, D. T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.114 · Cited by 7 · see citing works
TSSR: A Truncated and Signed Square Root Activation Function for Neural Networks
Gong, Y., Lin, Z., & Sun, J. (2023) · IEEE International Conference on Computer and Communications (ICCC) · doi:10.1109/iccc59590.2023.10507273 · Cited by 2 · see citing works
- Authors
- Scheurer, J., Campos, J. A., Korbak, T., Chan, J. S., Chen, A., Cho, K., & Perez, E.
- Affiliations
- New York University (US)
- University of the Basque Country (ES)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2303.16755
- OpenAlex
- W4361807044
- Cited by
- 14
- References
- 0
- Citations by year
- 2025: 2 · 2024: 5 · 2023: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1922658)
- Samsung Advanced Institute of Technology
- Open Philanthropy Project
- Ministerio de Educación, Cultura y Deporte
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- language model fine-tuning, human feedback, summarization, Bayesian inference, reinforcement learning from human feedback, factual accuracy, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
