LLF-Bench: Benchmark for Interactive Learning from Language Feedback
Open accessAbstract
We introduce a new benchmark, LLF-Bench (Learning from Language Feedback Benchmark; pronounced as "elf-bench"), to evaluate the ability of AI agents to interactively learn from natural language feedback and instructions. Learning from language feedback (LLF) is essential for people, largely because the rich information this feedback provides can help a learner avoid much of trial and error and thereby speed up the learning process. Large Language Models (LLMs) have recently enabled AI agents to comprehend natural language -- and hence AI agents can potentially benefit from language feedback during learning like humans do. But existing interactive benchmarks do not assess this crucial capability: they either use numeric reward feedback or require no learning at all (only planning or information retrieval). LLF-Bench is designed to fill this omission. LLF-Bench is a diverse collection of sequential decision-making tasks that includes user recommendation, poem writing, navigation, and robot control. The objective of an agent is to interactively solve these tasks based on their natural-language instructions and the feedback received after taking actions. Crucially, to ensure that the agent actually "learns" from the feedback, LLF-Bench implements several randomization techniques (such as paraphrasing and environment randomization) to ensure that the task isn't familiar to the agent and that the agent is robust to various verbalizations. In addition, LLF-Bench provides a unified OpenAI Gym interface for all its tasks and allows the users to easily configure the information the feedback conveys (among suggestion, explanation, and instantaneous performance) to study how agents respond to different types of feedback. Together, these features make LLF-Bench a unique research platform for developing and testing LLF agents.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Critiques distinguish hypotheses more efficiently than scalar rewards.
“By characterizing critiques as Reward-Informative Feedback (Cheng et al., 2023), we show they distinguish hypotheses more efficiently than scalar rewards.”
From Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback (2025) · cites this paper as Cheng et al., 2023
LLMs are not deterministic even at temperature zero.
“This can still lead to success after the first trial since LLMs aren't deterministic even at temperature zero.”
From AI Agents That Matter (2024) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Research works printed but not linked: 50
- M. Abdulhai, I. White, C. Snell, C. Sun, J. Hong, Y. Zhai, K. Xu, and S. Levine. LMRL Gym: Bench- marks for multi-turn reinforcement learning with language models. arXiv preprint arXiv:2311.18232, 2023. · arXiv
- D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. · arXiv
- S. Arora, A. Narayan, M. F. Chen, L. Orr, N. Guha, K. Bhatia, I. Chami, and C. Re. Ask me anything: A simple strategy for prompting language models. In ICLR, 2023.
- D. Bahdanau, F. Hill, J. Leike, E. Hughes, P. Kohli, and E. Grefenstette. Learning to understand goal specifications by modelling reward. In ICLR, 2019.
- C. Basu, E. Bıyık, Z. He, M. Singhal, and D. Sadigh. Active learning of reward dynamics from hierarchical queries. In IROS, 2019.
- Y. Bisk, D. Yuret, and D. Marcu. Natural language communication with robots. In NAACL-HLT, 2016.
- G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAI gym. arXiv preprint arXiv:1606.01540, 2016. · arXiv
- D. S. Chaplot, K. M. Sathyendra, R. K. Pasumarthi, D. Rajagopal, and R. Salakhutdinov. Gated- attention architectures for task-oriented language grounding. In AAAI, 2018.
- H. Chen, A. Suhr, D. Misra, N. Snavely, and Y. Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, 2019.
- M. Chevalier-Boisvert, D. Bahdanau, S. Lahlou, L. Willems, C. Saharia, T. H. Nguyen, and Y. Bengio. BabyAI: First steps towards grounded language learning with a human in the loop. In ICLR, 2019.
- F. Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547, 2019. · arXiv
- M.-A. Côté, A. Kádár, X. Yuan, B. Kybartas, T. Barnes, E. Fine, J. Moore, M. Hausknecht, L. El Asri, M. Adada, W. Tay, and A. Trischler. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018 Held in Conjunction with IJCAI 2018. Springer, 2019.
- S. Edunov, M. Ott, M. Auli, and D. Grangier. Understanding back-translation at scale. In EMNLP, 2018.
- Y. Ge, W. Hua, J. Ji, J. Tan, S. Xu, and Y. Zhang. OpenAGI: When LLM meets domain experts. arXiv preprint arXiv:2304.04370, 2023. · arXiv
- K. Guu, P. Pasupat, E. Liu, and P. Liang. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In ACL, 2017.
- A. W. Hanjie, V. Y. Zhong, and K. Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. In ICML, 2021.
- M. Janner, K. Narasimhan, and R. Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics, 6:49-61, 2018.
- S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language-driven representation learning for robotics. arXiv preprint arXiv:2302.12766, 2023. · arXiv
- H. Küttler, N. Nardelli, A. H. Miller, R. Raileanu, M. Selvatici, E. Grefenstette, and T. Rocktäschel. The NetHack learning environment. In NeurIPS, 2020.
- A. Leidinger, R. van Rooij, and E. Shutova. The language of prompting: What linguistic properties make a prompt successful? arXiv preprint arXiv:2311.01967, 2023. · arXiv
- E. Z. Liu, K. Guu, P. Pasupat, and P. Liang. Reinforcement learning on web interfaces using workflow- guided exploration. In ICLR, 2018.
- H. Liu, A. Chen, Y. Zhu, A. Swaminathan, A. Kolobov, and C.-A. Cheng. Interactive robot learning from verbal correction. arXiv preprint arXiv:2310.17555, 2023a. · arXiv
- X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang. AgentBench: Evaluating LLMs as Agents. arXiv preprint arXiv:2308.03688, 2023b. · arXiv
- Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, and Y. Liu. Jailbreaking ChatGPT via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860, 2023c. · arXiv
- D. Misra, J. Langford, and Y. Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In EMNLP, 2017.
- D. Misra, A. Bennett, V. Blukis, E. Niklasson, M. Shatkhin, and Y. Artzi. Mapping instructions to actions in 3D environments with visual goal prediction. In EMNLP, 2018.
- K. Narasimhan, T. D. Kulkarni, and R. Barzilay. Language understanding for text-based games using deep reinforcement learning. In EMNLP, 2015.
- K. X. Nguyen, D. Misra, R. Schapire, M. Dudík, and P. Shafto. Interactive learning from activity description. In ICML, 2021.
- M. T. Ribeiro, S. Singh, and C. Guestrin. Semantically equivalent adversarial rules for debugging nlp models. In ACL, 2018.
- S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011.
- T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. · arXiv
- T. Shi, A. Karpathy, L. Fan, J. Hernandez, and P. Liang. World of Bits: An open-domain platform for web-based agents. In ICML, 2017.
- M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, 2020.
- M. Shridhar, X. Yuan, M.-A. Côté, Y. Bisk, A. Trischler, and M. Hausknecht. ALFWorld: Aligning text and embodied environments for interactive learning. In ICLR, 2021.
- V. J. Shute. Focus on formative feedback. Review of educational research, 78:153-189, 2008.
- J. Spencer, S. Choudhury, A. Venkatraman, B. Ziebart, and J. A. Bagnell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2102.02872, 2021. · arXiv
- T. R. Sumers, M. K. Ho, R. D. Hawkins, K. Narasimhan, and T. L. Griffiths. Learning rewards from linguistic feedback. In AAAI, 2021.
- R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
- E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh. Universal adversarial triggers for attacking and analyzing NLP. In EMNLP, 2019.
- G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023a. · arXiv
- S. I. Wang, P. Liang, and C. D. Manning. Learning language games through interaction. In ACL, 2016.
- X. Wang, Z. Wang, J. Liu, Y. Chen, L. Yuan, H. Peng, and H. Ji. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. arXiv preprint arXiv:2309.10691, 2023b. · arXiv
- Y. Wolf, N. Wies, Y. Levine, and A. Shashua. Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304.11082, 2023. · arXiv
- Q. Wu, G. Bansal, J. Zhang, Y. Wu, S. Zhang, E. Zhu, B. Li, L. Jiang, X. Zhang, and C. Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155, 2023. · arXiv
- T.-Y. Yang, M. Hu, Y. Chow, P. Ramadge, and K. R. Narasimhan. Safe reinforcement learning with natural language constraints. In NeurIPS, 2021.
- S. Yao, H. Chen, J. Yang, and K. R. Narasimhan. WebShop: Towards scalable real-world web inter- action with grounded language agents. In NeurIPS, 2022.
- S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. · arXiv
- T. Yu, D. Quillen, Z. He, R. Julian, A. Narayan, H. Shively, A. Bellathur, K. Hausman, C. Finn, and S. Levine. Meta-World: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2019.
- V. Zhong, H. A. Wang, S. Wang, K. R. Narasimhan, and L. Zettlemoyer. SILG: The multi-domain symbolic interactive language grounding benchmark. In NeurIPS, 2021.
- K. Zhu, J. Wang, J. Zhou, Z. Wang, H. Chen, Y. Wang, L. Yang, W. Ye, Y. Zhang, N. Z. Gong, and X. Xie. PromptBench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528, 2023. · arXiv
Cited by
Sorted by citations
A Framework for Integrated Digital Forensic Investigation Employing AutoGen AI Agents
Wickramasekara, A., & Scanlon, M. (2024) · International Symposium on Digital Forensics and Security (ISDFS) · doi:10.1109/isdfs60797.2024.10527235 · Cited by 21 · see citing works
Kapoor, S., Stroebl, B., Siegel, Z. S., Nadgir, N., & Narayanan, A. (2024) · arXiv · doi:10.48550/arxiv.2407.01502 · Cited by 13 · see citing works
“These may include web navigation, visual and spatial reasoning, long term planning, etc. LLF-Bench[8] Interactive learning Measurestheaccuracyofagentsonasetoflearning-basedtasks.”
Reflect-RL: Two-Player Online RL Fine-Tuning for LMs
Zhou, R., Du, S., & Li, B. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.56 · Cited by 5 · see citing works
Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs
Cheng, C.-A., Nie, A., & Swaminathan, A. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-2287 · Cited by 4 · see citing works
Self-citationTurkingBench: A Challenge Benchmark for Web Agents
Xu, K. H., Kordi, Y., Nayak, T., Asija, A., Wang, Y., Sanders, K., Byerly, A., Zhang, J., Durme, B. V., & Khashabi, D. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.188 · Cited by 2 · see citing works
Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use
Xi, J., He, Y., Yang, J., Dai, Y., & Chai, J. Y. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.237 · Cited by 1 · see citing works
Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
Zhang, X., Zhang, Y., Sun, H., Feng, K., Lu, C., Yang, C., & Meng, H. (2025) · arXiv · doi:10.48550/arxiv.2506.03106
A Bilevel Reinforcement Learning Framework with Language Prior Knowledge
Yan, X., Song, Y., Cui, X., Christianos, F., Zhang, H., Wang, J., & Mguni, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06106-5_13
Reinforcement Learning from Human and AI Feedback for Large Language Model Alignment: A Review
Chowdhury, T. (2026) · International Journal on Smart & Sustainable Intelligent Computing · doi:10.63503/j.ijssic.2026.234
- Authors
- Cheng, C.-A., Kolobov, A., Misra, D., Nie, A., & Swaminathan, A.
- Affiliations
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2312.06853
- OpenAlex
- W4389761409
- Cited by
- 1
- References
- 0
- Citations by year
- 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- natural language feedback, interactive learning, AI agents, benchmark, sequential decision making, large language models, paraphrase, user recommendation, robot control, navigation, feedback types
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
