CoChat
Sign Up
PreprintCited by 13Reinforcement Learning in RoboticsAdaptive Dynamic Programming ControlArtificial Intelligence Applications

Large Language Models can Implement Policy Iteration

Open access
Brooks, Ethan, Walls, Logan, Lewis, Richard L., Singh, Satinder·2022·arXiv
DOI 10.48550/arxiv.2210.03821arXiv 2210.03821
Open this paper in CoChat — free accountOpen PDF

Abstract

This work presents In-Context Policy Iteration, an algorithm for performing Reinforcement Learning (RL), in-context, using foundation models. While the application of foundation models to RL has received considerable attention, most approaches rely on either (1) the curation of expert demonstrations (either through manual design or task-specific pretraining) or (2) adaptation to the task of interest using gradient methods (either fine-tuning or training of adapter layers). Both of these techniques have drawbacks. Collecting demonstrations is labor-intensive, and algorithms that rely on them do not outperform the experts from which the demonstrations were derived. All gradient techniques are inherently slow, sacrificing the "few-shot" quality that made in-context learning attractive to begin with. In this work, we present an algorithm, ICPI, that learns to perform RL tasks without expert demonstrations or gradients. Instead we present a policy-iteration method in which the prompt content is the entire locus of learning. ICPI iteratively updates the contents of the prompt from which it derives its policy through trial-and-error interaction with an RL environment. In order to eliminate the role of in-weights learning (on which approaches like Decision Transformer rely heavily), we demonstrate our algorithm using Codex, a language model with no prior knowledge of the domains on which we evaluate it.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2210.03821

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Brooks et al. suggest a policy iteration approach using in-context learning.

    “This adaption was inspired by Brooks et al. [3], who suggest a policy iteration approach using in-context learning.”

    From Reflexion: Language Agents with Verbal Reinforcement Learning (2023) · cites this paper as [3]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 23

  1. [2][2] Prithviraj Ammanabrolu et al. 'Learning Knowledge Graph-based World Models of Textual Environments'. In: Advances in Neural Information Processing Systems . Vol. 34. Curran Associates, Inc., 2021, pp. 3720-3731. URL: https://proceedings.neurips.cc/ paper/2021/hash/1e747ddbea997a1b933aaf58a7953c3c-Abstract.html (visited on 05/16/2022). -
  2. [5][5] Tom Brown et al. 'Language Models are Few-Shot Learners'. In: Advances in Neural Information Processing Systems . Vol. 33. Curran Associates, Inc., 2020, pp. 1877-1901. URL: https: / / papers . nips . cc / paper / 2020 / hash / 1457c0d6bfcb4967418bfb8ac142f64a Abstract.html (visited on 05/14/2022). -
  3. [6][6] Tom Brown et al. 'Language models are few-shot learners'. In: Advances in neural information processing systems 33 (2020), pp. 1877-1901. -
  4. [8][8] Lili Chen et al. 'Decision Transformer: Reinforcement Learning via Sequence Modeling'. In: arXiv:2106.01345 [cs] (June 2021). URL: http://arxiv.org/abs/2106.01345 (visited on 05/18/2022). - · arXiv
  5. [9][9] Mark Chen et al. Evaluating Large Language Models Trained on Code . Tech. rep. arXiv:2107.03374. arXiv, July 2021. URL: http://arxiv.org/abs/2107.03374 (visited on 05/15/2022). - · arXiv
  6. [10][10] Mark Chen et al. 'Evaluating large language models trained on code'. In: arXiv preprint arXiv:2107.03374 (2021). - · arXiv
  7. [11][11] Yanda Chen et al. On the Relation between Sensitivity and Accuracy in In-context Learning . arXiv:2209.07661 [cs]. Sept. 2022. URL: http://arxiv.org/abs/2209.07661 (visited on 09/28/2022). - · arXiv
  8. [12][12] Daniel Fried et al. 'InCoder: A Generative Model for Code Infilling and Synthesis'. In: arXiv:2204.05999 [cs] (Apr. 2022). URL: http://arxiv.org/abs/2204.05999 (visited on 05/15/2022). - · arXiv
  9. [13][13] Leo Gao et al. 'The Pile: An 800GB Dataset of Diverse Text for Language Modeling'. In: arXiv:2101.00027 [cs] (Dec. 2020). URL: http://arxiv.org/abs/2101.00027 (visited on 05/18/2022). - · arXiv
  10. [14][14] Divyansh Garg et al. 'LISA: Learning Interpretable Skill Abstractions from Language'. In: arXiv:2203.00054 [cs] (Feb. 2022). URL: http://arxiv.org/abs/2203.00054 (visited on 05/14/2022). - · arXiv
  11. [16][16] Felix Hill et al. 'Human Instruction-Following with Deep Reinforcement Learning via TransferLearning from Text'. In: arXiv:2005.09382 [cs] (May 2020). URL: http://arxiv.org/ abs/2005.09382 (visited on 05/15/2022). - · arXiv
  12. [18][18] Wenlong Huang et al. 'Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents'. In: arXiv:2201.07207 [cs] (Mar. 2022). URL: http: //arxiv.org/abs/2201.07207 (visited on 05/14/2022). - · arXiv
  13. [21][21] Michael Laskin et al. 'In-context reinforcement learning with algorithm distillation'. In: arXiv preprint arXiv:2210.14215 (2022). - · arXiv
  14. [23][23] Shuang Li et al. 'Pre-Trained Language Models for Interactive Decision-Making'. In: arXiv:2202.01771 [cs] (Feb. 2022). URL: http://arxiv.org/abs/2202.01771 (visited on 05/14/2022). - · arXiv
  15. [25][25] Arjun Majumdar et al. 'Improving Vision-and-Language Navigation with Image-Text Pairs from the Web'. In: arXiv:2004.14973 [cs] (May 2020). URL: http://arxiv.org/abs/ 2004.14973 (visited on 05/14/2022). - · arXiv
  16. [27][27] Xiangyu Peng et al. 'Inherently Explainable Reinforcement Learning in Natural Language'. In: arXiv:2112.08907 [cs] (Dec. 2021). URL: http://arxiv.org/abs/2112.08907 (visited on 05/14/2022). - · arXiv
  17. [28][28] Scott Reed et al. A Generalist Agent . arXiv:2205.06175 [cs]. May 2022. DOI: 10.48550/ arXiv.2205.06175 . URL: http://arxiv.org/abs/2205.06175 (visited on 08/09/2022). - · arXiv
  18. [29][29] Younggyo Seo et al. HARP: Autoregressive Latent Video Prediction with High-Fidelity Image Generator . arXiv:2209.07143 [cs]. Sept. 2022. URL: http://arxiv.org/abs/2209.07143 (visited on 09/28/2022). - · arXiv
  19. [30][30] Aaditya K. Singh et al. Know your audience: specializing grounded language models with the game of Dixit . arXiv:2206.08349 [cs]. June 2022. DOI: 10.48550/arXiv.2206.08349 . URL: http://arxiv.org/abs/2206.08349 (visited on 08/09/2022). - · DOI
  20. [31][31] Ishika Singh et al. 'Pre-trained Language Models as Prior Knowledge for Playing Text-based Games'. In: arXiv:2107.08408 [cs] (Dec. 2021). URL: http://arxiv.org/abs/2107. 08408 (visited on 05/14/2022). - · arXiv
  21. [32][32] Allison C. Tam et al. 'Semantic Exploration from Language Abstractions and Pretrained Representations'. In: arXiv:2204.05080 [cs] (Apr. 2022). URL: http://arxiv.org/abs/ 2204.05080 (visited on 05/14/2022). - · arXiv
  22. [33][33] Denis Tarasov et al. 'Prompts and Pre-Trained Language Models for Offline Reinforcement Learning'. en. In: Mar. 2022. URL: https://openreview.net/forum?id=Spf4TE6NkWq (visited on 05/14/2022). -
  23. [35][35] Jason Wei et al. 'Chain of Thought Prompting Elicits Reasoning in Large Language Models'. In: arXiv:2201.11903 [cs] (Apr. 2022). URL: http://arxiv.org/abs/2201.11903 (visited on 05/15/2022). - · arXiv

Cited by

Sorted by citations

Related

Authors
Brooks, E., Walls, L., Lewis, R. L., & Singh, S.
ORCID
Singh, Satinder
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
30349-30366
ISSN
1049-5258 (ISSN-L)
DOI
10.48550/arxiv.2210.03821
arXiv
2210.03821
OpenAlex
W7133212394
Cited by
9
References
0
Citations by year
2026: 6 · 2025: 1 · 2024: 2
FWCI
1.60
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Adaptive Dynamic Programming Control
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
large language models, policy iteration, reinforcement learning, policy optimization, language model fine-tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review