CoChat
Sign Up
PreprintCited by 1Reinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningAI-based Problem Solving and Planning

A Survey of In-Context Reinforcement Learning

Open access
Moeini, Amir, Wang, Jiuqi, Beck, Jacob, Blaser, Ethan, Whiteson, Shimon, Chandra, Rohan, Zhang, Shangtong·2025·arXiv
DOI 10.48550/arxiv.2502.07978arXiv 2502.07978
Open this paper in CoChat — free accountOpen PDF

Abstract

Reinforcement learning (RL) agents typically optimize their policies by performing expensive backward passes to update their network parameters. However, some agents can solve new tasks without updating any parameters by simply conditioning on additional context such as their action-observation histories. This paper surveys work on such behavior, known as in-context reinforcement learning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2502.07978

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In-context reinforcement learning (ICRL) extends the concept of inter-test-time in-context learning by maintaining histories of observations and actions within the agent's context window.

    “The field of in-context reinforcement learning (ICRL) (Moeini et al., 2025; Laskin et al., 2022; Lee et al., 2023) extends this concept by maintaining histories of observations and actions within the agent's context window.”

    From A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence (2025) · cites this paper as Moeini et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 66

  1. Pieter Abbeel and Andrew Y. Ng. Exploration and appren- ticeship learning in reinforcement learning. In Proceed- ings of the International Conference on Machine Learning, 2005.
  2. Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-Context Language Learning: Architectures and Algo- rithms. ArXiv preprint, 2024.
  3. Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In Proceedings of the International Conference on Machine Learning, 2017.
  4. Yu Bai and Chi Jin. Provable self-play algorithms for com- petitive reinforcement learning. In Proceedings of the In- ternational Conference on Machine Learning, 2020.
  5. Jakob Bauer, Kate Baumli, Feryal M. P. Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez Nieves, Ne- manja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Satinder Singh, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei M. Zhang. Human-timescale adaptation in an open-ended task space. In Proceedings of the International Conference on Machine Learning, 2023.
  6. Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A Survey of Meta-Reinforcement Learning. ArXiv preprint, 2023.
  7. Maximilian Beck, Korbinian Pöppel, Markus Spanring, An- dreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended Long Short-Term Memory. ArXiv preprint, 2024.
  8. David Brandfonbrener, Alberto Bietti, Jacob Buckman, Ro- main Laroche, and Joan Bruna. When does return- conditioned supervised learning work for offline reinforce- ment learning? In Advances in Neural Information Pro- cessing Systems, 2022.
  9. Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. · arXiv
  10. Rujikorn Charakorn, Poramate Manoonpong, and Nat Dilokthanakul. Learning to cooperate with unseen agent via meta-reinforcement learning. arXiv preprint arXiv:2111.03431, 2021. · arXiv
  11. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, 2021.
  12. Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Con- textual bandits with linear payoff functions. In Proceedings of the International Conference on Artificial Intelligence and Statistics, 2011.
  13. Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to bench- mark reinforcement learning. In Proceedings of the Inter- national Conference on Machine Learning, 2020.
  14. Jonathan Cook, Chris Lu, Edward Hughes, Joel Z. Leibo, and Jakob Nicolaus Foerster. Artificial Generational Intelli- gence: Cultural Accumulation in Reinforcement Learning. In The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024.
  15. Zhenwen Dai, Federico Tomasi, and Sina Ghiassian. In-context Exploration-Exploitation for Reinforcement Learning. ArXiv preprint, 2024.
  16. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui. A Survey on In-context Learning. ArXiv preprint, 2023.
  17. Juncheng Dong, Moyang Guo, Ethan X. Fang, Zhuoran Yang, and Vahid Tarokh. In-Context Reinforcement Learning Without Optimal Action Labels. In ICML 2024 Workshop on In-Context Learning, 2024.
  18. Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL$ˆ2$: Fast Reinforce- ment Learning via Slow Reinforcement Learning. ArXiv preprint, 2016.
  19. Ahmad Elawady, Gunjan Chhablani, Ram Ramrakhya, Karmesh Yadav, Dhruv Batra, Zsolt Kira, and Andrew Szot. ReLIC: A Recipe for 64k Steps of In-Context Re- inforcement Learning for Embodied AI. ArXiv preprint, 2024.
  20. Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline RL via supervised learning? In Proceedings of the International Conference on Learning Representations, 2022.
  21. Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Gen- eralized decision transformer for offline hindsight informa- tion matching. In Proceedings of the International Confer- ence on Learning Representations, 2022.
  22. Matthias Gerstgrasser and David C Parkes. Meta-rl for multi- agent rl: Learning to adapt to evolving agents. In Advances in Neural Information Processing Systems, 2022.
  23. Jake Grigsby, Linxi Fan, and Yuke Zhu. AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents. ArXiv preprint, 2023.
  24. Jake Grigsby, Justin Sasek, Samyak Parajuli, Daniel Adebi, Amy Zhang, and Yuke Zhu. AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers. ArXiv preprint, 2024.
  25. Albert Gu and Tri Dao. Mamba: Linear-Time Sequence Mod- eling with Selective State Spaces. ArXiv preprint, 2023.
  26. Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Si- mon Schmitt, Laurent Sifre, Theophane Weber, David Sil- ver, and Hado Van Hasselt. Muesli: Combining improve- ments in policy optimization. In International conference on machine learning, 2021.
  27. Sili Huang, Jifeng Hu, Hechang Chen, Lichao Sun, and Bo Yang. In-Context Decision Transformer: Reinforce- ment Learning via Hierarchical Chain-of-Thought. ArXiv preprint, 2024.
  28. Sili Huang, Jifeng Hu, Zhejian Yang, Liwei Yang, Tao Luo, Hechang Chen, Lichao Sun, and Bo Yang. Decision Mamba: Reinforcement Learning via Hybrid Selective Se- quence Modeling. ArXiv preprint, 2024.
  29. Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. ArXiv preprint, 2017.
  30. Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A Survey of Zero-shot Generalisation in Deep Reinforcement Learning. ArXiv preprint, 2021.
  31. Louis Kirsch, James Harrison, C. Daniel Freeman, Jascha Sohl-Dickstein, and Jürgen Schmidhuber. Towards General-Purpose In-Context Learning Agents. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
  32. Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language mod- els explore in-context? ArXiv preprint, 2024.
  33. Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan A. Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In Proceedings of the International Conference on Learning Representations, 2023.
  34. Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Super- vised pretraining can learn in-context reinforcement learn- ing. In Advances in Neural Information Processing Sys- tems, 2023.
  35. Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforce- ment learning. ArXiv preprint, 2019.
  36. Licong Lin, Yu Bai, and Song Mei. Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining. ArXiv preprint, 2023.
  37. Hao Liu and Pieter Abbeel. Emergent agentic transformer from chain of hindsight experience. In Proceedings of the International Conference on Machine Learning, 2023.
  38. Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal- conditioned reinforcement learning: Problems and solu- tions. In Proceedings of the International Joint Conference on Artificial Intelligence, 2022.
  39. Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob N. Foerster, Satinder Singh, and Feryal M. P. Be- hbahani. Structured state space models for in-context re- inforcement learning. In Advances in Neural Information Processing Systems, 2023.
  40. Luckeciano C. Melo. Transformers are meta-reinforcement learners. In Proceedings of the International Conference on Machine Learning, 2022.
  41. Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In Pro- ceedings of the International Conference on Learning Rep- resentations, 2018.
  42. Ben Norman and Jeff Clune. First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs. ArXiv preprint, 2023.
  43. Chanwoo Park, Xiangyu Liu, Asuman Ozdaglar, and Kaiqing Zhang. Do LLM Agents Have Regret? A Case Study in Online Learning and Games. ArXiv preprint, 2024.
  44. Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In Proceed- ings of the International Conference on Machine Learning, 2019.
  45. Sharath Chandra Raparthy, Eric Hambro, Robert Kirk, Mikael Henaff, and Roberta Raileanu. Generalization to New Sequential Decision Making Tasks with In-Context Learning. ArXiv preprint, 2023.
  46. Samuel Ritter, Jane X. Wang, Zeb Kurth-Nelson, Sid- dhant M. Jayakumar, Charles Blundell, Razvan Pascanu, and Matthew Botvinick. Been there, done that: Meta- learning with episodic recall. In Proceedings of the In- ternational Conference on Machine Learning, 2018.
  47. Thomas Schmied, Thomas Adler, Vihang Patil, Maximilian Beck, Korbinian Pöppel, Johannes Brandstetter, Günter Klambauer, Razvan Pascanu, and Sepp Hochreiter. A Large Recurrent Action Model: xLSTM enables Fast In- ference for Robotics Tasks. ArXiv preprint, 2024.
  48. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization al- gorithms. ArXiv preprint, 2017.
  49. Lucy Xiaoyang Shi, Yunfan Jiang, Jake Grigsby, Linxi Fan, and Yuke Zhu. Cross-episodic curriculum for transformer agents. In Advances in Neural Information Processing Sys- tems, 2023.
  50. Chengshuai Shi, Kun Yang, Jing Yang, and Cong Shen. Transformers as Game Players: Provable In-context Game- playing Capabilities of Pre-trained Models. ArXiv preprint, 2024.
  51. Viacheslav Sinii, Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, and Sergey Kolesnikov. In-Context Rein- forcement Learning for Variable Action Spaces. ArXiv preprint, 2023.
  52. Jimmy T. H. Smith, Andrew Warrington, and Scott W. Lin- derman. Simplified state space layers for sequence mod- eling. In Proceedings of the International Conference on Learning Representations, 2023.
  53. Bradly C. Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some Considerations on Learning to Explore via Meta- Reinforcement Learning. ArXiv preprint, 2019.
  54. Richard S Sutton and Andrew G Barto. Reinforcement Learn- ing: An Introduction (2nd Edition). MIT press, 2018.
  55. Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Proceedings of the International Conference on Intelligent Robots and Systems, 2012.
  56. Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hu- bert Soyer, Joel Z. Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to rein- forcement learn. ArXiv preprint, 2016.
  57. Jiuqi Wang, Ethan Blaser, Hadi Daneshmand, and Shangtong Zhang. Transformers Learn Temporal Difference Methods for In-Context Reinforcement Learning. ArXiv preprint, 2024.
  58. Zhe Wang, Haozhu Wang, and Yanjun Qi. Hierarchical Prompt Decision Transformer: Improving Few-Shot Pol- icy Generalization with Global and Adaptive Guidance. ArXiv preprint, 2024.
  59. Muning Wen, Runji Lin, Hanjing Wang, Yaodong Yang, Ying Wen, Luo Mai, Jun Wang, Haifeng Zhang, and Weinan Zhang. Large Sequence Models for Sequential Decision- Making: A Survey. ArXiv preprint, 2023.
  60. Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua B. Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In Proceedings of the International Conference on Machine Learning, 2022.
  61. Tengye Xu, Zihao Li, and Qinyuan Ren. Meta-Reinforcement Learning Robust to Distributional Shift Via Performing Lifelong In-Context Learning. In Proceedings of the In- ternational Conference on Machine Learning, 2024.
  62. Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Re- inforcement Learning. ArXiv preprint, 2021.
  63. Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language mod- els. Transactions of the Association for Computational Linguistics, 2024.
  64. Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Se- bastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes- adaptive deep RL via meta-learning. In Proceedings of the International Conference on Learning Representations, 2020.
  65. Ilya Zisman, Vladislav Kurenkov, Alexander Nikulin, Viach- eslav Sinii, and Sergey Kolesnikov. Emergence of In- Context Reinforcement Learning from Noise Distillation. ArXiv preprint, 2023.
  66. Ilya Zisman, Alexander Nikulin, Andrei Polubarov, Nikita Lyubaykin, and Vladislav Kurenkov. N-Gram Induction Heads for In-Context RL: Improving Stability and Reduc- ing Data Needs. ArXiv preprint, 2024.

Cited by

Sorted by citations

Authors
Moeini, A., Wang, J., Beck, J., Blaser, E., Whiteson, S., Chandra, R., & Zhang, S.
Affiliations
  • University of Virginia (US)
  • University of Oxford (GB)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2502.07978
arXiv
2502.07978
OpenAlex
W4407569677
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
National Science Foundation (2331904, 1842490, 2128019)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
contextual conditioning, reinforcement learning agents

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review