When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 13 of the 85 sources its claims cite were checked for support.
Abstract
Reinforcement learning (RL) algorithms face two distinct challenges: learning effective representations of past and present observations, and determining how actions influence future returns. Both challenges involve modeling long-term dependencies. The Transformer architecture has been very successful to solve problems that involve long-term dependencies, including in the RL domain. However, the underlying reason for the strong performance of Transformer-based RL methods remains unclear: is it because they learn effective memory, or because they perform effective credit assignment? After introducing formal definitions of memory length and credit assignment length, we design simple configurable tasks to measure these distinct quantities. Our empirical results reveal that Transformers can enhance the memory capability of RL algorithms, scaling up to tasks that require memorizing observations $1500$ steps ago. However, Transformers do not improve long-term credit assignment. In summary, our results provide an explanation for the success of Transformers in RL, while also highlighting an important area for future research and benchmark design. Our code is open-sourced at https://github.com/twni2016/Memory-RL
Verification finished with gaps
Some checks didn’t finish. The results below cover the steps that did.
- Registry recordVerified
- Full textRead the full text
- Reference list49 references found
- Reference check49 of 49 references checked
- StatisticsRecomputed
- Claims137 claims anchored to a quote
- Cited sourcesChecked
- Self-checkDidn’t finish — the rest of the report still applies.
- VerificationFindings verified
Overview
CoChat Summary
TL;DR
Transformers in online model-free RL substantially improve long-term memory but do not improve long-term credit assignment. They solve memory tasks up to 1500 steps, yet fail credit-assignment tasks at length 250 and are often less sample-efficient on short-term POMDPs.
Key findings
- Transformer-based agents consistently solve Passive T-Maze memory lengths from 50 up to 1500; LSTM-based agents start to falter at memory length 250, underperforming optimal Markovian policies with return 0.5.
- Transformers do not enhance long-term credit assignment: both Transformer- and LSTM-based agents fail Active T-Maze and Key-to-Door when credit assignment length extends to 250. Multi-layer Transformers help for credit assignment lengths up to 100 in Active T-Maze and 120 in Key-to-Door, but not long-term.
- In Passive Visual Match, Transformer agents slightly outperform LSTMs when memory length exceeds 120, but success rates remain close to the optimal Markovian policy around 1/3; they achieve higher returns with more apples collected.
- On PyBullet short-term POMDP benchmarks, Transformer-based agents show worse sample efficiency than LSTM-based agents in most tasks over 1.5M steps.
Methods
- The paper introduces formal definitions of memory lengths (reward, transition, policy, value) and credit assignment length, and proves that optimal policy memory length is upper bounded by the maximum of reward and transition memory lengths (Theorem 1).
- It designs configurable tasks to decouple memory from credit assignment: Passive T-Maze for pure long-term memory, Active T-Maze for memory plus credit assignment, pixel-based Passive Visual Match and Key-to-Door, and standard PyBullet occlusion benchmarks for short-term dependencies.
- Agents compare GPT-2 Transformer and LSTM sequence models with the same hidden size 128 and layers varied over 1, 2, 4; context length is full horizon except PyBullet uses 64. RL algorithms are DDQN for T-Maze, SAC-Discrete for pixel tasks, and TD3 for PyBullet; results average 10 seeds.
Limitations
- The paper states that developing programs to calculate memory and credit assignment lengths for any task remains open; worst-case complexity of \(c_M\) can be \(O(|S||A|T^2)\) for a finite MDP, while \(m_M\) can be \(O((|O|+|A|)T |O||A|T)\) for a finite POMDP.
- The paper states that \(c_M\) does not account for variance of intermediate rewards, and \(m_M\) does not account for memory capacity or robustness to observation noise.
- The paper states empirical evaluation is limited to a specific relatively small GPT-2 architecture in online model-free RL; the text also makes evident that many prior-task length estimates in Table 1 are manual and some exact lengths are hard to quantify.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures












Verification
Claims the paper makes
10 Valid · 6 Not valid · 53 Not checked · 68 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Since rewards are relatively immediate with respect to the agent's actions, the credit assignment length of optimal policy is upper bounded by the length of the shortest path needed to traverse the grid-world, i.e. 7 + 11 = 18.”
§ When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment > MiniGrid-Memory (Chevalier-Boisvert et al., 2018).
What the sources say
- Partly supportedcites Hung et al., 2018 sourceExperimental — accuracy not yet measured
The Results (Episodic RL; page unavailable) say that Passive Visual Match does not require long-term credit assignment because actions in the final phase precede reward by a short interval. Our copy does not give an 18-step upper bound, a 7 + 11 calculation, or a shortest grid-world path for this task. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The credit assignment length is thus 1.”
§ When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment > Memory Cards (Esslinger et al., 2022). The task can be viewed as a discrete version of Numpad.
What the sources say
- Partly supportedcites Esslinger et al., 2022 sourceExperimental — accuracy not yet measured
Appendix B.3 (Memory cards; page not provided) says each guess receives an immediate reward: 0 if correct and −1 if incorrect. That supports a one-step reward delay, but the paper does not define or state that the task’s credit assignment length is exactly 1. (LLM, conf=0.77)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, some tasks designed for long-term memory (Beck et al., 2019; Osband et al., 2020; Hung et al., 2018; Yang and Nguyen, 2021; Esslinger et al., 2022) or credit assignment (Osband et al., 2020) indeed solely evaluate the respective capability.”
§ When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment > 3 Environment Analysis of Memory and Credit Assignment Lengths
What the sources say
- Not matchedcites Beck et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not matchedcites Osband et al., 2020
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Hung et al., 2018 source
The Results section (Episodic RL; page unavailable) describes Passive Visual Match as a task with a long delay and memory dependence but without long-term credit assignment. The Results section (The Reconstructive Memory Agent; page unavailable) adds that solving it requires encoding and retrieving a memory, not long-term credit assignment. This supports Hung et al. as an example of a task that isolates long-term memory. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Nevertheless, the optimal policy memory length can be much longer than O(N 2 ) steps and close to the horizon T .”
§ When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment > Memory Cards (Esslinger et al., 2022). The task can be viewed as a discrete version of Numpad.
What the sources say
- Partly supportedcites Pasukonis et al., 2022 sourceExperimental — accuracy not yet measured
Section 3.1 (page unavailable) says the maze layout and object locations remain unchanged throughout an episode, and the Agent Inputs section says agents revisit objects and must memorize their positions. This supports a need for long-term memory, but the paper does not state that an optimal policy needs memory much longer than O(N²) steps or close to the horizon T. (LLM, conf=0.82)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Although their longest horizon T can be 4000, the credit assignment length is upper bounded by the size of the maze.”
§ When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment > Memory Cards (Esslinger et al., 2022). The task can be viewed as a discrete version of Numpad.
What the sources say
- Partly supportedcites Pasukonis et al., 2022 sourceExperimental — accuracy not yet measured
The introduction (under “Agent Inputs”; page unavailable) says Memory Maze is designed to isolate long-term memory from credit assignment. Section 3.1 describes rewards for reaching targets and shorter paths through the maze, but does not establish that credit-assignment length is upper bounded by maze size. The cited passages also do not substantiate the stated 4000-step horizon. (LLM, conf=0.78)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
58 more claims · 4 with their cited sources checked
Create a free account to see all 137 claims and to verify the 27 references still open.
- Probably not a substantial claim. 49
- It may state two separate claims as one. 34
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
This check didn’t finish.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Continuous control with deep reinforcement learning
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2015) · arXiv · doi:10.48550/arxiv.1509.02971 · Cited by 10,782
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550
seaborn: statistical data visualization
Waskom, M. (2021) · Journal of Open Source Software · doi:10.21105/joss.03021 · Cited by 7,310
The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat
O'Keefe, J., & Dostrovsky, J. (1971) · Brain Research · doi:10.1016/0006-8993(71)90358-1 · Cited by 6,151
MuJoCo: A physics engine for model-based control
Todorov, E., Erez, T., & Tassa, Y. (2012) · 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2012.6386109 · Cited by 4,767
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149
Addressing Function Approximation Error in Actor-Critic Methods
Fujimoto, S., van Hoof, H., & Meger, D. (2018) · arXiv · doi:10.48550/arxiv.1802.09477 · Cited by 3,650
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347
Deep Reinforcement Learning with Double Q-Learning
Van Hasselt, H., Guez, A., & Silver, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10295 · Cited by 3,318
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
The Arcade Learning Environment: An Evaluation Platform for General Agents
Bellemare, M. G., Naddaf, Y., Veness, J., & Bowling, M. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3912 · Cited by 2,161
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446
How to Construct Deep Recurrent Neural Networks
Pascanu, R., Gülçehre, Ç., Cho, K., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6026 · Cited by 228
RUDDER: Return Decomposition for Delayed RewardsPreprint
Arjona-Medina, J. A., Gillhofer, M., Widrich, M., Unterthiner, T., Brandstetter, J., & Hochreiter, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07857 · Cited by 220
Lee, D.-H., Zhang, S., Fischer, A., & Bengio, Y. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-23528-8_31 · Cited by 216
Behaviour Suite for Reinforcement LearningPreprint
Osband, I., Doron, Y., Hessel, M., Aslanides, J., Sezener, E., Saraiva, A., McKinney, K., Lattimore, T., Szepesvári, C., Singh, S. P., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.03568 · Cited by 216
Soft Actor-Critic for Discrete Action Settings
Christodoulou, P. (2019) · arXiv · doi:10.48550/arxiv.1910.07207 · Cited by 204
Stabilizing Transformers for Reinforcement Learning
Parisotto, E., Song, H. F., Rae, J. W., Pascanu, R., Gulcehre, C., Jayakumar, S. M., Jaderberg, M., Kaufman, R. L., Clark, A., Noury, S., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.06764 · Cited by 199
TransDreamer: Reinforcement Learning with Transformer World ModelsPreprint
Chen, C., Wu, Y., Yoon, J., & Ahn, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.09481 · Cited by 188
Memory-based Deep Reinforcement Learning for POMDPs
Meng, L., Gorbet, R., & Kulić, D. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636140 · Cited by 178
Transformer-based World Models Are Happy With 100k InteractionsPreprint
Robine, J., Höftmann, M., Uelwer, T., & Stefan Harmeling, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.07109 · Cited by 163
Optimizing agent behavior over long time scales by transporting value
Hung, C.-C., Lillicrap, T., Abramson, J., Wu, Y., Mirza, M., Carnevale, F., Ahuja, A., & Wayne, G. (2019) · Nature Communications · doi:10.1038/s41467-019-13073-w · Cited by 146
Markov decision processes with delays and asynchronous cost collection
Katsikopoulos, K. V., & Engelbrecht, S. E. (2003) · IEEE Transactions on Automatic Control · doi:10.1109/tac.2003.809799 · Cited by 129
A Survey on Transformers in Reinforcement Learning
Li, W., Luo, H., Lin, Z., Zhang, C., Lu, Z., & Ye, D. (2023) · Transactions on machine learning research · doi:10.48550/arxiv.2301.03044 · Cited by 100
Online Decision TransformerPreprint
Zheng, Q., Zhang, A., & Grover, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.05607 · Cited by 96
Meta reinforcement learning as task inference
Humplik, J., Galashov, A., Hasenclever, L., Ortega, P. A., Teh, Y. W., & Heess, N. (2019) · arXiv · doi:10.48550/arxiv.1905.06424 · Cited by 82
Modeling the role of working memory and episodic memory in behavioral tasks
Zilli, E. A., & Hasselmo, M. E. (2008) · Hippocampus · doi:10.1002/hipo.20382 · Cited by 77
Optimizing Agent Behavior over Long Time Scales by Transporting Value
Hung, C.-C., Lillicrap, T., Abramson, J., Wu, Y., Mirza, M., Carnevale, F., Ahuja, A., & Wayne, G. (2018) · arXiv · doi:10.48550/arxiv.1810.06721 · Cited by 75
Variational Recurrent Models for Solving Partially Observable Control Tasks
Han, D., Doya, K., & Tani, J. (2019) · Institutional Repositories DataBase (IRDB) · doi:10.48550/arxiv.1912.10703 · Cited by 61
Sparse Attentive Backtracking: Temporal CreditAssignment Through Reminding
Ke, N. R., Goyal, A., Bilaniuk, O., Binas, J., Mozer, M. C., Pal, C., & Bengio, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03702 · Cited by 58
Transformers in Reinforcement Learning: A SurveyPreprint
Agarwal, P., Rahman, A. A., St-Charles, P.-L., Prince, S. J. D., & Kahou, S. E. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.05979 · Cited by 57
Multi-Game Decision Transformers
Lee, K.-H., Nachum, O., Yang, M. (., Lee, L., Freeman, D., Guadarrama, S., Fischer, I., Xu, W., Jang, E., Michalewski, H., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2025 · Cited by 54
Towards mental time travel: a hierarchical memory for reinforcement learning agentsPreprint
Lampinen, A. K., Chan, S. C. Y., Banino, A., & Hill, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.14039 · Cited by 51
Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
Ferret, J., Marinier, R., Geist, M., & Pietquin, O. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/368 · Cited by 44
Learning Long-Term Reward Redistribution via Randomized Return DecompositionPreprint
Ren, Z., Guo, R., Zhou, Y., & Peng, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.13485 · Cited by 44
AMAGO: Scalable In-Context Reinforcement Learning for Adaptive AgentsPreprint
Grigsby, J., Fan, L., & Zhu, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.09971 · Cited by 40
Deep Transformer Q-Networks for Partially Observable Reinforcement Learning
Esslinger, K., Platt, R., & Amato, C. (2022) · arXiv · doi:10.48550/arxiv.2206.01078 · Cited by 38
Transformers are Sample Efficient World Models
Micheli, V., Alonso, E., & Fleuret, F. (2022) · arXiv · doi:10.48550/arxiv.2209.00588 · Cited by 25
Recurrent Off-policy Baselines for Memory-based Continuous Control
Yang, Z., & Nguyen, H. (2021) · arXiv · doi:10.48550/arxiv.2110.12628 · Cited by 17
Warming up recurrent neural networks to maximise reachable multistability greatly improves learning
Lambrechts, G., Geeter, F. D., Vecoven, N., Ernst, D., & Drion, G. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.07.023 · Cited by 14
POPGym: Benchmarking Partially Observable Reinforcement Learning
Morad, S., Kortvelesy, R., Bettini, M., Liwicki, S., & Prorok, A. (2023) · arXiv · doi:10.48550/arxiv.2303.01859 · Cited by 13
Synthetic Returns for Long-Term Credit Assignment
Raposo, D., Ritter, S., Santoro, A., Wayne, G., Weber, T., Botvinick, M., van Hasselt, H., & Song, F. (2021) · arXiv · doi:10.48550/arxiv.2102.12425 · Cited by 12
Warming up recurrent neural networks to maximise reachable multistability greatly improves learning
Lambrechts, G., De Geeter, F., Vecoven, N., Ernst, D., & Drion, G. (2021) · arXiv · doi:10.48550/arxiv.2106.01001 · Cited by 9
Evaluating Long-Term Memory in 3D Mazes
Pasukonis, J., Lillicrap, T., & Hafner, D. (2022) · arXiv · doi:10.48550/arxiv.2210.13383 · Cited by 3
Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning
Liu, Y., Luo, Y., Zhong, Y., Chen, X., Liu, Q., & Peng, J. (2019) · arXiv · doi:10.48550/arxiv.1905.13420
Decision Transformer: Reinforcement Learning via Sequence Modeling
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., & Mordatch, I. (2021) · arXiv · doi:10.48550/arxiv.2106.01345
Generalization of Reinforcement Learners with Working and Episodic Memory
Fortunato, M., Tan, M., Faulkner, R., Hansen, S., Badia, A. P., Buttimore, G., Deck, C., Leibo, J. Z., & Blundell, C. (2019) · arXiv · doi:10.48550/arxiv.1910.13406
Research works printed but not linked: 17
- B. Bakker. Reinforcement learning with long short-term memory. In Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada], 2001. 5, 6, 18, 19
- J. Beck, K. Ciosek, S. Devlin, S. Tschiatschek, C. Zhang, and K. Hofmann. Amrl: Aggregated memory for reinforcement learning. In International Conference on Learning Representations, 2019. 5, 10, 18
- A. B. Blankenship. Memory span: a review of the literature. Psychological Bulletin, 35(1):1, 1938.
- F. N. Dempster. Memory span: Sources of individual and developmental differences. Psychological Bulletin, 89 (1):63, 1981.
- E. Derman, G. Dalal, and S. Mannor. Acting in delayed environments with non-stationary markov policies. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. 5, 17
- B. Ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L. Luu, K. Lee, Y. Kuang, S. Jesmonth, N. J. Joshi, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu. Do as I can, not as I say: Grounding language in robotic affordances. In Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, 2022.
- M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, 2021.
- L. C. Melo. Transformers are meta-reinforcement learners. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, 2022.
- T. Mesnard, T. Weber, F. Viola, S. Thakoor, A. Saade, A. Harutyunyan, W. Dabney, T. S. Stepleton, N. Heess, A. Guez, E. Moulines, M. Hutter, L. Buesing, and R. Munos. Counterfactual credit assignment in model-free reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, 2021. 2, 10, 20
- M. Minsky. Steps toward artificial intelligence. Proceedings of the IRE, 49(1):8-30, 1961.
- T. Ni, B. Eysenbach, and R. Salakhutdinov. Recurrent model-free RL can be a strong baseline for many pomdps. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, 2022. 5, 6, 7, 9, 10, 21
- M. Pleines, M. Pallasch, F. Zimmer, and M. Preuss. Memory gym: Partially observable challenges to memory- based agents. In The Eleventh International Conference on Learning Representations, 2023. 5, 7, 20
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 1, 7, 22
- R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
- S. Thrun. Monte carlo pomdps. In Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 -December 4, 1999], 1999.
- D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021a. 21, 22
- D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pineau, and R. Fergus. Improving sample efficiency in model- free reinforcement learning from images. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, 2021b. 21, 22
Cited by
AMAGO: Scalable In-Context Reinforcement Learning for Adaptive AgentsPreprint
Grigsby, J., Fan, L., & Zhu, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.09971 · Cited by 40
Bridging State and History Representations: Understanding Self-Predictive RLPreprint
Ni, T., Eysenbach, B., Seyedsalehi, E., Ma, M., Gehring, C., Mahajan, A., & Bacon, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.08898 · Cited by 67
Self-citationFindingDory: A Benchmark to Evaluate Memory in Embodied Agents
Yadav, K., Ali, Y., Gupta, G., Gal, Y., & Kira, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37086-0_13 · Cited by 28
Large Language Models as Generalizable Policies for Embodied Tasks
Szot, A., Schwarzer, M., Agrawal, H., Mazoure, B., Talbott, W., Metcalf, K., Mackraz, N., Hjelm, D., & Toshev, A. (2023) · arXiv · doi:10.48550/arxiv.2310.17722 · Cited by 14
ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AIPreprint
Elawady, A., Chhablani, G., Ramrakhya, R., Yadav, K., Batra, D., Kira, Z., & Szot, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.02751 · Cited by 17
Metadata
- Authors
- Ni, T., Ma, M., Eysenbach, B., & Bacon, P.-L.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 50429-50452
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133216054
- Cited by
- 7
- References
- 19
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 3
- FWCI
- 1.25
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Alliance de recherche numérique du Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Keywords
- transformer models, reinforcement learning, memory, credit assignment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.