CoChat

When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment

Open access
Ni, Tianwei, Ma, Michel, Eysenbach, Benjamin, Bacon, Pierre-Luc · 2023 · Advances in Neural Information Processing Systems 36

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; 13 of the 85 sources its claims cite were checked for support.

Abstract

Reinforcement learning (RL) algorithms face two distinct challenges: learning effective representations of past and present observations, and determining how actions influence future returns. Both challenges involve modeling long-term dependencies. The Transformer architecture has been very successful to solve problems that involve long-term dependencies, including in the RL domain. However, the underlying reason for the strong performance of Transformer-based RL methods remains unclear: is it because they learn effective memory, or because they perform effective credit assignment? After introducing formal definitions of memory length and credit assignment length, we design simple configurable tasks to measure these distinct quantities. Our empirical results reveal that Transformers can enhance the memory capability of RL algorithms, scaling up to tasks that require memorizing observations $1500$ steps ago. However, Transformers do not improve long-term credit assignment. In summary, our results provide an explanation for the success of Transformers in RL, while also highlighting an important area for future research and benchmark design. Our code is open-sourced at https://github.com/twni2016/Memory-RL

Verification finished with gaps

Some checks didn’t finish. The results below cover the steps that did.

  1. Registry recordVerified
  2. Full textRead the full text
  3. Reference list49 references found
  4. Reference check49 of 49 references checked
  5. StatisticsRecomputed
  6. Claims137 claims anchored to a quote
  7. Cited sourcesChecked
  8. Self-checkDidn’t finish — the rest of the report still applies.
  9. VerificationFindings verified

Overview

CoChat Summary

TL;DR

Transformers in online model-free RL substantially improve long-term memory but do not improve long-term credit assignment. They solve memory tasks up to 1500 steps, yet fail credit-assignment tasks at length 250 and are often less sample-efficient on short-term POMDPs.

Key findings

  • Transformer-based agents consistently solve Passive T-Maze memory lengths from 50 up to 1500; LSTM-based agents start to falter at memory length 250, underperforming optimal Markovian policies with return 0.5.
  • Transformers do not enhance long-term credit assignment: both Transformer- and LSTM-based agents fail Active T-Maze and Key-to-Door when credit assignment length extends to 250. Multi-layer Transformers help for credit assignment lengths up to 100 in Active T-Maze and 120 in Key-to-Door, but not long-term.
  • In Passive Visual Match, Transformer agents slightly outperform LSTMs when memory length exceeds 120, but success rates remain close to the optimal Markovian policy around 1/3; they achieve higher returns with more apples collected.
  • On PyBullet short-term POMDP benchmarks, Transformer-based agents show worse sample efficiency than LSTM-based agents in most tasks over 1.5M steps.

Methods

  • The paper introduces formal definitions of memory lengths (reward, transition, policy, value) and credit assignment length, and proves that optimal policy memory length is upper bounded by the maximum of reward and transition memory lengths (Theorem 1).
  • It designs configurable tasks to decouple memory from credit assignment: Passive T-Maze for pure long-term memory, Active T-Maze for memory plus credit assignment, pixel-based Passive Visual Match and Key-to-Door, and standard PyBullet occlusion benchmarks for short-term dependencies.
  • Agents compare GPT-2 Transformer and LSTM sequence models with the same hidden size 128 and layers varied over 1, 2, 4; context length is full horizon except PyBullet uses 64. RL algorithms are DDQN for T-Maze, SAC-Discrete for pixel tasks, and TD3 for PyBullet; results average 10 seeds.

Limitations

  • The paper states that developing programs to calculate memory and credit assignment lengths for any task remains open; worst-case complexity of \(c_M\) can be \(O(|S||A|T^2)\) for a finite MDP, while \(m_M\) can be \(O((|O|+|A|)T |O||A|T)\) for a finite POMDP.
  • The paper states that \(c_M\) does not account for variance of intermediate rewards, and \(m_M\) does not account for memory capacity or robustness to observation noise.
  • The paper states empirical evaluation is limited to a specific relatively small GPT-2 architecture in online model-free RL; the text also makes evident that many prior-task length estimates in Table 1 are manual and some exact lengths are hard to quantify.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.52202/075280-2194

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 T-Maze task. In the passive version, the agent starts at the oracle state ('S' and 'O' are the same state), while the active version requires that the agent navigate left from the initial state to collect information from the oracle state.
Figure 1 T-Maze task. In the passive version, the agent starts at the oracle state ('S' and 'O' are the same state), while the active version requires that the agent navigate left from the initial state to collect information from the oracle state.CC-BY · From the PDF
Figure 2 Pixel-based tasks, Passive Visual Match (left) and Key-to-Door (right), evaluated in our experiments. Each task has three phases, where we adjust the length of Phase 2 and keep the lengths of the others constant. The agent (in beige) only observes nearby grids (within white borders) and cannot pass through the walls (in black). In Passive Visual Match, the agent observes a randomized target color (blue in this case) in Phase 1, collects apples (in green) in Phase 2, and reaches the grid with the matching color in Phase 3. In Key-to-Door, the agent reaches the key (in brown) in Phase 1, collects apples in Phase 2, and reaches the door (in cyan) in Phase 3.
Figure 2 Pixel-based tasks, Passive Visual Match (left) and Key-to-Door (right), evaluated in our experiments. Each task has three phases, where we adjust the length of Phase 2 and keep the lengths of the others constant. The agent (in beige) only observes nearby grids (within white borders) and cannot pass through the walls (in black). In Passive Visual Match, the agent observes a randomized target color (blue in this case) in Phase 1, collects apples (in green) in Phase 2, and reaches the grid with the matching color in Phase 3. In Key-to-Door, the agent reaches the key (in brown) in Phase 1, collects apples in Phase 2, and reaches the door (in cyan) in Phase 3.CC-BY · From the PDF
Image 3
CC-BY · From the PDF
Figure 3 Transformer-based RL outperforms LSTM-based RL in tasks (purely) requiring long-term memory. Left: results in Passive T-Mazes with varying memory lengths from 50 to 1500 ; Right: results in Passive Visual Match with varying memory lengths from 60 to 1000 . We also show the performance of the optimal Markovian policies. Each data point in the figure represents the final performance of an agent with the memory length indicated on the x-axis. All the figures in this paper show the mean and its 95% confidence interval over 10 seeds.
Figure 3 Transformer-based RL outperforms LSTM-based RL in tasks (purely) requiring long-term memory. Left: results in Passive T-Mazes with varying memory lengths from 50 to 1500 ; Right: results in Passive Visual Match with varying memory lengths from 60 to 1000 . We also show the performance of the optimal Markovian policies. Each data point in the figure represents the final performance of an agent with the memory length indicated on the x-axis. All the figures in this paper show the mean and its 95% confidence interval over 10 seeds.CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Figure 5 Transformer-based agent is more sample-efficient in long-term memory highdimensional tasks. In Passive Visual Match with long memory length (750 on the left; 1000 on the right), Transformer-based agents show a slight advantage over LSTM-based agents in the memory capability (as measured by success rate ), while being much higher in total rewards (as measured by return ).
Figure 5 Transformer-based agent is more sample-efficient in long-term memory highdimensional tasks. In Passive Visual Match with long memory length (750 on the left; 1000 on the right), Transformer-based agents show a slight advantage over LSTM-based agents in the memory capability (as measured by success rate ), while being much higher in total rewards (as measured by return ).CC-BY · From the PDF
Image 7
CC-BY · From the PDF
Image 8
CC-BY · From the PDF
Figure 9 An MDP that shows the credit assignment length over optimal policies is not a constant. Each node is a state with its name on the top and its reward at the bottom. The initial state is P1. The action space is { x, y } . The finite horizon T ≥ 4 and γ = 1 .
Figure 9 An MDP that shows the credit assignment length over optimal policies is not a constant. Each node is a state with its name on the top and its reward at the bottom. The initial state is P1. The action space is { x, y } . The finite horizon T ≥ 4 and γ = 1 .CC-BY · From the PDF
Figure 10 Agent architectures for memory-based DDQN (left) and TD3 (right) in our implementation. While rewards are not used as inputs in our experiments, they are included in this figure for completeness. Compared to POMDP-baselines (Ni et al., 2022), we have removed shortcuts from o t to the MLPs and shared the same sequence model between the actor and critic.
Figure 10 Agent architectures for memory-based DDQN (left) and TD3 (right) in our implementation. While rewards are not used as inputs in our experiments, they are included in this figure for completeness. Compared to POMDP-baselines (Ni et al., 2022), we have removed shortcuts from o t to the MLPs and shared the same sequence model between the actor and critic.CC-BY · From the PDF
Image 11
CC-BY · From the PDF
Image 12
CC-BY · From the PDF

Verification

Claims the paper makes

10 Valid · 6 Not valid · 53 Not checked · 68 Own results and interpretations

58 more claims · 4 with their cited sources checked

Create a free account to see all 137 claims and to verify the 27 references still open.

See all claims — free account
  • Probably not a substantial claim. 49
  • It may state two separate claims as one. 34

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

This check didn’t finish.

Citations

Cites

Research works printed but not linked: 17

  1. B. Bakker. Reinforcement learning with long short-term memory. In Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada], 2001. 5, 6, 18, 19
  2. J. Beck, K. Ciosek, S. Devlin, S. Tschiatschek, C. Zhang, and K. Hofmann. Amrl: Aggregated memory for reinforcement learning. In International Conference on Learning Representations, 2019. 5, 10, 18
  3. A. B. Blankenship. Memory span: a review of the literature. Psychological Bulletin, 35(1):1, 1938.
  4. F. N. Dempster. Memory span: Sources of individual and developmental differences. Psychological Bulletin, 89 (1):63, 1981.
  5. E. Derman, G. Dalal, and S. Mannor. Acting in delayed environments with non-stationary markov policies. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. 5, 17
  6. B. Ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L. Luu, K. Lee, Y. Kuang, S. Jesmonth, N. J. Joshi, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu. Do as I can, not as I say: Grounding language in robotic affordances. In Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, 2022.
  7. M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, 2021.
  8. L. C. Melo. Transformers are meta-reinforcement learners. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, 2022.
  9. T. Mesnard, T. Weber, F. Viola, S. Thakoor, A. Saade, A. Harutyunyan, W. Dabney, T. S. Stepleton, N. Heess, A. Guez, E. Moulines, M. Hutter, L. Buesing, and R. Munos. Counterfactual credit assignment in model-free reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, 2021. 2, 10, 20
  10. M. Minsky. Steps toward artificial intelligence. Proceedings of the IRE, 49(1):8-30, 1961.
  11. T. Ni, B. Eysenbach, and R. Salakhutdinov. Recurrent model-free RL can be a strong baseline for many pomdps. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, 2022. 5, 6, 7, 9, 10, 21
  12. M. Pleines, M. Pallasch, F. Zimmer, and M. Preuss. Memory gym: Partially observable challenges to memory- based agents. In The Eleventh International Conference on Learning Representations, 2023. 5, 7, 20
  13. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 1, 7, 22
  14. R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
  15. S. Thrun. Monte carlo pomdps. In Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 -December 4, 1999], 1999.
  16. D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021a. 21, 22
  17. D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pineau, and R. Fergus. Improving sample efficiency in model- free reinforcement learning from images. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, 2021b. 21, 22

Cited by

See all 25

Metadata

Authors
Ni, T., Ma, M., Eysenbach, B., & Bacon, P.-L.
Venue
Advances in Neural Information Processing Systems 36
Published
2023
Pages
50429-50452
ISSN
1049-5258 (ISSN-L)
OpenAlex
W7133216054
Cited by
7
References
19
Citations by year
2026: 2 · 2025: 2 · 2024: 3
FWCI
1.25
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Alliance de recherche numérique du Canada
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
transformer models, reinforcement learning, memory, credit assignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.