The DOI matches its DataCite record.
Optimizing Agent Behavior over Long Time Scales by Transporting Value
Open accessAbstract
Humans spend a remarkable fraction of waking life engaged in acts of "mental time travel". We dwell on our actions in the past and experience satisfaction or regret. More than merely autobiographical storytelling, we use these event recollections to change how we will act in similar scenarios in the future. This process endows us with a computationally important ability to link actions and consequences across long spans of time, which figures prominently in addressing the problem of long-term temporal credit assignment; in artificial intelligence (AI) this is the question of how to evaluate the utility of the actions within a long-duration behavioral sequence leading to success or failure in a task. Existing approaches to shorter-term credit assignment in AI cannot solve tasks with long delays between actions and consequences. Here, we introduce a new paradigm for reinforcement learning where agents use recall of specific memories to credit actions from the past, allowing them to solve problems that are intractable for existing algorithms. This paradigm broadens the scope of problems that can be investigated in AI and offers a mechanistic account of behaviors that may inspire computational models in neuroscience, psychology, and behavioral economics.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Passive Visual Match is a partially observable 7×11 grid-world where the agent observes a 5×5 grid surrounding itself.
“The environment consists of a partially observable 7 × 11 grid-world, where the agent can only observe a 5×5 grid surrounding itself.”
From When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment (2023) · cites this paper as Hung et al., 2018
In Passive Visual Match, the credit assignment length of an optimal policy is upper bounded by the length of the shortest path needed to traverse the grid-world, i.e. 18.
“Since rewards are relatively immediate with respect to the agent's actions, the credit assignment length of optimal policy is upper bounded by the length of the shortest path needed to traverse the grid-world, i.e. 7 + 11 = 18.”
From When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment (2023) · cites this paper as Hung et al., 2018
Some tasks designed for long-term memory or credit assignment indeed solely evaluate the respective capability.
“First, some tasks designed for long-term memory (Beck et al., 2019; Osband et al., 2020; Hung et al., 2018; Yang and Nguyen, 2021; Esslinger et al., 2022) or credit assignment (Osband et al., 2020) indeed solely evaluate the respective capability.”
From When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment (2023) · cites this paper as Hung et al., 2018
Active Visual Match requires the policy to memorize the color.
“Thus, the policy needs to memorize the color.”
From When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment (2023) · cites this paper as Hung et al., 2018
Citations
Cited by
Sorted by citations
When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment
Ni, T., Ma, M., Eysenbach, B., & Bacon, P.-L. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2194 · Cited by 7
Metadata
- Authors
- Hung, C.-C., Lillicrap, T., Abramson, J., Wu, Y., Mirza, M., Carnevale, F., Ahuja, A., & Wayne, G.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- Corresponding author
- Wayne, Greg
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Volume
- 10
- Issue
- 1
- Pages
- 5223-5223
- ISSN
- 2041-1723 (ISSN-L)
- arXiv
- 1810.06721
- PMC
- PMC6864102
- OpenAlex
- W2897023205
- Cited by
- 75
- References
- 0
- Citations by year
- 2026: 7 · 2025: 2 · 2024: 4 · 2023: 9 · 2022: 6 · 2021: 21 · 2020: 15 · 2019: 10 · 2018: 1
- FWCI
- 5.87
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Nature Communications (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- mental time travel, memory retrieval, delayed rewards, temporal credit assignment, computational model of memory
- MeSH terms
- Algorithms, Artificial Intelligence, Humans, Learning, Mental Processes, Models, Psychological, Problem Solving, Reinforcement, Psychology, Transfer, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
