Emergence of belief-like representations through reinforcement learning
Open accessThe DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index; 17 of the 61 sources its claims cite were checked for support.
Abstract
To behave adaptively, animals must learn to predict future reward, or value. To do this, animals are thought to learn reward predictions using reinforcement learning. However, in contrast to classical models, animals must learn to estimate value using only incomplete state information. Previous work suggests that animals estimate value in partially observable tasks by first forming “beliefs”—optimal Bayesian estimates of the hidden states in the task. Although this is one way to solve the problem of partial observability, it is not the only way, nor is it the most computationally scalable solution in complex, real-world environments. Here we show that a recurrent neural network (RNN) can learn to estimate value directly from observations, generating reward prediction errors that resemble those observed experimentally, without any explicit objective of estimating beliefs. We integrate statistical, functional, and dynamical systems perspectives on beliefs to show that the RNN’s learned representation encodes belief information, but only when the RNN’s capacity is sufficiently large. These results illustrate how animals can estimate value in tasks without explicitly estimating beliefs, yielding a representation useful for systems with limited capacity.
Overview
CoChat Summary
TL;DR
RNNs trained by TD learning to estimate value in partially observable associative learning tasks develop representations that resemble Bayesian beliefs, even though they are never trained to estimate beliefs. Belief-like encoding and decoding emerge with sufficient network capacity, while belief-like dynamics distinguish trained RNNs from untrained or reservoir networks.
Key findings
- Value RNNs reproduced experimentally observed dopamine RPE patterns and matched Belief model RPEs after training; this held for N = 12 networks per Starkweather task and for N = 12 Value RNNs on the Babayan task, including intermediate reward sizes.
- Linear readouts of trained RNN activity explained most belief variance: Task 1 R² = 0.61 ± 0.01 and Task 2 R² = 0.67 ± 0.02 (mean ± SE, N = 12), versus untrained RNNs at 0.38 ± 0.01 and 0.41 ± 0.00. State-decoding log-likelihoods also approached those of beliefs and exceeded untrained RNNs.
- Value RNNs with as few as 2 hidden units could estimate value, but belief-like linear encoding and state decoding increased with hidden-unit number. The dynamics summary statistic—odor memory longer than reward memory—was positive for every trained RNN across hidden-unit counts, but not for untrained RNNs.
- Value ESNs could estimate value and support belief readout or state decoding when appropriately initialized, but their odor and reward memories were nearly identical (e.g., gain 1.9 odor memory around 200 time steps versus Task 2 beliefs returning after about 15 time steps), unlike trained Value RNNs.
Methods
- Tasks: Starkweather Tasks 1 and 2 and the Babayan task, modeled as discrete-time Markov/microstate processes with 200 ms time bins and observations of odor, reward, or null. Task 2 omitted reward on 10% of trials; the Babayan task used hidden blocks of 5 trials with reward sizes of 1 versus 10.
- Models: a Belief model (Bayesian belief update plus TD learning), a Value RNN (GRU with H hidden units and linear value readout trained by TD with backpropagation through time), an Untrained RNN, and a Value ESN (frozen random reservoir with only readout trained). Hidden-unit counts included H ∈ {2, 5, 10, 20, 50, 100}, with N = 12 networks per task.
- Analyses: value weights fit by least-squares TD; RPE mean squared error versus the Belief model; multivariate linear regression for belief R²; multinomial logistic regression for state decoding log-likelihood; dynamical-systems analyses of fixed points and odor/reward memory durations.
Limitations
- Stated: the tasks have simple state spaces and action-independent trial structure, so value is linear in beliefs; future work is needed for larger state spaces and tasks where policy influences visited states. The RNN is a generic single-layer model, not a model of individual brain regions, and cannot distinguish different sources of uncertainty.
- Stated: the Belief model assumes known transition and observation dynamics; the Value RNN avoids this, but the paper does not test multi-task generalization or other training objectives such as next-observation prediction.
- Evident: conclusions about biological belief representations are indirect because the models are not fit to neural recordings, and the findings are limited to the specific tasks and model classes analyzed.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig 1 Associative learning tasks with probabilistic rewards and hidden states. A. Trial structure in Starkweather et al. (2017) [7]. Each trial consisted of a variable delay (the intertrial interval, or ITI), followed by an odor, a second delay (the interstimulus interval, or ISI), and a potential subsequent reward. Reward times were sampled from a discretized Gaussian ranging from 1.2–2.8s (see Materials and methods). B-C. In both versions of the task, there were two underlying states: the ITI and the ISI. In Task 1, every trial was rewarded. As a result, an odor always indicated a transition from the ITI to the ISI, while a reward always indicated a transition from the ISI to the ITI. In Task 2, rewards were omitted on 10% of trials; as a result, an odor did not reveal whether or not the state transitioned to the ISI.](https://figures.cochat.ai/paper-figures/aa0d8321890048e19e76a5614ad0a8b6/emergence-of-belief-like-representations-through-reinforcement-learning-figure-1.jpg)

![Fig 3 RPEs of the Value RNN resemble both mouse dopamine activity and the Belief model. A. Average phasic dopamine activity in the ventral tegmental area (VTA) recorded from mice trained in each task separately. Black traces indicate trial-averaged RPEs relative to an odor observated at time 0, prior to reward; colored traces indicate the RPEs following each of nine possible reward times. RPEs exhibit opposite dependence on reward time across tasks. Reproduced from Starkweather et al. (2017) [7]. B-C. Average RPEs of the Belief model and an example Value RNN, respectively. Same conventions as panel A. D. Mean squared error (MSE) between the RPEs of the Value RNN and Belief model, before and after training each Value RNN. Small dots depict the MSE of each of N = 12 Task 1 RNNs and N = 12 Task 2 RNNs, and circles depict the median across RNNs.](https://figures.cochat.ai/paper-figures/ecbd52515bc6421abd880ba45d388fbf/emergence-of-belief-like-representations-through-reinforcement-learning-figure-3.jpg)



![Fig 7 Value RNNs trained on Babayan et al. (2018) [10] reproduce Belief RPEs and learn belief-like representations. A. Task environment of Babayan et al. (2018) [10]. Each trial consists of an odor and a subsequent reward. The reward amount depends on the block identity, which is resampled uniformly every five trials. B. Average phasic dopamine activity in the VTA of mice trained on the task at the time of odor (left) and reward (right) delivery. Activity is shown separately as a function of the trial index within the block (x-axis) and the current/previous block identity (colors). Reproduced from Babayan et al. (2018) [10]. C. Average RPEs of the Belief model (dashed lines) and an example Value RNN (solid lines). Same conventions as panel B. D. Total variance of beliefs explained (R2) using a linear transformation of model activity. Same conventions as Fig 4B. E. Cross-validated log-likelihood of linear decoders trained to estimate true states using RNN activity. Same conventions as Fig 4C. F. Dynamics of beliefs in the absence of observations. Same conventions as Fig 5A. G. Trajectories of an example Value RNN’s activity, in the 2D subspace identified using PCA, during an example trial from Block 1 (left) and Block 2 (right). These two dimensions explained 68% of the total variance in the Value RNN’s activity across trials. Putative ITI states indicated as purple circles. Same conventions as Fig 5B.](https://figures.cochat.ai/paper-figures/ba46de7a202a44c8b1acec5b7181f01a/emergence-of-belief-like-representations-through-reinforcement-learning-figure-7.jpg)

Verification
Claims the paper makes
10 Valid · 4 Not valid · 36 Not checked · 118 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“We first show that RNNs can be trained to optimally estimate value in two previously studied associative learning tasks [7,10], and that the RPEs of these models resemble experimentally observed dopamine neuron activity.”
§ Emergence of belief-like representations through reinforcement learning > Introduction
What the sources say
- Supportedcites [7] CK Starkweather, 2017, Dopamine reward prediction errors reflect hidden-state inference across time, Nature neuroscience source
The Introduction describes two associative learning tasks differing in reward delivery (100% versus 90%), and the Methods report training a value-learning model on each. The citation identifies the previously studied tasks; the citing authors’ statement that they trained RNNs is their own result, not a claim attributed to this source. (LLM, conf=0.89)
- Supportedcites [10] BM Babayan, 2018, Belief state representation in the dopamine system, Nature communications source
The citation identifies a previously studied associative learning task, not the citing authors’ RNN training result. The cited paper’s Results (Fig. 1 task design) describe mice trained on cue–reward trials and measurements of dopaminergic reward prediction errors, so it supports its role as one of the two tasks. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“One way of understanding this is to note that the value of an observation may depend on the longterm past [22].”
§ Emergence of belief-like representations through reinforcement learning > Reinforcement learning in partially observable environments using belief states
What the sources say
- Supportedcites [22] SJ Gershman, 2017, Reinforcement learning and episodic memory in humans and animals: an integrative framework, Annual Review of Psychology source
The 'Computational shortcomings of the current picture' section states that immediate sensory observations 'rarely satisfy the Markov property; i.e., other information observed in the past but not currently observable affects future state and reward expectancies,' including 'long-run dependencies between day-to-day events.' The 'Overcoming state aliasing' section reinforces this: when states are indistinguishable on the basis of the current observation, 'the value is not conditionally independent of the agent's history given the current observation,' and outcomes following a state can 'depend on events that happened prior to' it, with remembered trajectories capturing 'arbitrary long-range, non-Markovian dependencies.' The hedged phrasing ('may depend') matches the source's own framing. (LLM, conf=0.88)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“RPEs exhibit opposite dependence on reward time across tasks.”
§ Emergence of belief-like representations through reinforcement learning > Odor and reward memory duration (Fig 5C and 5D).
What the sources say
- Supportedcites Starkweather et al. (2017) [7] source
The Results section, “Dopamine RPEs show opposing patterns of modulation across time in Tasks 1 and 2,” reports opposite effects of reward timing on post-reward dopamine firing: negative modulation in Task 1 and positive modulation in Task 2. The distinction is specifically about post-reward responses; pre-reward firing declined over time in both tasks. (LLM, conf=0.93)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Average phasic dopamine activity in the VTA of mice trained on the task at the time of odor (left) and reward (right) delivery.Activity is shown separately as a function of the trial index within the block (x-axis) and the current/previous block identity (colors).”
§ Emergence of belief-like representations through reinforcement learning > Odor and reward memory duration (Fig 5C and 5D).
What the sources say
- Supportedcites Babayan et al. (2018) [10] source
The Results section (“Behavior and dopamine neuron activity on training blocks,” Fig. 2; page not provided) shows odor and reward responses across trials and reports effects of previous and current block identity, particularly on trial 1. The Methods (“Data analysis”) describes mean activity measured after odor and reward onset, and the Results (“Testing prediction error modulation by belief state”) identifies recordings from dopamine neurons in the VTA. (LLM, conf=0.91)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We first show that RNNs can be trained to optimally estimate value in two previously studied associative learning tasks [7,10], and that the RPEs of these models resemble experimentally observed dopamine neuron activity.”
§ Emergence of belief-like representations through reinforcement learning > Introduction
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
95 more claims
Create a free account to see all 168 claims and to verify the 44 references still open.
- Probably not a substantial claim. 41
- It may state two separate claims as one. 33
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Emergence of belief-like representations through reinforcement learning > Reward prediction errors.
“For the Babayan task (Fig 5E), we calculated the log-likelihood on all trials except the first trial in each block.”
§ Emergence of belief-like representations through reinforcement learning > Odor and reward memory duration (Fig 5C and 5D).
“E. Difference between each RNN's odor memory and reward memory, for Untrained RNNs and Value RNNs trained on each task.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Emergence of belief-like representations through reinforcement learning > Generalization to other tasks
“We found that Value RNNs exhibited nearly identical RPEs as the Belief model (Fig 7D).”
§ Emergence of belief-like representations through reinforcement learning > Odor and reward memory duration (Fig 5C and 5D).
“C. Average RPEs of the Belief model (dashed lines) and an example Value RNN (solid lines).Same conventions as panel B. D. Total variance of beliefs explained (R 2 ) using a linear transformation of model activity.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Emergence of belief-like representations through reinforcement learning > Generalization to other tasks
“Previous work showed that the dopamine activity of animals trained on this task depended on the number of trials in the current block (Fig 7C)”
§ Emergence of belief-like representations through reinforcement learning > Odor and reward memory duration (Fig 5C and 5D).
“B. Average phasic dopamine activity in the VTA of mice trained on the task at the time of odor (left) and reward (right) delivery.Activity is shown separately as a function of the trial index within the block (x-axis) and the current/previous block identity (colors).Reproduced from Babayan et al. (2018) [10].C. Average RPEs of the Belief model”
Citations
Cites
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 2×
“For each task, we trained the Belief model, and a Value RNN (using 141 a gated-recurrent unit cell [24], or GRU, with H = 50 hidden units), on a series of observations from that 142 task to estimate value using TD learning (see Materials and Methods).”
Method(2010) · Cited by 14,134
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · cited 1×
“In particular, one successful theoretical model posits that associative learning is driven by the activity of dopamine neurons in the midbrain, where spiking activity resembles the reward prediction error (RPE) signal in an RL algorithm called temporal difference (TD) learning [1–4].”
BackgroundReinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
Colman (2022) · doi:10.1073/pnas · Cited by 4,370
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · cited 2×
“One standard answer is the posterior probability distribution over hidden states given the history of observations and actions, also known as the belief state [5]: which stipulates how to update the belief in state i given observation o t , and action a t − 1 .”
BackgroundMethodReservoir computing approaches to recurrent neural network training
Lukoševičius, M., & Jaeger, H. (2009) · Computer Science Review · doi:10.1016/j.cosrev.2009.03.005 · Cited by 2,956
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
Hochreiter, S. (1998) · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems · doi:10.1142/s0218488598000094 · Cited by 2,939 · cited 1×
“We suspect that the gate-less RNNs struggled to converge due to the vanishing gradient problem, which is a known issue with these networks in the presence of long time lags [47].”
BackgroundDaw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · cited 1×
“Indeed, there are well-documented examples of animals failing to learn or use the correct environment model [16].”
BackgroundNeuron-type-specific signals for reward and punishment in the ventral tegmental area
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B., & Uchida, N. (2012) · Nature · doi:10.1038/nature10754 · Cited by 1,318
Self-citationMidbrain dopamine neurons encode a quantitative reward prediction error signal
Bayer, H. M., & Glimcher, P. W. (2005) · Neuron · doi:10.1016/j.neuron.2005.05.020 · Cited by 1,304 · cited 1×
“In particular, one successful theoretical model 33 posits that associative learning is driven by the activity of dopamine neurons in the midbrain, where spiking 34 activity resembles the reward prediction error (RPE) signal in an RL algorithm called temporal difference 35 (TD) learning [1, 2, 3, 4].”
BackgroundHippocampal ‘time cells’ bridge the gap in memory for discontiguous events
MacDonald, C. J., Lepage, K. Q., Eden, U. T., & Eichenbaum, H. (2011) · Neuron · doi:10.1016/j.neuron.2011.07.012 · Cited by 1,193 · cited 1×
“Similarly tuned “time cells” have 363 been observed in the striatum [35], hippocampus [36], and prefrontal cortex [37].”
BackgroundExact solutions to the nonlinear dynamics of learning in deep linear neural networks
Saxe, A., McClelland, J. L., & Ganguli, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6120 · Cited by 931 · cited 3×
“In agreement with previous work, we 307 found that when the gain was above a critical value (“the edge of chaos” [30]), the network’s activity never 308 decayed back to baseline (e.”
BackgroundMethodInfluentialOrbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885 · cited 2×
“The brain may have dedicated machinery, perhaps 49 in prefrontal cortex [12, 13], for computing belief states, which could then be provided to downstream areas, 50”
BackgroundComputation Through Neural Population Dynamics
Vyas, S., Golub, M. D., Sussillo, D., & Shenoy, K. V. (2020) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-092619-094115 · Cited by 671 · cited 1×
“Moreover, there is a growing toolkit for reverse engineering RNN solutions [25, 26, 38], which can shed light on learned mechanisms of value computation.”
BackgroundProbabilistic brains: knowns and unknowns
Pouget, A., Beck, J. M., Ma, W. J., & Latham, P. E. (2013) · Nature Neuroscience · doi:10.1038/nn.3495 · Cited by 615 · cited 1×
“This architectural division of labor resonates with the broader literature on probabilistic computation in cortex, which has identified several different ways in which belief states could be encoded by neural activity [15].”
BackgroundPredictive Representations of State
Littman, M. L., & Sutton, R. S. (2001) · neural information processing systems · Cited by 600
Prefrontal cortex as a meta-reinforcement learning system
Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., & Botvinick, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0147-8 · Cited by 596 · cited 1×
“…the intersection of neuroscience and machine learning has shown how agents performing reinforcement learning can use RNNs to solve a variety of tasks featuring state and task uncertainty, including tasks involving motor control [30], navigation [31–33], foraging [34], and decision-making [35, 36].”
BackgroundOpening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks
Sussillo, D., & Barak, O. (2013) · Neural Computation · doi:10.1162/neco_a_00409 · Cited by 575
Jaeger, H. (2007) · Scholarpedia · doi:10.4249/scholarpedia.2330 · Cited by 459
Shen, W., Plotkin, J. L., Francardo, V., Ko, W. K. D., Xie, Z., Li, Q., Fieblinger, T., Wess, J., Neubig, R. R., Lindsley, C. W., et al. (2015) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.1016/j.neuron · Cited by 452
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446 · cited 1×
“One way of understanding this is to note that the value of an observation may depend on the long-term past [22].”
BackgroundSelf-citationThe Neural Basis of Timing: Distributed Mechanisms for Diverse Functions
Paton, J. J., & Buonomano, D. V. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.045 · Cited by 442 · cited 1×
“This viewpoint resonates with the idea (reviewed in [45]) that delay encoding can arise as an emergent property of neural network dynamics.”
BackgroundA Scalable Population Code for Time in the Striatum
Mello, G. B. M., Soares, S., & Paton, J. J. (2015) · Current Biology · doi:10.1016/j.cub.2015.02.036 · Cited by 392 · cited 1×
“Similarly tuned “time cells” have been observed in the striatum [42], hippocampus [43], and prefrontal cortex [44].”
BackgroundArithmetic and local circuitry underlying dopamine prediction errors
Eshel, N., Bukwich, M., Rao, V., Hemmelder, V., Tian, J., & Uchida, N. (2015) · Nature · doi:10.1038/nature14855 · Cited by 392 · cited 1×
“In particular, one successful theoretical model 33 posits that associative learning is driven by the activity of dopamine neurons in the midbrain, where spiking 34 activity resembles the reward prediction error (RPE) signal in an RL algorithm called temporal difference 35 (TD) learning [1, 2, 3, 4].”
BackgroundSelf-citationDeep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269 · cited 1×
“In deep RL, rather than explicitly learning beliefs, an agent uses nonlinear function approximation to learn a hidden representation that is sufficient for performing the task [20].”
BackgroundFinding approximate POMDP solutions through belief compression
Roy, N., Gordon, G., & Thrun, S. (2005) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1496 · Cited by 254 · cited 1×
“This suggests that there may be other representations that are sufficient for the particular task of estimating future reward, but easier to compute than the full belief state [18, 19].”
BackgroundGershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236 · cited 1×
“The brain may have dedicated machinery, perhaps in prefrontal cortex [12, 13], for computing belief states, which could then be provided to downstream areas, such as the basal ganglia, to perform standard RL algorithms such as TD learning [14].”
Self-citationDopamine reward prediction errors reflect hidden-state inference across time
Starkweather, C. K., Babayan, B. M., Uchida, N., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4520 · Cited by 206 · cited 10×
“To formalize these tasks, we largely followed previous work [7, 13].”
BackgroundMethodInfluentialSelf-citationRao, R. P. N. (2010) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2010.00146 · Cited by 191 · cited 3×
“Previous work in computational neuroscience has explored whether RNNs can be used to directly compute beliefs [17].”
BackgroundInfluentialRepresentation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · cited 5×
“In this study, we considered classical conditioning tasks where previous studies had suggested that dopamine activity reflected a belief representation [6–11].”
BackgroundMethodInfluentialMidbrain dopamine neurons signal belief in choice accuracy during a perceptual decision
Lak, A., Nomoto, K., Keramati, M., Sakagami, M., & Kepecs, A. (2017) · Current Biology · doi:10.1016/j.cub.2017.02.026 · Cited by 170 · cited 1×
“The brain may have dedicated machinery, perhaps in prefrontal cortex [12, 13], for computing belief states, which could then be provided to downstream areas, such as the basal ganglia, to perform standard RL algorithms such as TD learning [14].”
Connecting language to the world
Roy, D., & Reiter, E. (2005) · Artificial Intelligence · doi:10.1016/j.artint.2005.06.002 · Cited by 169
Stimulus representation and the timing of reward-prediction errors in models of the dopamine system
Ludvig, E. A., Sutton, R. S., & Kehoe, E. J. (2008) · Neural Computation · doi:10.1162/neco.2008.11-07-654 · Cited by 148 · cited 1×
“We observed that individual units in our RNNs were tuned to elapsed time relative to observations, and that the temporal precision of tuning decreased with elapsed time (S1 Fig), both of which are standard assumptions of microstate representations [41].”
BackgroundThe medial prefrontal cortex shapes dopamine reward prediction errors under state uncertainty
Starkweather, C. K., Gershman, S. J., & Uchida, N. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.036 · Cited by 136 · cited 6×
“To formalize these tasks, we largely followed previous work [7, 13].”
BackgroundMethodInfluentialSelf-citationValue-Directed Compression of POMDPs
Poupart, P., & Boutilier, C. E. (2002) · Cited by 103
Sequential Firing Codes for Time in Rodent Medial Prefrontal Cortex
Tiganj, Z., Jung, M. W., Kim, J., & Howard, M. W. (2017) · Cerebral Cortex · doi:10.1093/cercor/bhw336 · Cited by 103
Belief state representation in the dopamine system
Babayan, B. M., Uchida, N., & Gershman, S. J. (2018) · Nature Communications · doi:10.1038/s41467-018-04397-0 · Cited by 101 · cited 3×
“We next assessed whether the same basic insights generalized to a different task, that of Babayan et al. (2018) [10].”
BackgroundSelf-citationThe role of state uncertainty in the dynamics of dopamine
Mikhael, J. G., Kim, H. R., Uchida, N., & Gershman, S. J. (2022) · Current Biology · doi:10.1016/j.cub.2022.01.025 · Cited by 93
Self-citationTime representation in reinforcement learning models of the basal ganglia
Gershman, S. J., Moustafa, A. A., & Ludvig, E. A. (2014) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2013.00194 · Cited by 81
Self-citationEmergent behaviour and neural dynamics in artificial agents tracking odour plumes
Singh, S. H., van Breugel, F., Rao, R. P. N., & Brunton, B. W. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-022-00599-w · Cited by 61 · cited 1×
“…the intersection of neuroscience and machine learning has shown how agents performing reinforcement learning can use RNNs to solve a variety of tasks featuring state and task uncertainty, including tasks involving motor control [30], navigation [31–33], foraging [34], and decision-making [35, 36].”
BackgroundDeep neuroethology of a virtual rodent
Merel, J., Aldarondo, D., Marshall, J., Tassa, Y., Wayne, G., & Ölveczky, B. (2019) · arXiv · doi:10.48550/arxiv.1911.09451 · Cited by 44
Universality and individuality in neural dynamics across large populations of recurrent networks
Maheswaranathan, N., Williams, A. H., Golub, M. D., Ganguli, S., & Sussillo, D. (2019) · PubMed · doi:10.48550/arxiv.1907.08549 · Cited by 36
Sarno, S., de Lafuente, V., Romo, R., & Parga, N. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1712479114 · Cited by 31 · cited 1×
“The brain may have dedicated machinery, perhaps in prefrontal cortex [12, 13], for computing belief states, which could then be provided to downstream areas, such as the basal ganglia, to perform standard RL algorithms such as TD learning [14].”
How do real animals account for the passage of time during associative learning?
Namboodiri, V. M. K. (2022) · Behavioral Neuroscience · doi:10.1037/bne0000516 · Cited by 28 · cited 1×
“Traditional models of how animals perform trace conditioning tasks like the ones we consider here make a variety of implicit assumptions about how animals represent the passage of time [39, 40].”
BackgroundRational thoughts in neural codes
Wu, Z., Kwon, M., Daptardar, S., Schrater, P., & Pitkow, X. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912336117 · Cited by 27
Meta-trained agents implement Bayes-optimal agents
Mikulik, V., Delétang, G., McGrath, T., Genewein, T., Martic, M., Legg, S., & Ortega, P. A. (2020) · arXiv · doi:10.48550/arxiv.2010.11223 · Cited by 24 · cited 1×
“…the intersection of neuroscience and machine learning has shown how agents performing reinforcement learning can use RNNs to solve a variety of tasks featuring state and task uncertainty, including tasks involving motor control [30], navigation [31–33], foraging [34], and decision-making [35, 36].”
BackgroundInductive biases of neural network modularity in spatial navigation
Zhang, R., Pitkow, X., & Angelaki, D. E. (2024) · Science Advances · doi:10.1126/sciadv.adk1256 · Cited by 20
Reverse engineering recurrent neural networks with Jacobian switching linear dynamical systemsPreprint
Smith, J. T. H., Linderman, S. W., & Sussillo, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.01256 · Cited by 10 · cited 1×
“Moreover, there is a growing toolkit for reverse engineering RNN solutions [25, 26, 38], which can shed light on learned mechanisms of value computation.”
BackgroundSelf-citation(2005) · Artificial Intelligence Review · doi:10.1007/s10462-005-1061-5 · Cited by 1
Research works printed but not linked: 4
- Ni T, Eysenbach B, Salakhutdinov R. Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs. In: Chaudhuri K, Jegelka S, Song L, Szepesvari C, Niu G, Sabato S, editors. Proceedings of the 39th International Conference on Machine Learning. vol. 162 of Proceedings of Machine Learning Research. PMLR; 2022. p. 16691–16723.
- Xu T, Barak O. Implementing Inductive bias for different navigation tasks through diverse RNN attractors. arXiv preprint arXiv:200202496. 2020;.
- R Zhang, 2022, Inductive biases of neural networks for generalization in spatial navigation, bioRxiv, 2022
- Lambrechts G, Bolland A, Ernst D. Recurrent networks, hidden states and beliefs in partially observable environments. Transactions on Machine Learning Research. 2022;.
Cited by
A flexible hippocampal population code for experience relative to reward
Sosa, M., Plitt, M. H., & Giocomo, L. M. (2025) · Nature Neuroscience · doi:10.1038/s41593-025-01985-4 · Cited by 65
Prospective contingency explains behavior and dopamine signals during associative learning
Qian, L., Burrell, M., Hennig, J. A., Matias, S., Murthy, V. N., Gershman, S. J., & Uchida, N. (2025) · Nature Neuroscience · doi:10.1038/s41593-025-01915-4 · Cited by 44
“Previous work showed recurrent neural networks (RNNs), https://doi.org/10.1038/s41593-025-01915-4 trained to estimate value directly from observations (‘value-RNNs’), without aiming to develop belief-like representations 39 .”
InfluentialSelf-citationQian, L., Burrell, M., Hennig, J. A., Matias, S., Murthy, V. N., Gershman, S. J., & Uchida, N. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.02.05.578961 · Cited by 16
Self-citationCone, I., Clopath, C., & Shouval, H. Z. (2024) · Nature Communications · doi:10.1038/s41467-024-50205-3 · Cited by 18
“It is indeed possible to learn complex tasks with plastic RNN ’ s without assuming an a prioritemporal basis and the resulting neuronal dynamics might resemble cortical dynamics 21 .”
Nucleus accumbens dopamine release reflects Bayesian inference during instrumental learning
Qü, A. J., Tai, L.-H., Hall, C. D., Tu, E. M., Eckstein, M. K., Mishchanchuk, K., Lin, W. C., Chase, J. B., MacAskill, A. F., Collins, A. G. E., et al. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1013226 · Cited by 13
“To solve such tasks, evidence suggests that the brain represents a “belief state” (a probability distribution over hidden states), updated by Bayesian inference [8–12], or possibly learned implicitly in an end-to-end fashion [13].”
Self-citation
Metadata
- Authors
- Hennig, J. A., Romero Pinto, S. A., Yamaguchi, T., Linderman, S. W., Uchida, N., & Gershman, S. J.
- Affiliations
- Harvard University (US)
- Stanford University (US)
- Corresponding author
- Hennig, Jay A.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2023-09-11
- Volume
- 19
- Issue
- 9
- Pages
- e1011067
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- PMC
- PMC10513382
- OpenAlex
- W4386597955
- Cited by
- 41
- References
- 47
- Citations by year
- 2026: 10 · 2025: 23 · 2024: 7 · 2023: 1
- FWCI
- 3.85
- Type
- journal article
- Language
- en
- Update notices
- new_version (2023-09-21)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Computational Biology (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (U19 NS113201-01)
- Air Force Research Laboratory (Office of Scientific Research FA9550-20-1-0413)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- belief representation, reinforcement learning, reward prediction, value estimation, partial observability, recurrent neural networks, Bayesian estimation, hidden states, network capacity, reward prediction error, animal learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.