The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Decision Making Under Uncertainty: A Neural Model Based on Partially Observable Markov Decision Processes
Open accessAbstract
A fundamental problem faced by animals is learning to select actions based on noisy sensory information and incomplete knowledge of the world. It has been suggested that the brain engages in Bayesian inference during perception but how such probabilistic representations are used to select actions has remained unclear. Here we propose a neural model of action selection and decision making based on the theory of partially observable Markov decision processes (POMDPs). Actions are selected based not on a single "optimal" estimate of state but on the posterior distribution over states (the "belief" state). We show how such a model provides a unified framework for explaining experimental results in decision making that involve both information gathering and overt actions. The model utilizes temporal difference (TD) learning for maximizing expected reward. The resulting neural architecture posits an active role for the neocortex in belief computation while ascribing a role to the basal ganglia in belief representation, value computation, and action selection. When applied to the random dots motion discrimination task, model neurons representing belief exhibit responses similar to those of LIP neurons in primate neocortex. The appropriate threshold for switching from information gathering to overt actions emerges naturally during reward maximization. Additionally, the time course of reward prediction error in the model shares similarities with dopaminergic responses in the basal ganglia during the random dots task. For tasks with a deadline, the model learns a decision making strategy that changes with elapsed time, predicting a collapsing decision threshold consistent with some experimental studies. The model provides a new framework for understanding neural decision making and suggests an important role for interactions between the neocortex and the basal ganglia in learning the mapping between probabilistic sensory representations and actions that maximize rewards.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149
Marr, D. (1969) · The Journal of Physiology · doi:10.1113/jphysiol.1969.sp008820 · Cited by 3,253
A theory of cerebellar function
Albus, J. S. (1971) · Mathematical Biosciences · doi:10.1016/0025-5564(71)90051-4 · Cited by 2,539 · cited 1×
“Similar networks for function approximation, some-times called “radial-basis function” networks (Haykin, 2008), have been used to model a number of aspects of brain function (Marr, 1969; Albus, 1971; Poggio, 1990; Salinas and Abbott, 1995; Pouget and Sejnowski, 1997; Deneve and Pouget, 2003).”
Neural Basis of a Perceptual Decision in the Parietal Cortex (Area LIP) of the Rhesus Monkey
Shadlen, M. N., & Newsome, W. T. (2001) · Journal of Neurophysiology · doi:10.1152/jn.2001.86.4.1916 · Cited by 1,755
Neural correlates of decision variables in parietal cortex
Platt, M. L., & Glimcher, P. W. (1999) · Nature · doi:10.1038/22268 · Cited by 1,721 · cited 1×
“It has been shown that LIP neurons can be modulated by other variables such as value (Platt and Glimcher, 1999; Sugrue et al., 2004).”
Roitman, J. D., & Shadlen, M. N. (2002) · The Journal of Neuroscience · doi:10.1523/jneurosci.22-21-09475.2002 · Cited by 1,589 · cited 5×
“The random walk-like ramping behavior of the belief computing neurons in the model is comparable to the responses of cortical neurons in area LIP in the monkey ( Figure 9 , right panel) (Roitman and Shadlen, 2002).”
InfluentialBayesian inference with probabilistic population codes
Ma, W. J., Beck, J. M., Latham, P. E., & Pouget, A. (2006) · Nature Neuroscience · doi:10.1038/nn1790 · Cited by 1,457
Cited by
The Cost of Accumulating Evidence in Perceptual Decision Making
Drugowitsch, J., Moreno-Bote, R., Churchland, A. K., Shadlen, M. N., & Pouget, A. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.4010-11.2012 · Cited by 577
“Rao (2010) applied reinforcement learning to learn optimal strategies for tasks similar to the ones we consider, but assumed a constant cost function and a long sequence of similar trials rather than using average-reward reinforcement learning.”
Reinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446
Decision Making and Sequential Sampling from Memory
Shadlen, M. N., & Shohamy, D. (2016) · Neuron · doi:10.1016/j.neuron.2016.04.036 · Cited by 418
Dopamine neurons projecting to the posterior striatum form an anatomically distinct subclass
Menegas, W., Bergan, J. F., Ogawa, S. K., Isogai, Y., Umadevi Venkataraju, K., Osten, P., Uchida, N., & Watabe-Uchida, M. (2015) · eLife · doi:10.7554/elife.10032 · Cited by 322
Emotion and decision-making: affect-driven belief systems in anxiety and depression
Paulus, M. P., & Yu, A. J. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2012.07.009 · Cited by 287
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236
Dopamine reward prediction errors reflect hidden-state inference across time
Starkweather, C. K., Babayan, B. M., Uchida, N., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4520 · Cited by 206
Active inference and agency: optimal control without cost functions
Friston, K., Samothrakis, S., & Montague, R. (2012) · Biological Cybernetics · doi:10.1007/s00422-012-0512-8 · Cited by 241
“In many scenarios this is unrealistic, in that an agent cannot know the exact state of the world, given noisy or partial observations (Rao 2010).”
BackgroundDeeply Felt Affect: The Emergence of Valence in Deep Active Inference
Hesp, C., Smith, R., Parr, T., Allen, M., Friston, K. J., & Ramstead, M. J. D. (2021) · Neural Computation · doi:10.1162/neco_a_01341 · Cited by 246
“In cases where expected ambiguity is negligible, AC becomes equivalent to RPE, as both score differences in utility between expected and obtained outcomes (see Rao, 2010; Colombo, 2014; FitzGerald, Dolan, & Friston, 2015).”
BackgroundPlanning and navigation as active inference
Kaplan, R., & Friston, K. J. (2018) · Biological Cybernetics · doi:10.1007/s00422-018-0753-2 · Cited by 203
Orbitofrontal Cortex Is Required for Optimal Waiting Based on Decision Confidence
Lak, A., Costa, G. M., Romberg, E., Koulakov, A. A., Mainen, Z. F., & Kepecs, A. (2014) · Neuron · doi:10.1016/j.neuron.2014.08.039 · Cited by 209
Midbrain dopamine neurons signal belief in choice accuracy during a perceptual decision
Lak, A., Nomoto, K., Keramati, M., Sakagami, M., & Kepecs, A. (2017) · Current Biology · doi:10.1016/j.cub.2017.02.026 · Cited by 170
The successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191
“For simplicity of exposition, we have chosen to omit actions, though our treatment extends straightforwardly to handle actions (Russek et al., 2017).”
BackgroundMethodInfluentialModel-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161
Pezzulo, G., Rigoli, F., & Chersi, F. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00092 · Cited by 163
The medial prefrontal cortex shapes dopamine reward prediction errors under state uncertainty
Starkweather, C. K., Gershman, S. J., & Uchida, N. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.036 · Cited by 136
Belief state representation in the dopamine system
Babayan, B. M., Uchida, N., & Gershman, S. J. (2018) · Nature Communications · doi:10.1038/s41467-018-04397-0 · Cited by 101
Anterior insula reflects surprise in value-based decision-making and perception
Loued-Khenissi, L., Pfeuffer, A., Einhäuser, W., & Preuschoff, K. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116549 · Cited by 66
Khalvati, K., Kiani, R., & Rao, R. P. N. (2021) · Nature Communications · doi:10.1038/s41467-021-25419-4 · Cited by 64
Self-citationEmergence of belief-like representations through reinforcement learning
Hennig, J. A., Romero Pinto, S. A., Yamaguchi, T., Linderman, S. W., Uchida, N., & Gershman, S. J. (2023) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011067 · Cited by 41
Dopamine, inference, and uncertainty
Gershman, S. J. (2017) · Neural Computation · doi:10.1162/neco_a_01023 · Cited by 40
Rational thoughts in neural codes
Wu, Z., Kwon, M., Daptardar, S., Schrater, P., & Pitkow, X. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1912336117 · Cited by 27
Kuo, P.-C., Hou, H., Dabney, W., & Walker, E. Y. (2025) · arXiv
“…upcoming observations and updates its internal world models [17, 18], provides a neurally plausible implementation of Bayesian inference [19, 20] and has been instrumental in explaining diverse neural circuits including feature learning in sensory cortices [21, 22], motor control in the…”
Metadata
- Authors
- Rao, R. P. N.
- Venue
- Frontiers in Computational Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2010
- Volume
- 4
- Pages
- 146-146
- ISSN
- 1662-5188 (ISSN-L)
- OpenAlex
- W2045833919
- Cited by
- 191
- References
- 24
- FWCI
- 4.71
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.