The DOI matches its Crossref record; 96 of the 96 references it cites were found in a registry or an index.
Mechanisms of Hierarchical Reinforcement Learning in Corticostriatal Circuits 1: Computational Analysis
Open accessAbstract
Growing evidence suggests that the prefrontal cortex (PFC) is organized hierarchically, with more anterior regions having increasingly abstract representations. How does this organization support hierarchical cognitive control and the rapid discovery of abstract action rules? We present computational models at different levels of description. A neural circuit model simulates interacting corticostriatal circuits organized hierarchically. In each circuit, the basal ganglia gate frontal actions, with some striatal units gating the inputs to PFC and others gating the outputs to influence response selection. Learning at all of these levels is accomplished via dopaminergic reward prediction error signals in each corticostriatal circuit. This functionality allows the system to exhibit conditional if-then hypothesis testing and to learn rapidly in environments with hierarchical structure. We also develop a hybrid Bayesian-reinforcement learning mixture of experts (MoE) model, which can estimate the most likely hypothesis state of individual participants based on their observed sequence of choices and rewards. This model yields accurate probabilistic estimates about which hypotheses are attended by manipulating attentional states in the generative neural model and recovering them with the MoE model. This 2-pronged modeling approach leads to multiple quantitative predictions that are tested with functional magnetic resonance imaging in the companion paper.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
An Integrative Theory of Prefrontal Cortex Function
Miller, E. K., & Cohen, J. D. (2001) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.24.1.167 · Cited by 12,803
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Parallel organization of functionally segregated circuits linking basal ganglia and cortex
Alexander, G. E., DeLong, M. R., & Strick, P. L. (1986) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.09.030186.002041 · Cited by 8,527
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384
Holroyd, C. B., & Coles, M. G. H. (2002) · Psychological Review · doi:10.1037/0033-295x.109.4.679 · Cited by 4,027
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
THE BASAL GANGLIA: FOCUSED SELECTION AND INHIBITION OF COMPETING MOTOR PROGRAMS
MINK, J. W. (1996) · Progress in Neurobiology · doi:10.1016/s0301-0082(96)00042-1 · Cited by 2,486
The Prefrontal Cortex: Anatomy, Physiology, and Neuropsychology of the Frontal Lobe
Fuster, J. M. (1980) · Cited by 2,407
Cited by
The Cognitive Neuroscience of Working Memory
D'Esposito, M., & Postle, B. R. (2015) · Annual Review of Psychology · doi:10.1146/annurev-psych-010814-015031 · Cited by 1,653
Toward a rational and mechanistic account of mental effort
Shenhav, A., Musslick, S., Lieder, F., Kool, W., Griffiths, T. L., Cohen, J. D., & Botvinick, M. M. (2017) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-072116-031526 · Cited by 1,077
Dolan, R. J., & Dayan, P. (2013) · Neuron · doi:10.1016/j.neuron.2013.09.007 · Cited by 977
Motivation and cognitive control: from behavior to neural mechanism
Botvinick, M., & Braver, T. (2015) · Annual Review of Psychology · doi:10.1146/annurev-psych-010814-015044 · Cited by 909
Frontal Cortex and the Hierarchical Control of Behavior
Badre, D., & Nee, D. E. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.11.005 · Cited by 611
Self-citationWiers, R. W., Gladwin, T. E., Hofmann, W., Salemink, E., & Ridderinkhof, K. R. (2013) · Clinical Psychological Science · doi:10.1177/2167702612466547 · Cited by 486
Action, Outcome, and Value: A Dual-System Framework for Morality
Cushman, F. (2013) · Personality and Social Psychology Review · doi:10.1177/1088868313495594 · Cited by 473
Cognitive control over learning: Creating, clustering, and generalizing task-set structure.
Collins, A. G. E., & Frank, M. J. (2013) · Psychological Review · doi:10.1037/a0030852 · Cited by 426
“Second, we also include a diagonal input from the PFC to the striatum of the second loop, thereby contextualizing motor action selection according to cognitive state (see also Frank & Badre, 2012).”
InfluentialSelf-citationCollins, A. G. E., & Frank, M. J. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07980.x · Cited by 417
Self-citationA computational model of inhibitory control in frontal cortex and basal ganglia.
Wiecki, T. V., & Frank, M. J. (2013) · Psychological Review · doi:10.1037/a0031542 · Cited by 404
Self-citationThe ubiquity of model-based reinforcement learning
Doll, B. B., Simon, D. A., & Daw, N. D. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.08.003 · Cited by 380
Reinforcement Learning in Multidimensional Environments Relies on Attention Mechanisms
Niv, Y., Daniel, R., Geana, A., Gershman, S. J., Leong, Y. C., Radulescu, A., & Wilson, R. C. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.2978-14.2015 · Cited by 362
Hierarchical active inference: A theory of motivated control
Pezzulo, G., Rigoli, F., & Friston, K. J. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.01.009 · Cited by 334
“Calling on early cybernetic models [16–18], the view that the brain uses control hierarchies has inspired many recent proposals [19–25]; for example, hierarchical temporal structures [26], hierarchical reinforcement learning [9], hierarchical mixture of experts [23], distributed adaptive control [8,27] and hierarchica…”
Closed-loop training of attention with real-time brain imaging
deBettencourt, M. T., Cohen, J. D., Lee, R. F., Norman, K. A., & Turk-Browne, N. B. (2015) · Nature Neuroscience · doi:10.1038/nn.3940 · Cited by 324
Rostrolateral Prefrontal Cortex and Individual Differences in Uncertainty-Driven Exploration
Badre, D., Doll, B. B., Long, N. M., & Frank, M. J. (2012) · Neuron · doi:10.1016/j.neuron.2011.12.025 · Cited by 309
Self-citationDopamine Does Double Duty in Motivating Cognitive Effort
Westbrook, A., & Braver, T. S. (2016) · Neuron · doi:10.1016/j.neuron.2015.12.029 · Cited by 308
Multiple gates on working memory
Chatham, C. H., & Badre, D. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2014.08.001 · Cited by 268
Self-citationHabits, action sequences and reinforcement learning
Dezfouli, A., & Balleine, B. W. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08050.x · Cited by 297
Computational approaches to psychiatry
Stephan, K. E., & Mathys, C. (2014) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2013.12.007 · Cited by 260
The frontal aslant tract (FAT) and its role in speech, language and executive function
Dick, A. S., Garic, D., Graziano, P., & Tremblay, P. (2019) · Cortex · doi:10.1016/j.cortex.2018.10.015 · Cited by 274
Reinforcement learning in artificial and biological systems
Neftci, E. O., & Averbeck, B. B. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0025-4 · Cited by 268
Towards Continual Reinforcement Learning: A Review and Perspectives
Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13673 · Cited by 224
Banishing the Control Homunculi in Studies of Action Control and Behavior Change
Verbruggen, F., McLaren, I. P. L., & Chambers, C. D. (2014) · Perspectives on Psychological Science · doi:10.1177/1745691614526414 · Cited by 214
Stratified medicine for mental disorders
Schumann, G., Binder, E. B., Holte, A., de Kloet, E. R., Oedegaard, K. J., Robbins, T. W., Walker-Tilley, T. R., Bitter, I., Brown, V. J., Buitelaar, J., et al. (2014) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2013.09.010 · Cited by 186
Cognitive Control: Core Constructs and Current Considerations
Cohen, J. D. (2017) · The Wiley Handbook of Cognitive Control · doi:10.1002/9781118920497.ch1 · Cited by 179
Structured Event Memory: A neuro-symbolic model of event cognition.
Franklin, N. T., Norman, K. A., Ranganath, C., Zacks, J. M., & Gershman, S. J. (2020) · Psychological Review · doi:10.1037/rev0000177 · Cited by 177
Pezzulo, G., Rigoli, F., & Chersi, F. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00092 · Cited by 163
“In contrast, the control of behavior has been recently linked to hierarchical reinforcement learning models (Botvinick, 2008; Botvinick et al., 2009; Frank and Badre, 2012), in which actions can be specified at different levels of abstractness and temporal extension (see also Verschure et al.”
Jiang, J., Beck, J., Heller, K., & Egner, T. (2015) · Nature Communications · doi:10.1038/ncomms9165 · Cited by 160
TOWARDS A GENERAL MODEL OF TEMPORAL DISCOUNTING
van den Bos, W., & McClure, S. M. (2013) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.6 · Cited by 149
“…between brain valuation systems that learn direct associations between stimuli or actions and rewards, and more sophisticated systems in which actions are planned in a goaldirection manner using prospective reasoning (Botvinick, Niv, & Barto, 2009; Daw, Niv, & Dayan, 2005; Frank & Badre, 2012).”
Toward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119
Arousal-related adjustments of perceptual biases optimize perception in dynamic environments
Krishnamurthy, K., Nassar, M. R., Sarode, S., & Gold, J. I. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0107 · Cited by 117
Integrative frontal-parietal dynamics supporting cognitive control
Nee, D. E. (2021) · eLife · doi:10.7554/elife.57244 · Cited by 112
Degree of automaticity and the prefrontal cortex
Jeon, H.-A., & Friederici, A. D. (2015) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2015.03.003 · Cited by 60
Prefrontal Goal Codes Emerge as Latent States in Probabilistic Value Learning
Stoianov, I., Genovesio, A., & Pezzulo, G. (2016) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00886 · Cited by 44
“The overall nonparametric approach has a viable biological implementation, too, and points to hierarchical statistical learning in prefrontal hierarchies (Friston, 2008; Frank and Badre, 2012; Koechlin and Summerfield, 2007) shaped by reinforcement-related signals through prefrontal – (ventral) basal ganglia loops (O’…”
BackgroundInfluentialSolomon, M., Ragland, J. D., Niendam, T. A., Lesh, T. A., Beck, J. S., Matter, J. C., Frank, M. J., & Carter, C. S. (2015) · Journal of the American Academy of Child & Adolescent Psychiatry · doi:10.1016/j.jaac.2015.08.010 · Cited by 34
Self-citation(2025) · doi:10.1371/journal.pcbi.1012872 · Cited by 3
“The behavioral task used was a variant of the classic RLWM task (Collins & Frank 2012).”
Connecting Context-specific Adaptation in Humans to Meta-learning
Dubey, R., Grant, E., Luo, M., Narasimhan, K., & Griffiths, T. (2020) · arXiv · doi:10.48550/arxiv.2011.13782 · Cited by 1
Metadata
- Authors
- Frank, M. J., & Badre, D.
- Venue
- Cerebral Cortex
- Publisher
- Oxford University Press (OUP)
- Published
- 2011-06-21
- Volume
- 22
- Issue
- 3
- Pages
- 509-526
- ISSN
- 1047-3211 (ISSN-L)
- 1460-2199
- OpenAlex
- W2140956413
- Cited by
- 337
- References
- 96
- FWCI
- 9.95
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.