Strategic cognitive sequencing: a computational cognitive neuroscience approach
Open accessAbstract
We address strategic cognitive sequencing, the "outer loop" of human cognition: how the brain decides what cognitive process to apply at a given moment to solve complex, multistep cognitive tasks. We argue that this topic has been neglected relative to its importance for systematic reasons but that recent work on how individual brain systems accomplish their computations has set the stage for productively addressing how brain regions coordinate over time to accomplish our most impressive thinking. We present four preliminary neural network models. The first addresses how the prefrontal cortex (PFC) and basal ganglia (BG) cooperate to perform trial-and-error learning of short sequences; the next, how several areas of PFC learn to make predictions of likely reward, and how this contributes to the BG making decisions at the level of strategies. The third models address how PFC, BG, parietal cortex, and hippocampus can work together to memorize sequences of cognitive actions from instruction (or "self-instruction"). The last shows how a constraint satisfaction process can find useful plans. The PFC maintains current and goal states and associates from both of these to find a "bridging" state, an abstract plan. We discuss how these processes could work together to produce strategic cognitive sequencing and discuss future directions in this area.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The complementary learning systems hypothesis: the hippocampus plays a key role in processes where learning must occur quickly on the basis of single episodes, whereas the cortex learns more slowly by aggregating and integrating patterns across large amounts of data.
“An influential idea—known as the “complementary learning systems hypothesis”—is that the hippocampus plays a key role in certain processes where learning must occur quickly on the basis of single episodes, whereas the cortex learns more slowly by aggregating and integrating patterns across large amounts of data (Herd et al., 2013; Leibo et al., 2015a; Blundell et al., 2016; Kumaran et al., 2016).”
From Toward an integration of deep learning and neuroscience (2016) · cites this paper as Herd et al., 2013
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Artificial intelligence: a modern approach
Kumar, D. A., Balasubramanian, S., Byeon, D. H., & Manerkar, P. G. V. (1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 22,182 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Parallel organization of functionally segregated circuits linking basal ganglia and cortex
Alexander, G. E., DeLong, M. R., & Strick, P. L. (1986) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.09.030186.002041 · Cited by 8,527 · see citing works
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855 · see citing works
Anderson, J. R. (2013) · doi:10.4324/9781315799438 · Cited by 6,761 · see citing works
Specific impairments of planning
Shallice, T. (1982) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1982.0082 · Cited by 3,741 · see citing works
An Integrated Theory of the Mind.
Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C. J., & Qin, Y. (2004) · Psychological Review · doi:10.1037/0033-295x.111.4.1036 · Cited by 3,048 · see citing works
The role of the basal ganglia in habit formation
Yin, H. H., & Knowlton, B. J. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn1919 · Cited by 2,402 · see citing works
Solso, R. L., & Chase, W. G. (1974) · The American Journal of Psychology · doi:10.2307/1421395 · Cited by 2,133 · see citing works
The prefontral cortex and cognitive control
Miller, E. K. (2000) · Nature Reviews Neuroscience · doi:10.1038/35036228 · Cited by 2,016 · see citing works
By Carrot or by Stick: Cognitive Reinforcement Learning in Parkinsonism
Frank, M. J., Seeberger, L. C., & O'Reilly, R. C. (2004) · Science · doi:10.1126/science.1102941 · Cited by 1,964 · see citing works
Self-citationSlovic, P., Fischhoff, B., & Lichtenstein, S. (1977) · Annual Review of Psychology · doi:10.1146/annurev.ps.28.020177.000245 · Cited by 1,731 · see citing works
Fast planning through planning graph analysis
Blum, A. L., & Furst, M. L. (1997) · Artificial Intelligence · doi:10.1016/s0004-3702(96)00047-1 · Cited by 1,566 · see citing works
Temporal difference models and reward-related learning in the human brain
O'Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan, R. J. (2003) · Neuron · doi:10.1016/s0896-6273(03)00169-7 · Cited by 1,522 · see citing works
Planning in a hierarchy of abstraction spaces
Sacerdoti, E. D. (1974) · Artificial Intelligence · doi:10.1016/0004-3702(74)90026-5 · Cited by 1,185 · see citing works
O’Reilly, R. C., & Frank, M. J. (2005) · Neural Computation · doi:10.1162/089976606775093909 · Cited by 1,066 · see citing works
Self-citationA Contingency Model for the Selection of Decision Strategies
Beach, L. R., & Mitchell, T. R. (1978) · Academy of Management Review · doi:10.5465/amr.1978.4305717 · Cited by 973 · see citing works
Frank, M. J. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/0898929052880093 · Cited by 925 · see citing works
Orbitofrontal Cortex and Its Contribution to Decision-Making
Wallis, J. D. (2007) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.30.051606.094334 · Cited by 895 · see citing works
Semantic Cognition: A Parallel Distributed Processing Approach
Rogers, T. T., & McClelland, J. L. (2004) · Cited by 891 · see citing works
Interactions between frontal cortex and basal ganglia in working memory: A computational model
FRANK, M. J., LOUGHRY, B., & O'REILLY, R. C. (2001) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.1.2.137 · Cited by 879 · see citing works
Self-citationReward, Motivation, and Reinforcement Learning
Dayan, P., & Balleine, B. W. (2002) · Neuron · doi:10.1016/s0896-6273(02)00963-7 · Cited by 855 · see citing works
YOU CAN'T PLAY 20 QUESTIONS WITH NATURE AND WIN: PROJECTIVE COMMENTS ON THE PAPERS OF THIS SYMPOSIUM
Newell, A. (1973) · Visual Information Processing · doi:10.1016/b978-0-12-170150-5.50012-3 · Cited by 820 · cited 1× · see citing works
“…28 May 2013 Academic Editor: Giorgio Ascoli Copyright © 2013 Seth A. Herd et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.”
MethodThe parallel distributed processing approach to semantic cognition
McClelland, J. L., & Rogers, T. T. (2003) · Nature reviews. Neuroscience · doi:10.1038/nrn1076 · Cited by 796 · see citing works
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 769 · see citing works
The short-latency dopamine signal: a role in discovering novel actions?
Redgrave, P., & Gurney, K. (2006) · Nature Reviews Neuroscience · doi:10.1038/nrn2022 · Cited by 762 · see citing works
Frank, M. J., & Claus, E. D. (2006) · Psychological Review · doi:10.1037/0033-295x.113.2.300 · Cited by 578 · see citing works
Effort-Based Cost–Benefit Valuation and the Human Brain
Croxson, P. L., Walton, M. E., O'Reilly, J. X., Behrens, T. E. J., & Rushworth, M. F. S. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.4515-08.2009 · Cited by 560 · see citing works
Learning to Learn: Introduction and Overview
Thrun, S., & Pratt, L. (1998) · Learning to Learn · doi:10.1007/978-1-4615-5529-2_1 · Cited by 521 · see citing works
Actor–critic models of the basal ganglia: new anatomical and computational perspectives
Joel, D., Niv, Y., & Ruppin, E. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00047-3 · Cited by 500 · see citing works
Time-Derivative Models of Pavlovian Reinforcement
Sutton, R. S., & Barto, A. G. (1990) · Cited by 496 · see citing works
Production System Models of Learning and Development
(1987) · The MIT Press eBooks · doi:10.7551/mitpress/5605.001.0001 · Cited by 487 · see citing works
A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling
Baxter, J. (1997) · Machine Learning · doi:10.1023/a:1007327622663 · Cited by 460 · see citing works
Role of the Lateral Prefrontal Cortex in Executive Behavioral Control
Tanji, J., & Hoshi, E. (2008) · Physiological Reviews · doi:10.1152/physrev.00014.2007 · Cited by 451 · see citing works
Hazy, T. E., Frank, M. J., & O’Reilly, R. C. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2055 · Cited by 450 · see citing works
Self-citationTask Set and Prefrontal Cortex
Sakai, K. (2008) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.31.060407.125642 · Cited by 390 · see citing works
Banishing the homunculus: Making working memory work
Hazy, T. E., Frank, M. J., & O’Reilly, R. C. (2006) · Neuroscience · doi:10.1016/j.neuroscience.2005.04.067 · Cited by 385 · see citing works
Self-citationMapping the network for planning: a correlational PET activation study with the Tower of London task
Dagher, A., Owen, A. M., Boecker, H., & Brooks, D. J. (1999) · Brain · doi:10.1093/brain/122.10.1973 · Cited by 338 · see citing works
Clever animals and killjoy explanations in comparative psychology
Shettleworth, S. J. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.07.002 · Cited by 292 · see citing works
Tracking the emergence of conceptual knowledge during human decision making
Kumaran, D., Summerfield, J. J., Hassabis, D., & Maguire, E. A. (2009) · Neuron · doi:10.1016/j.neuron.2009.07.030 · Cited by 260 · see citing works
The role of the striatum and hippocampus in planning: A PET activation study in Parkinson's disease
Dagher, A. (2001) · Brain · doi:10.1093/brain/124.5.1020 · Cited by 260 · see citing works
Categorization of behavioural sequences in the prefrontal cortex
Shima, K., Isoda, M., Mushiake, H., & Tanji, J. (2006) · Nature · doi:10.1038/nature05470 · Cited by 259 · see citing works
Fernández, G., Weyerts, H., Schrader-Bölsche, M., Tendolkar, I., Smid, H. G. O. M., Tempelmann, C., Hinrichs, H., Scheich, H., Elger, C. E., Mangun, G. R., et al. (1998) · Journal of Neuroscience · doi:10.1523/jneurosci.18-05-01841.1998 · Cited by 253 · see citing works
van den Heuvel, O. A., Groenewegen, H. J., Barkhof, F., Lazeron, R. H. C., van Dyck, R., & Veltman, D. J. (2003) · NeuroImage · doi:10.1016/s1053-8119(02)00010-1 · Cited by 247 · see citing works
The Cognitive Psychology of Planning
Ward, G. (2004) · Psychology Press eBooks · doi:10.4324/9780203493564 · Cited by 224 · see citing works
Contextual encoding by ensembles of medial prefrontal cortex neurons
Hyman, J. M., Ma, L., Balaguer‐Ballester, E., Durstewitz, D., & Seamans, J. K. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1114415109 · Cited by 222 · see citing works
Building portable options: skill transfer in reinforcement learning
Konidaris, G., & Barto, A. G. (2007) · Cited by 220 · see citing works
A model of perceptual classification in children and adults.
Smith, L. B. (1989) · Psychology Review · doi:10.1037/0033-295x.96.1.125 · Cited by 210 · see citing works
Instructional control of reinforcement learning: A behavioral and neurocomputational investigation
Doll, B. B., Jacobs, W. J., Sanfey, A. G., & Frank, M. J. (2009) · Brain Research · doi:10.1016/j.brainres.2009.07.007 · Cited by 209 · see citing works
Recurrent Processing during Object Recognition
O’Reilly, R. C., Wyatte, D., Herd, S. A., Mingus, B., & Jilk, D. J. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00124 · Cited by 197 · see citing works
Self-citationKennerley, S. W., & Walton, M. E. (2011) · Behavioral Neuroscience · doi:10.1037/a0023575 · Cited by 179 · see citing works
Re-evaluating the role of the orbitofrontal cortex in reward and reinforcement
Noonan, M. P., Kolling, N., Walton, M. E., & Rushworth, M. F. S. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08023.x · Cited by 169 · see citing works
Behavioural evidence for mental time travel in nonhuman animals
Suddendorf, T., & Corballis, M. C. (2010) · Behavioural Brain Research · doi:10.1016/j.bbr.2009.11.044 · Cited by 168 · see citing works
SAL: an explicitly pluralistic cognitive architecture
Jilk, D. J., Lebiere, C. J., O’Reilly, R. C., & Anderson, J. R. (2008) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/09528130802319128 · Cited by 161 · see citing works
Self-citationPhasic Nucleus Accumbens Dopamine Release Encodes Effort- and Delay-Related Costs
Day, J. J., Jones, J. L., Wightman, R. M., & Carelli, R. M. (2010) · Biological Psychiatry · doi:10.1016/j.biopsych.2010.03.026 · Cited by 160 · see citing works
How instructed knowledge modulates the neural systems of reward learning
Li, J., Delgado, M. R., & Phelps, E. A. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1014938108 · Cited by 149 · see citing works
Bacchus, F. (2001) · AI Magazine · doi:10.1609/aimag.v22i3.1571 · Cited by 144 · see citing works
PVLV: The Primary Value and Learned Value Pavlovian Learning Algorithm.
O’Reilly, R. C., Frank, M. J., Hazy, T. E., & Watz, B. (2007) · Behavioral Neuroscience · doi:10.1037/0735-7044.121.1.31 · Cited by 144 · see citing works
Self-citationPlanning as constraint satisfaction: Solving the planning graph by compiling it into CSP
Binh, M., & Kambhampati, S. (2001) · Artificial Intelligence · doi:10.1016/s0004-3702(01)00128-x · Cited by 136 · see citing works
From an Executive Network to Executive Control: A Computational Model of the<i>n</i>-back Task
Chatham, C. H., Herd, S. A., Brant, A. M., Hazy, T. E., Miyake, A., O'Reilly, R., & Friedman, N. P. (2011) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00047 · Cited by 115 · see citing works
Self-citationComputational Models for the Combination of Advice and Individual Learning
Biele, G., Rieskamp, J., & Gonzalez, R. (2009) · Cognitive Science · doi:10.1111/j.1551-6709.2009.01010.x · Cited by 114 · see citing works
Computational models of cognitive control
O’Reilly, R. C., Herd, S. A., & Pauli, W. M. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.01.008 · Cited by 113 · see citing works
Self-citationNeural mechanisms of acquired phasic dopamine responses in learning
Hazy, T. E., Frank, M. J., & O’Reilly, R. C. (2009) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2009.11.019 · Cited by 103 · see citing works
Self-citationModulation of the feedback-related negativity by instruction and experience
Walsh, M. M., & Anderson, J. R. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1117189108 · Cited by 102 · see citing works
Task rules, working memory, and fluid intelligence
Duncan, J., Schramm, M. W. J., Thompson, R., & Dumontheil, I. (2012) · Psychonomic Bulletin & Review · doi:10.3758/s13423-012-0225-y · Cited by 96 · see citing works
Hypothalamic neuron involvement in integration of reward, aversion, and cue signals
Ono, T., Nakamura, K., Nishijo, H., & Fukuda, M. (1986) · Journal of Neurophysiology · doi:10.1152/jn.1986.56.1.63 · Cited by 87 · see citing works
Reactivity of limbic neurons of the monkey to appetitive and aversive signals.
Fuster, J., & Uyeda, A. A. (1971) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(71)90111-8 · Cited by 85 · see citing works
Bilinearity, rules, and prefrontal cortex
Dayan, P. (2007) · Frontiers in Computational Neuroscience · doi:10.3389/neuro.10.001.2007 · Cited by 66 · see citing works
Reward, memory and substance abuse: functional neuronal circuits in the nucleus accumbens
Deadwyler, S. A., Hayashizaki, S., Cheer, J. F., & Hampson, R. E. (2003) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2003.11.011 · Cited by 64 · see citing works
The Tower of London: the impact of instructions, cueing, and learning on planning abilities.
Unterrainer, J. M., Rahm, B., Leonhart, R., Ruff, C. C., & Halsband, U. (2003) · Brain research. Cognitive brain research · doi:10.1016/s0926-6410(03)00191-5 · Cited by 61 · see citing works
Fu, W., & Anderson, J. R. (2007) · Psychological Research · doi:10.1007/s00426-007-0113-7 · Cited by 52 · see citing works
Meta-Cognitive Strategy Instruction in Intelligent Tutoring Systems: How, When, and Why
Chi, M., & VanLehn, K. A. (2010) · Cited by 48 · see citing works
Proto-transfer Learning in Markov Decision Processes Using Spectral Methods
Ferguson, K., & Mahadevan, S. (2006) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 47 · see citing works
Investigating Knowledge Integration in Web-Based Thematic Learning Using Concept Mapping Assessment.
Liu, M., & Wang, J.-Y. (2010) · Cited by 45 · see citing works
Understanding strategy selection
Roberts, M. J., & Newton, E. J. (2001) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.2000.0434 · Cited by 39 · see citing works
The Leabra Cognitive Architecture: How to Play 20 Principles with Nature and Win!
O’Reilly, R. C., Hazy, T. E., & Herd, S. A. (2012) · Cited by 39 · see citing works
Self-citationFormalizing the PRODIGY Planning Algorithm
Fink, E., & Veloso, M. (2018) · Figshare · doi:10.1184/r1/6605837 · Cited by 27 · see citing works
Multi-Tasking and Cognitive Workload in an ACT-R Model of a Simplified Air Traffic Control Task
Lebiere, C., Anderson, J. R., & Bothell, D. (2018) · KiltHub Repository · doi:10.1184/r1/6617216 · Cited by 26 · see citing works
Owen, A. M. (1997) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(97)01042-5 · Cited by 19 · see citing works
Constraint Based Planning with Composable Substate Graphs
Peter, G., Derek, L., & Maria, F. (2010) · Frontiers in artificial intelligence and applications · doi:10.3233/978-1-60750-606-5-453 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [15]O'Reilly RC Munakata Y Computational Explorations in Cognitive Neuroscience: Understanding the Mind By Simulating the Brain 2000 Cambridge, Mass, USA The MIT Press
- [17]Newell A Simon HA Human Problem Solving 1972 Englewood Cliffs, NJ, USA Prentice Hall
- [26]Anderson JR How Can the Human Mind Occur in the Physical Universe? 2007 New York, NY, USA Oxford University Press
- [45]Sutton RS Barto AG Reinforcement Learning: An Introduction 1998 Cambridge, Mass, USA MIT Press
- [65]Noelle DC Cottrell GW Moore JD Lehman JF A connectionist model of instruction following, pages Proceedings of the 17th Annual Conference of the Cognitive Science Society January 1995 Mahwah, NJ, USA Lawrence Erlbaum Associates 369 374
Cited by
Sorted by citations
Toward an integration of deep learning and neuroscience
Marblestone, A. H., Wayne, G., & Kording, K. P. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00094 · Cited by 119 · see citing works
- Authors
- Herd, S. A., Krueger, K. A., Kriete, T. E., Huang, T.-R., Hazy, T. E., & O'Reilly, R. C.
- Affiliations
- University of Colorado Boulder (US)
- Department of Psychology, University of Colorado Boulder, Boulder, CO 80309, USA
- Corresponding author
- Herd, Seth A.
- Venue
- Computational Intelligence and Neuroscience
- Publisher
- Wiley
- Published
- 2013
- Volume
- 2013
- Pages
- 1-18
- ISSN
- 1687-5265 (ISSN-L)
- 1687-5273
- PMC
- PMC3722785
- OpenAlex
- W2028574200
- Cited by
- 20
- References
- 80
- Citations by year
- 2024: 1 · 2023: 3 · 2022: 1 · 2021: 3 · 2018: 1 · 2017: 2 · 2016: 3 · 2015: 1 · 2014: 4 · 2013: 1
- FWCI
- 1.64
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Computational Intelligence and Neuroscience (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Office of Naval Research (N00014-10-1-0177, MH079485, D10PC20021)
- National Institutes of Health (N00014-10-1-0177, MH079485, D10PC20021)
- Intelligence Advanced Research Projects Activity (N00014-10-1-0177, MH079485, D10PC20021)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Cognitive Functions and Memory
- Keywords
- prefrontal cortex, basal ganglia, trial-and-error learning, sequence learning, reward prediction, decision-making, self-education, hippocampus, parietal cortex, constraint satisfaction, planning, neural network model
- MeSH terms
- Brain, Cognition, Humans, Models, Neurological, Neurosciences, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
