Learning latent structure: carving nature at its joints
Open accessAbstract
Reinforcement learning (RL) algorithms provide powerful explanations for simple learning and decision-making behaviors and the functions of their underlying neural substrates. Unfortunately, in real-world situations that involve many stimuli and actions, these algorithms learn pitifully slowly, exposing their inferiority in comparison to animal and human learning. Here we suggest that one reason for this discrepancy is that humans and animals take advantage of structure that is inherent in real-world tasks to simplify the learning problem. We survey an emerging literature on 'structure learning'--using experience to infer the structure of a task--and how this can be of service to RL, with an emphasis on structure in perception and action.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Mismatches between expected and actual outcomes might produce boundaries and later shifts in context, aligning with findings that such mismatches promote segmentation in learning.
“One possibility is that prediction errors, or discrepancies between expected and actual outcomes, generate event boundaries and ensuing context shifts [38–40], consistent with evidence that prediction errors can drive segmentation in learning [41,42].”
From Does mental context drift or shift? (2017) · cites this paper as [42]
Processes of information selection tap into high cognitive functions, including intrinsic curiosity, advance planning, and forming internal models of complex tasks.
“Recent evidence from computational and behavioral studies makes it clear that processes of information selection tap into some of our highest cognitive functions, involving, among others, intrinsic curiosity and the ability for advance planning and forming internal models of complex tasks (e.g., (Gershman and Niv, 2010; Johnson et al., 2012)).”
From Attention, learning, and the value of information (2012) · cites this paper as Gershman and Niv, 2010
To determine which sources of uncertainty should be optimally resolved, the brain must know which steps are causal or predictive of the future outcome.
“In other words, to determine which sources of uncertainty should be optimally resolved, the brain must know which steps are causal or predictive of the future outcome (Gershman and Niv, 2010).”
From Attention, learning, and the value of information (2012) · cites this paper as Gershman and Niv, 2010
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 7,936 · see citing works
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 7,268 · see citing works
Information theory, inference, and learning algorithms
(2004) · Choice Reviews Online · doi:10.5860/choice.41-5949 · Cited by 6,535 · see citing works
Information Theory, Inference, and Learning Algorithms
(2004) · Kybernetes · doi:10.1108/03684920410534506 · Cited by 4,937 · see citing works
Classical conditioning II: Current research and theory.
Black, A. H., & Prokasy, W. F. (1972) · Appleton-Century-Crofts eBooks · Cited by 4,449 · see citing works
Neural Information Processing Systems
(2018) · The MIT Press eBooks · doi:10.7551/mitpress/11474.003.0014 · Cited by 3,775 · see citing works
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
Self-citationEffects of Different Brain Lesions on Card Sorting: The Role of the Frontal Lobes
MILNER, B. (1963) · Archives of Neurology · doi:10.1001/archneur.1963.00460070100010 · Cited by 2,217 · see citing works
Handbuch der physiologischen Optik
Cz., P. (1896) · Monatshefte für Mathematik · doi:10.1007/bf01708548 · Cited by 2,071 · see citing works
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053 · see citing works
Handbuch der physiologischen Optik
Helmholtz, H. L. F. (2011) · DFG Viewer (DFG Viewer) · Cited by 1,947 · see citing works
Context, time, and memory retrieval in the interference paradigms of Pavlovian learning
Bouton, M. E. (1993) · Psychological Bulletin · doi:10.1037/0033-2909.114.1.80 · Cited by 1,896 · see citing works
Effects of an expanding-spaced vs massed exposure schedule on fear reduction and return of fear
Rowe, M. K., & Craske, M. G. (1998) · Behaviour Research and Therapy · doi:10.1016/s0005-7967(97)10016-x · Cited by 1,778 · see citing works
Control of Cognitive Processes: Attention and Performance XVIII
Monsell, S., & Driver, J. (2000) · UCL Discovery (University College London) · Cited by 1,451 · see citing works
Perception as Bayesian Inference
(1996) · doi:10.1017/cbo9780511984037 · Cited by 1,425 · see citing works
Models of Information Processing in the Basal Ganglia
(1994) · doi:10.7551/mitpress/4708.001.0001 · Cited by 1,190 · see citing works
O’Reilly, R. C., & Frank, M. J. (2005) · Neural Computation · doi:10.1162/089976606775093909 · Cited by 1,066 · see citing works
Contextual control of the extinction of conditioned fear
Bouton, M. E., & Bolles, R. C. (1979) · Learning and Motivation · doi:10.1016/0023-9690(79)90057-2 · Cited by 796 · see citing works
Reinstatement of fear to an extinguished conditioned stimulus.
Rescorla, R. A., & Heth, C. D. (1975) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.1.1.88 · Cited by 768 · see citing works
Owen, A. M., Roberts, A. C., Hodges, J. R., & Robbins, T. W. (1993) · Brain · doi:10.1093/brain/116.5.1159 · Cited by 639 · see citing works
Dissociating working memory from task difficulty in human prefrontal cortex
Barch, D. M., Braver, T. S., Nystrom, L. E., Forman, S. D., Noll, D. C., & Cohen, J. D. (1997) · Neuropsychologia · doi:10.1016/s0028-3932(97)00072-9 · Cited by 591 · see citing works
A Model of How the Basal Ganglia Generate and Use Neural Signals That Predict Reinforcement
Houk, J. C., Adams, J. L., & Barto, A. G. (1994) · The MIT Press eBooks · doi:10.7551/mitpress/4708.003.0020 · Cited by 555 · see citing works
Cognition and control in schizophrenia: a computational model of dopamine and prefrontal function
Braver, T. S., Barch, D. M., & Cohen, J. D. (1999) · Biological Psychiatry · doi:10.1016/s0006-3223(99)00116-x · Cited by 507 · see citing works
Role of conditioned contextual stimuli in reinstatement of extinguished fear.
Bouton, M. E., & Bolles, R. C. (1979) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037//0097-7403.5.4.368 · Cited by 481 · see citing works
On the Control of Control: The Role of Dopamine in Regulating Prefrontal Function and Working Memory
(2000) · The MIT Press eBooks · doi:10.7551/mitpress/1481.003.0044 · Cited by 442 · see citing works
Prefrontal cortex and flexible cognitive control: Rules without symbols
Rougier, N. P., Noelle, D. C., Braver, T. S., Cohen, J. D., & O’Reilly, R. C. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0502455102 · Cited by 397 · see citing works
Context, learning, and extinction
Gershman, S. J., Blei, D. M., & Niv, Y. (2010) · Psychological Review · doi:10.1037/a0017808 · Cited by 386 · cited 1× · see citing works
“So far we have been dealing with static structure contained in the mosaic of sensory inputs; however, certain tasks require inferring dynamic structure from a sequence of sensory inputs.”
BackgroundSelf-citationRescorla, R. A. (2004) · Learning & Memory · doi:10.1101/lm.77504 · Cited by 380 · see citing works
Bush, W. T., & Mach, E. (1916) · The Journal of Philosophy Psychology and Scientific Methods · doi:10.2307/2012973 · Cited by 362 · see citing works
The probabilistic mind: prospects for Bayesian cognitive science
Chater, N., & Oaksford, M. (2008) · doi:10.1093/acprof:oso/9780199216093.001.0001 · Cited by 357 · see citing works
Redish, A. D., Jensen, S., Johnson, A., & Kurth-Nelson, Z. (2007) · Psychological Review · doi:10.1037/0033-295x.114.3.784 · Cited by 346 · see citing works
Models of information processing in the basal ganglia
Alexander, G. E. (1996) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0921-884x(96)89703-7 · Cited by 278 · see citing works
Structured statistical models of inductive reasoning
Kemp, C., & Tenenbaum, J. B. (2009) · Psychological Review · doi:10.1037/a0014282 · Cited by 259 · see citing works
Motor Task Variation Induces Structural Learning
Braun, D. A., Aertsen, A., Wolpert, D. M., & Mehring, C. (2009) · Current Biology · doi:10.1016/j.cub.2009.01.036 · Cited by 248 · see citing works
Mechanisms underlying spontaneous recovery in autoshaping.
Robbins, S. J. (1990) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.16.3.235 · Cited by 234 · see citing works
Leeson, V. C., Robbins, T. W., Matheson, E., Hutton, S. B., Ron, M. A., Barnes, T. R. E., & Joyce, E. M. (2009) · Biological Psychiatry · doi:10.1016/j.biopsych.2009.05.016 · Cited by 219 · see citing works
Braun, D. A., Mehring, C., & Wolpert, D. M. (2010) · Behavioural Brain Research · doi:10.1016/j.bbr.2009.08.031 · Cited by 210 · see citing works
Renewal of Extinguished Lever-Press Responses upon Return to the Training Context
Nakajima, S., Tanaka, S., Urushihara, K., & Imada, H. (2000) · Learning and Motivation · doi:10.1006/lmot.2000.1064 · Cited by 181 · see citing works
Electrolytic lesions of the dorsal hippocampus disrupt renewal of conditional fear after extinction.Preprint
Ji, J.-Z., & Maren, S. (2005) · Learning & memory (Cold Spring Harbor, N.Y.) · doi:10.1101/lm.91705 · Cited by 174 · see citing works
Representation and timing in theories of the dopamine system
Daw, N. D., Courville, A. C., & Touretzky, D. S. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1637 · Cited by 173 · see citing works
Gershman, S. J., Pesaran, B., & Daw, N. D. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2469-09.2009 · Cited by 118 · see citing works
Self-citationVul, E., Alvarez, G. A., Tenenbaum, J. B., & Black, M. J. (2009) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 114 · see citing works
Intra-dimensional/extra-dimensional set-shifting performance in schizophrenia: Impact of distractors
Jazbec, S., Pantelis, C., Robbins, T. W., Weickert, T. W., Weinberger, D. R., & Goldberg, T. E. (2006) · Schizophrenia Research · doi:10.1016/j.schres.2006.08.014 · Cited by 113 · see citing works
Set-Shifting Ability and Schizophrenia: A Marker of Clinical Illness or an Intermediate Phenotype?
Ceaser, A. E., Goldberg, T. E., Egan, M. F., McMahon, R. P., Weinberger, D. R., & Gold, J. M. (2008) · Biological Psychiatry · doi:10.1016/j.biopsych.2008.05.009 · Cited by 100 · see citing works
Learning Policies with External MemoryPreprint
Peshkin, L., Meuleau, N., & Kaelbling, L. P. (2001) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0103003 · Cited by 92 · see citing works
Brain Hemispheres Selectively Track the Expected Value of Contralateral Options
Palminteri, S., Boraud, T., Lafargue, G., Dubois, B., & Pessiglione, M. (2009) · Journal of Neuroscience · doi:10.1523/jneurosci.1500-09.2009 · Cited by 91 · see citing works
Extinction in the developing rat: an examination of renewal effects.
Yap, C., & Richardson, R. (2007) · Developmental Psychobiology · doi:10.1002/dev.20244 · Cited by 84 · see citing works
Spontaneous recovery varies inversely with the training-extinction interval
Rescorla, R. A. (2004) · Learning & Behavior · doi:10.3758/bf03196037 · Cited by 81 · see citing works
Daw, N. D., & Courville, A. C. (2007) · neural information processing systems · Cited by 77 · see citing works
Modeling the role of working memory and episodic memory in behavioral tasks
Zilli, E. A., & Hasselmo, M. E. (2007) · Hippocampus · doi:10.1002/hipo.20382 · Cited by 76 · see citing works
Todd, M. T., Niv, Y., & Cohen, J. D. (2008) · Cited by 69 · see citing works
Self-citationBilinearity, rules, and prefrontal cortex
Dayan, P. (2007) · Frontiers in Computational Neuroscience · doi:10.3389/neuro.10.001.2007 · Cited by 66 · see citing works
Structure Learning in Human Sequential Decision-Making
Acuna, D. E., & Schrater, P. (2010) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001003 · Cited by 60 · see citing works
Similarity and Discrimination in Classical Conditioning: A Latent Variable Account
Courville, A. C., Daw, N. D., & Touretzky, D. S. (2004) · Cited by 52 · see citing works
Model Uncertainty in Classical Conditioning
Courville, A. C., Gordon, G. J., Touretzky, D. S., & Daw, N. D. (2003) · Cited by 51 · see citing works
Perceptual Multistability as Markov Chain Monte Carlo Inference
Gershman, S. J., Vul, E., & Tenenbaum, J. B. (2009) · Cited by 34 · see citing works
Self-citationModeling Human Performance in Restless Bandits with Particle Filters
Yi, M. S. K., Steyvers, M., & Lee, M. (2009) · The Journal of Problem Solving · doi:10.7771/1932-6246.1060 · Cited by 32 · see citing works
Structure learning in human sequential decision-making
Schrater, P. (2009) · Frontiers in Systems Neuroscience · doi:10.3389/conf.neuro.06.2009.03.238 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [8]Doya K Ishii S Alexandre P Rao RPN Bayesian brain 2007 MIT Press
Cited by
Sorted by citations
Orbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885 · see citing works
Self-citationA Bayesian foundation for individual learning under uncertainty
Mathys, C. (2011) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2011.00039 · Cited by 655 · see citing works
Whittington, J. C. R., Muller, T. H., Mark, S., Chen, G., Barry, C., Burgess, N., & Behrens, T. E. J. (2020) · Cell · doi:10.1016/j.cell.2020.10.024 · Cited by 660 · see citing works
Human orbitofrontal cortex represents a cognitive map of state space
Schuck, N. W., Cai, M. B., Wilson, R. C., & Niv, Y. (2016) · Neuron · doi:10.1016/j.neuron.2016.08.019 · Cited by 602 · see citing works
Self-citationReinforcement Learning in Multidimensional Environments Relies on Attention Mechanisms
Niv, Y., Daniel, R., Geana, A., Gershman, S. J., Leong, Y. C., Radulescu, A., & Wilson, R. C. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.2978-14.2015 · Cited by 362 · see citing works
Self-citationDynamic Interaction between Reinforcement Learning and Attention in Multidimensional Environments
Leong, Y. C., Radulescu, A., Daniel, R., DeWoskin, V., & Niv, Y. (2017) · Neuron · doi:10.1016/j.neuron.2016.12.040 · Cited by 294 · see citing works
Self-citationExpectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex
Takahashi, Y. K., Roesch, M. R., Wilson, R. C., Toreson, K., O'Donnell, P., Niv, Y., & Schoenbaum, G. (2011) · Nature Neuroscience · doi:10.1038/nn.2957 · Cited by 280 · see citing works
Self-citationComputational approaches to psychiatry
Stephan, K. E., & Mathys, C. (2014) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2013.12.007 · Cited by 260 · see citing works
Attention, learning, and the value of information
Gottlieb, J. (2012) · Neuron · doi:10.1016/j.neuron.2012.09.034 · Cited by 241 · see citing works
“…studies makes it clear that processes of information selection tap into some of our highest cognitive functions, involving, among others, intrinsic curiosity and the ability for advance planning and forming internal models of complex tasks (e.g., Gershman and Niv, 2010; Johnson et al., 2012).”
BackgroundInfluentialActive inference on discrete state-spaces: A synthesis
Da Costa, L., Parr, T., Sajid, N., Veselic, S., Neacsu, V., & Friston, K. (2020) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2020.102447 · Cited by 226 · see citing works
The computational nature of memory modification
Gershman, S. J., Monfils, M.-H., Norman, K. A., & Niv, Y. (2017) · eLife · doi:10.7554/elife.23763 · Cited by 172 · see citing works
Self-citationThe Anterior Cingulate Cortex Predicts Future States to Mediate Model-Based Action Selection
Akam, T., Rodrigues-Vaz, I., Marcelo, I., Zhang, X., Pereira, M., Oliveira, R. F., Dayan, P., & Costa, R. M. (2021) · Neuron · doi:10.1016/j.neuron.2020.10.013 · Cited by 170 · see citing works
Meta-learning in natural and artificial intelligence
Wang, J. X. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.01.002 · Cited by 171 · see citing works
Model-based predictions for dopamine
Langdon, A. J., Sharpe, M. J., Schoenbaum, G., & Niv, Y. (2018) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.10.006 · Cited by 161 · see citing works
Self-citationSpatial attention, precision, and Bayesian inference: a study of saccadic response speed
Vossel, S., Mathys, C., Daunizeau, J., Bauer, M., Driver, J., Friston, K. J., & Stephan, K. E. (2014) · Cerebral Cortex · doi:10.1093/cercor/bhs418 · Cited by 146 · see citing works
Discovering latent causes in reinforcement learning
Gershman, S. J., Norman, K. A., & Niv, Y. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.07.007 · Cited by 151 · see citing works
Self-citationDoes mental context drift or shift?
DuBrow, S., Rouhani, N., Niv, Y., & Norman, K. A. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.08.003 · Cited by 118 · see citing works
Self-citationDeep temporal models and active inference
Friston, K. J., Rosch, R., Parr, T., Price, C., & Bowman, H. (2018) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2018.04.004 · Cited by 119 · see citing works
A probability distribution over latent causes, in the orbitofrontal cortex
Chan, S. C. Y., Niv, Y., & Norman, K. A. (2016) · The Journal of Neuroscience · doi:10.1523/jneurosci.0659-16.2016 · Cited by 97 · see citing works
Self-citationDegeneracy and Redundancy in Active Inference
Sajid, N., Parr, T., Hope, T. M., Price, C. J., & Friston, K. J. (2020) · Cerebral Cortex · doi:10.1093/cercor/bhaa148 · Cited by 49 · see citing works
Combining fMRI and behavioral measures to examine the process of human learning
Karuza, E. A., Emberson, L. L., & Aslin, R. N. (2014) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2013.09.012 · Cited by 50 · see citing works
Reward maximization through discrete active inference
Da Costa, L., Sajid, N., Parr, T., Friston, K., & Smith, R. (2023) · Neural Computation · doi:10.1162/neco_a_01574 · Cited by 35 · see citing works
Cultivating creativity: predictive brains and the enlightened room problem
Constant, A., Friston, K. J., & Clark, A. (2024) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2022.0415 · Cited by 19 · see citing works
- Authors
- Gershman, S. J., & Niv, Y.
- Affiliations
- Princeton University (US)
- Corresponding author
- Gershman, Samuel J
- Niv, Yael
- Venue
- Current Opinion in Neurobiology
- Publisher
- Elsevier BV
- Published
- 2010-04
- Volume
- 20
- Issue
- 2
- Pages
- 251-256
- ISSN
- 0959-4388 (ISSN-L)
- PMC
- PMC2862793
- OpenAlex
- W2133282234
- Cited by
- 326
- References
- 62
- Citations by year
- 2026: 29 · 2025: 28 · 2024: 35 · 2023: 28 · 2022: 18 · 2021: 35 · 2020: 26 · 2019: 17 · 2018: 15 · 2017: 32 · 2016: 17 · 2015: 12 · 2014: 9 · 2013: 10 · 2012: 9
- FWCI
- 1.84
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Current Opinion in Neurobiology (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health
- National Institute of Mental Health
- National Institute on Drug Abuse (T90DA022770)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Keywords
- structure learning, latent structure, reinforcement learning, learning speed, human learning, animal learning, perception, action, experiential learning
- MeSH terms
- Algorithms, Animals, Artificial Intelligence, Brain, Humans, Learning, Models, Neurological, Neuropsychology, Perception, Psychomotor Performance, Reinforcement, Psychology, Models, Animal
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
