Context, learning, and extinction
The DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
Abstract
A. Redish et al. (2007) proposed a reinforcement learning model of context-dependent learning and extinction in conditioning experiments, using the idea of "state classification" to categorize new observations into states. In the current article, the authors propose an interpretation of this idea in terms of normative statistical inference. They focus on renewal and latent inhibition, 2 conditioning paradigms in which contextual manipulations have been studied extensively, and show that online Bayesian inference within a model that assumes an unbounded number of latent causes can characterize a diverse set of behavioral results from such manipulations, some of which pose problems for the model of Redish et al. Moreover, in both paradigms, context dependence is absent in younger animals, or if hippocampal lesions are made prior to training. The authors suggest an explanation in terms of a restricted capacity to infer new causes.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Large prediction errors signal that the current latent cause is no longer relevant (i.e., the situation has changed) and a new or stored latent cause should be inferred.”
From Does mental context drift or shift? (2017) · cites this paper as [43]
In the citing paper
“According to a recent theory of latent cause inference, people build up a library of statistical models of their environment, and then infer which is the hidden, or “latent,” cause of their current sensory observations [43–46].”
From Does mental context drift or shift? (2017) · cites this paper as [43]
In the citing paper
“From a statistical perspective, superposition of latent causes is straightforward to model; in fact, existing latent cause models infer a continuous probability distribution over latent causes [43].”
From Does mental context drift or shift? (2017) · cites this paper as [43]
Citations
Cites
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 9,459
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
(1978) · UA Campus Repository (The University of Arizona) · Cited by 8,010
Introduction to the theory of neural computation
(1994) · Neural Networks · doi:10.1016/0893-6080(94)90014-0 · Cited by 6,401
McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995) · Psychological Review · doi:10.1037/0033-295x.102.3.419 · Cited by 5,256
Classical conditioning II: Current research and theory.
Black, A. H., & Prokasy, W. F. (1972) · Appleton-Century-Crofts eBooks · Cited by 4,449
Theoretical Neuroscience: Computational and Mathematical Modeling of Neural Systems
Dayan, P., & Abbott, L. F. (2001) · Cited by 4,099
Simple memory: a theory for archicortex
Marr, D. (1971) · Philosophical Transactions of the Royal Society of London. B, Biological Sciences · doi:10.1098/rstb.1971.0078 · Cited by 2,771 · cited 2×
“Our model may also shed new light on a long-standing question about the hippocampus and memory in general (Marr, 1971; McNaughton & Morris, 1987): When a new observation is made, under what circumstances is a new trace encoded or an old trace retrieved? Our model frames this as a choice between assigning an observatio…”
BackgroundMethod
Cited by
Computational psychiatry as a bridge from neuroscience to clinical applications
Huys, Q. J. M., Maia, T. V., & Frank, M. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4238 · Cited by 1,002
“Using models that allow for the learning of such latent variables provides a better account of standard extinction phenomena 134 and predicts that stable unlearning can in fact occur as long as there is no obvious sudden switch, which was verified experimentally 135 .”
Orbitofrontal Cortex as a Cognitive Map of Task Space
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y. (2014) · Neuron · doi:10.1016/j.neuron.2013.11.005 · Cited by 885
“Consequently, similar to the model of reversal learning, we modeled extinction with a twostate system (see also Gershman et al., 2010).”
InfluentialSelf-citationFrom reinforcement learning models to psychiatric and neurological disorders
Maia, T. V., & Frank, M. J. (2011) · Nature Neuroscience · doi:10.1038/nn.2723 · Cited by 722
“may be relevant for disorders that involve serotonergic abnormalities; and models of Pavlovian conditioning and extinctio”
BackgroundA tutorial on Bayesian nonparametric models
Gershman, S. J., & Blei, D. M. (2012) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2011.08.004 · Cited by 605
“A number of recent authors have extended this work (Griffiths et al., 2007; Heller et al., 2009; Sanborn et al., 2010) and applied it to other domains, such as classical conditioning (Gershman et al., 2010).”
Self-citationReasoning, Learning, and Creativity: Frontal Lobe Function and Human Decision-Making
Collins, A., & Koechlin, E. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001293 · Cited by 558
“new task sets for driving action. This issue raises a computational problem that statistical learning models based on Dirichlet process mixtures address”
BackgroundReinforcement learning and episodic memory in humans and animals: an integrative framework
Gershman, S. J., & Daw, N. D. (2017) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033625 · Cited by 446
“Related ideas have begun to be explored in RL tasks (Gershman et al. 2010, Niv et al. 2015, Vaidya & Fellows 2015).”
BackgroundSelf-citationRedish, A. D. (2016) · Nature Reviews Neuroscience · doi:10.1038/nrn.2015.30 · Cited by 479
Cognitive control over learning: Creating, clustering, and generalizing task-set structure.
Collins, A. G. E., & Frank, M. J. (2013) · Psychological Review · doi:10.1037/a0030852 · Cited by 426
“N N k ϭ ␣ ϩ N is a normalizing constant (Gershman et al., 2010).”
Foundations of human reasoning in the prefrontal cortex
Donoso, M., Collins, A. G. E., & Koechlin, E. (2014) · Science · doi:10.1126/science.1252254 · Cited by 406
Behavioral and neurobiological mechanisms of pavlovian and instrumental extinction learning
Bouton, M. E., Maren, S., & McNally, G. P. (2021) · Physiological Reviews · doi:10.1152/physrev.00016.2020 · Cited by 395
Learning task-state representations
Niv, Y. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0470-8 · Cited by 356
Self-citationOver the river, through the woods: cognitive maps in the hippocampus and orbitofrontal cortex
Wikenheiser, A. M., & Schoenbaum, G. (2016) · Nature Reviews Neuroscience · doi:10.1038/nrn.2016.56 · Cited by 359
Dunsmoor, J. E., Niv, Y., Daw, N., & Phelps, E. A. (2015) · Neuron · doi:10.1016/j.neuron.2015.09.028 · Cited by 329
Self-citationLearning latent structure: carving nature at its joints
Gershman, S. J., & Niv, Y. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.02.008 · Cited by 326
“So far we have been dealing with static structure contained in the mosaic of sensory inputs; however, certain tasks require inferring dynamic structure from a sequence of sensory inputs.”
BackgroundSelf-citationContextual modulation of value signals in reward and punishment learning
Palminteri, S., Khamassi, M., Joffily, M., & Coricelli, G. (2015) · Nature Communications · doi:10.1038/ncomms9096 · Cited by 286
“…the most parsimonious algorithm implementation of a model that, departing from experienced raw values, learns to identify, for each situation (context) the “best” and the “worst” possible outcomes, based on an explicit representation of the underlying generative process (the task structure) 49,50 .”
Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex
Takahashi, Y. K., Roesch, M. R., Wilson, R. C., Toreson, K., O'Donnell, P., Niv, Y., & Schoenbaum, G. (2011) · Nature Neuroscience · doi:10.1038/nn.2957 · Cited by 280
Self-citationDeep reinforcement learning and its neuroscientific implications
Botvinick, M., Wang, J. X., Dabney, W., Miller, K. J., & Kurth-Nelson, Z. (2020) · Neuron · doi:10.1016/j.neuron.2020.06.014 · Cited by 269
“…as in the machine learning context, however, RL itself has until recently offered neuroscience little guidance in thinking about the problem of representation (for discussion, see Botvinick et al., 2015; Wilson et al., 2014b; Stachenfeld et al., 2017; Behrens et al., 2018; Gershman et al., 2010).”
Contextual inference underlies the learning of sensorimotor repertoires
Heald, J. B., Lengyel, M., & Wolpert, D. M. (2021) · Nature · doi:10.1038/s41586-021-04129-3 · Cited by 238
“i=1 ∏ t∈T N (at|u t , σ(2) m), (47) where each u(i) ∼ p(u|η(i),x⋆, q, θ) was obtained by running the COIN model inference algorithm 906 conditioned on an observation noise sequence η(i) sampled from p(η|θ) = ∏T t=1N (ηt|0, σ2 r ).”
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236
Self-citationMesolimbic dopamine release conveys causal associations
Jeong, H., Taylor, A., Floeder, J. R., Lohmann, M., Mihalas, S., Wu, B., Zhou, M., Burke, D. A., & Namboodiri, V. M. K. (2022) · Science · doi:10.1126/science.abq6740 · Cited by 231
Retrospective revaluation in sequential decision making: a tale of two systems
Gershman, S. J., Markman, A. B., & Otto, A. R. (2014) · Journal of Experimental Psychology General · doi:10.1037/a0030844 · Cited by 217
Self-citationDopamine reward prediction errors reflect hidden-state inference across time
Starkweather, C. K., Babayan, B. M., Uchida, N., & Gershman, S. J. (2017) · Nature Neuroscience · doi:10.1038/nn.4520 · Cited by 206
“However, in the real world, stimuli often provide ambiguous information about states; the true underlying states are “hidden” and must therefore be inferred 6,7 .”
Self-citationThe successor representation: Its computational logic and neural substrates
Gershman, S. J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0151-18.2018 · Cited by 191
“Some theories posit that the brain forms a posterior distribution over hidden states conditional on sensory data (Daw et al., 2006; Gershman et al., 2010; Rao, 2010; Soto et al., 2014; Babayan et al., 2018; Starkweather et al., 2018), in which case the SR would need to be defined over the…”
BackgroundSelf-citationThe Functional Anatomy of Time: What and When in the Brain
Friston, K., & Buzsáki, G. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.05.001 · Cited by 183
“See [60,61] for related treatments of context in a Bayesian setting.”
BackgroundAbstract context representations in primate amygdala and prefrontal cortex
Saez, A., Rigotti, M., Ostojic, S., Fusi, S., & Salzman, C. D. (2015) · Neuron · doi:10.1016/j.neuron.2015.07.024 · Cited by 183
“…of internal states corresponding to external hidden variables like context are available, an agent might mimic goal-directed instrumental responses by implementing a reactive habitual policy over these internal states (Gershman et al., 2010; Redish et al., 2007; Rigotti et al., 2010).”
BackgroundHippocampal remapping as hidden state inference
Sanders, H., Wilson, M. A., & Gershman, S. J. (2020) · eLife · doi:10.7554/elife.51140 · Cited by 163
Self-citationThe computational nature of memory modification
Gershman, S. J., Monfils, M.-H., Norman, K. A., & Niv, Y. (2017) · eLife · doi:10.7554/elife.23763 · Cited by 172
“…which outputs a parse of experience into latent causes—hypothetical entities in the environment that govern the distribution of stimulus configurations ( Courville, 2006 ; Courville et al., 2006 ; Gershman et al., 2010 ; Gershman and Niv, 2012 ; Gershman et al., 2013a , 2015 ; Soto et al., 2014 ).”
BackgroundMethodResultInfluentialSelf-citationA Unifying Probabilistic View of Associative Learning
Gershman, S. J. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004567 · Cited by 166
“Other models derive stimulus representation from a clustering process that partitions stimulus inputs into a discrete set of states [7, 71, 74, 75].”
BackgroundSelf-citationA model for learning based on the joint estimation of stochasticity and volatility
Piray, P., & Daw, N. D. (2021) · Nature Communications · doi:10.1038/s41467-021-26731-9 · Cited by 153
“…and conditioning, which has grounded biological plasticity mechanisms (specifically, error-driven updating) in terms of a normative analysis of the problem faced by the organism (Courville et al., 2006; Daunizeau et al., 2010; Dayan and Long, 1998; Dayan et al., 2000; Gershman et al., 2010).”
BackgroundReliance on small samples, the wavy recency effect, and similarity-based learning.
Plonsky, O., Teodorescu, K., & Erev, I. (2015) · Psychological Review · doi:10.1037/a0039413 · Cited by 162
Hippocampal Contributions to Model-Based Planning and Spatial Memory
Vikbladh, O. M., Meager, M. R., King, J., Blackmon, K., Devinsky, O., Shohamy, D., Burgess, N., & Daw, N. D. (2019) · Neuron · doi:10.1016/j.neuron.2019.02.014 · Cited by 161
“, 2015); this might also implicate hippocampus for other, confounding reasons, such as its involvement in trace conditioning and latent states (Solomon et al., 1986; B€ uchel et al., 1999; Gershman et al., 2010).”
Gradual extinction prevents the return of fear: implications for the discovery of state
Gershman, S. J., Jones, C. E., Norman, K. A., Monfils, M.-H., & Niv, Y. (2013) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2013.00164 · Cited by 160
“Our results fit well with emerging theoretical ideas about the role of state discovery in Pavlovian conditioning and other learning paradigms (Redish et al., 2007; Gershman et al., 2010; Gershman and Niv, 2012).”
InfluentialSelf-citationExploring a latent cause theory of classical conditioning
Gershman, S. J., & Niv, Y. (2012) · Learning & Behavior · doi:10.3758/s13420-012-0080-8 · Cited by 149
“Renewal In Gershman et al. (2010), we presented an extensive discussion of renewal effects.”
InfluentialSelf-citationDiscovering latent causes in reinforcement learning
Gershman, S. J., Norman, K. A., & Niv, Y. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.07.007 · Cited by 151
“By positing that animals infer the existence of different latent causes during conditioning and extinction, the latent cause model can explain both why and when the conditioned response survives extinction [8].”
BackgroundInfluentialSelf-citationThe medial prefrontal cortex shapes dopamine reward prediction errors under state uncertainty
Starkweather, C. K., Gershman, S. J., & Uchida, N. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.036 · Cited by 136
“That is, the current state is ‘‘hidden’’ and needs to be inferred from partial information (Courville et al., 2006; Gershman et al., 2010, 2015).”
Self-citationDoes mental context drift or shift?
DuBrow, S., Rouhani, N., Niv, Y., & Norman, K. A. (2017) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2017.08.003 · Cited by 118
“From a statistical perspective, superposition of latent causes is straightforward to model; in fact, existing latent cause models infer a continuous probability distribution over latent causes [43].”
Self-citationAbove and beyond the concrete: The diverse representational substrates of the predictive brain
Gilead, M., Trope, Y., & Liberman, N. (2020) · Behavioral and Brain Sciences · doi:10.1017/s0140525x19002000 · Cited by 133
Taking stock of value in the orbitofrontal cortex
Knudsen, E. B., & Wallis, J. D. (2022) · Nature reviews. Neuroscience · doi:10.1038/s41583-022-00589-2 · Cited by 157
Human memory reconsolidation can be explained using the temporal context model
Sederberg, P. B., Gershman, S. J., Polyn, S. M., & Norman, K. A. (2011) · Psychonomic Bulletin & Review · doi:10.3758/s13423-011-0086-9 · Cited by 125
“When the model infers (based on the reminder) that the previously-active “list A” cause is present, we should see updating of the original list A memory (Gershman et al., 2010).”
BackgroundSelf-citationBeyond dichotomies in reinforcement learning
Collins, A. G. E., & Cockburn, J. (2020) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-0355-6 · Cited by 125
Moustafa, A. A., Gilbertson, M. W., Orr, S. P., Herzallah, M. M., Servatius, R. J., & Myers, C. E. (2012) · Brain and Cognition · doi:10.1016/j.bandc.2012.10.005 · Cited by 146
Structuring Memory Through Inference‐Based Event Segmentation
Shin, Y. S., & DuBrow, S. (2021) · Topics in Cognitive Science · doi:10.1111/tops.12505 · Cited by 119
Reconsidering the evidence for learning in single cells
Gershman, S. J., Balbi, P. E., Gallistel, C. R., & Gunawardena, J. (2021) · eLife · doi:10.7554/elife.61907 · Cited by 153
Self-citationFear conditioning and extinction: emotional states encoded by distinct signaling pathways
Tronson, N. C., Corcoran, K. A., Jovasevic, V., & Radulovic, J. (2012) · Trends in Neurosciences · doi:10.1016/j.tins.2011.10.003 · Cited by 121
“RLSC describes extinction as a value change and reinterpretation of the CS [25, 29].”
Pavlovian valuation systems in learning and decision making
Clark, J. J., Hollon, N. G., & Phillips, P. E. (2012) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2012.06.004 · Cited by 121
“Indeed, theoretical work has provided computational accounts for such state representation and its influence on Pavlovian conditioning not generally captured by previous modelfree reinforcement learning algorithms [70,71].”
BackgroundControllability governs the balance between Pavlovian and instrumental action selection
Dorfman, H. M., & Gershman, S. J. (2019) · Nature Communications · doi:10.1038/s41467-019-13737-7 · Cited by 142
Self-citationSharpe, M. J., Stalnaker, T., Schuck, N. W., Killcross, S., Schoenbaum, G., & Niv, Y. (2019) · Annual Review of Psychology · doi:10.1146/annurev-psych-010418-102824 · Cited by 116
“Rather, they will encode the two contingencies as two different states of the task (i.e., state A, in which R1–O1 and R2–nothing are true, and state B, in which R1–nothing and R2–O1 are true) (Gershman et al. 2010, Wilson et al. 2014).”
Self-citationLearning the Structure of Social Influence
Gershman, S. J., Pouncy, T., & Gweon, H. (2017) · Cognitive Science · doi:10.1111/cogs.12480 · Cited by 120
Self-citationA probability distribution over latent causes, in the orbitofrontal cortex
Chan, S. C. Y., Niv, Y., & Norman, K. A. (2016) · The Journal of Neuroscience · doi:10.1523/jneurosci.0659-16.2016 · Cited by 97
“This inference can be concretely formulated in terms of Bayesian latent cause models (e.g., Gershman et al., 2010).”
BackgroundMethodSelf-citationToward the neural implementation of structure learning
Tervo, D. G. R., Tenenbaum, J. B., & Gershman, S. J. (2016) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2016.01.014 · Cited by 106
“For more than a century, the challenge of determining how structure learning and inductive inference can be efficiently performed was tackled by statisticians [1], linguists [2], computer scientists [3] and cognitive scientists [4–7].”
BackgroundSelf-citationBelief state representation in the dopamine system
Babayan, B. M., Uchida, N., & Gershman, S. J. (2018) · Nature Communications · doi:10.1038/s41467-018-04397-0 · Cited by 101
Self-citationStatistical computations underlying the dynamics of memory updating
Gershman, S. J., Radulescu, A., Norman, K. A., & Niv, Y. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003939 · Cited by 101
“This prior has previously been used to model category formation [34,35], Pavlovian conditioning in multiple contexts [24,45], word segmentation [46] and task-set learning [47] (for a review of this literature, see [48]).”
MethodSelf-citationContextual inference in learning and memory
Heald, J. B., Lengyel, M., & Wolpert, D. M. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.10.004 · Cited by 95
Novelty and Inductive Generalization in Human Reinforcement Learning
Gershman, S. J., & Niv, Y. (2015) · Topics in Cognitive Science · doi:10.1111/tops.12138 · Cited by 98
“…of Bayesian statistics, which has received considerable support as the basis of human inductive inferences (Griffiths et al., 2010), and has been applied to RL in a number of previous investigations (Kakade and Dayan, 2002a; Courville et al., 2006; Behrens et al., 2007; Gershman et al., 2010).”
Self-citationMultiplicity of control in the basal ganglia: computational roles of striatal subregions
Bornstein, A. M., & Daw, N. D. (2011) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2011.02.009 · Cited by 98
“Last, since the perceptual system, on this view, must in general base its percepts on learning about the statistical structure of the task and percepts, this idea situates the RL circuit alongside substantial recent work on Bayesian models of learning latent structure [87–90].”
Laboratory models of post-traumatic stress disorder: The elusive bridge to translation
Dunsmoor, J. E., Cisler, J. M., Fonzo, G. A., Creech, S. K., & Nemeroff, C. B. (2022) · Neuron · doi:10.1016/j.neuron.2022.03.001 · Cited by 104
Redish, A. D., & Mizumori, S. J. Y. (2014) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2014.08.014 · Cited by 108
Franklin, N. T., & Frank, M. J. (2015) · eLife · doi:10.7554/elife.12029 · Cited by 95
“, 2010) or infer a change in a latent context or rule-structure (Gershman et al., 2010; Donoso et al., 2014).”
BackgroundThe role of learning-related dopamine signals in addiction vulnerability
Huys, Q. J. M., Tobler, P. N., Hasler, G., & Flagel, S. B. (2014) · Progress in brain research · doi:10.1016/b978-0-444-63425-2.00003-9 · Cited by 107
“As reviewed above, extinction does not lead to unlearning, but rather to the re-engagement of prefrontal cortices and novel learning (Bouton, 2004; Gershman et al., 2010).”
BackgroundHuman fear conditioning: From neuroscience to the clinic
Fullana, M. A., Dunsmoor, J. E., Schruers, K. R. J., Savage, H. S., Bach, D. R., & Harrison, B. J. (2020) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.103528 · Cited by 88
Collins, A. G. E., & Frank, M. J. (2016) · Cognition · doi:10.1016/j.cognition.2016.04.002 · Cited by 96
“Our model used the ''Chinese restaurant prior " , building on existing models of conditioning (Gershman et al., 2010).”
The Dopamine Prediction Error: Contributions to Associative Models of Reward Learning
Nasser, H. M., Calu, D. J., Schoenbaum, G., & Sharpe, M. J. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00244 · Cited by 86
“, contextual cues) available on that trial (Nakahara et al., 2004; Redish et al., 2007; Gershman et al., 2010; Nakahara and Hikosaka, 2012; Nakahara, 2014), basically anything that has been directly experienced as associated with reward in the past.”
BackgroundInfluentialA probabilistic clustering theory of the organization of visual short-term memory.
Orhan, A. E., & Jacobs, R. A. (2013) · Psychological Review · doi:10.1037/a0031541 · Cited by 88
A feature-specific prediction error model explains dopaminergic heterogeneity
Lee, R. S., Sagiv, Y., Engelhard, B., Witten, I. B., & Daw, N. D. (2024) · Nature Neuroscience · doi:10.1038/s41593-024-01689-1 · Cited by 100
Powell, N. J., & Redish, A. D. (2016) · Nature Communications · doi:10.1038/ncomms12830 · Cited by 98
Soto, F. A., Gershman, S. J., & Niv, Y. (2014) · Psychological Review · doi:10.1037/a0037018 · Cited by 81
“On the other hand, the number of causes that could possibly be present (K) is not known in advance, as in the model presented by Gershman et al. (2010).”
Self-citationHuys, Q. J. M., Guitart-Masip, M., Dolan, R. J., & Dayan, P. (2015) · Clinical Psychological Science · doi:10.1177/2167702614562040 · Cited by 80
The Computational and Neural Bases of Context-Dependent Learning
Heald, J. B., Wolpert, D. M., & Lengyel, M. (2023) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-092322-100402 · Cited by 102
Evidence for model-based computations in the human amygdala during Pavlovian conditioning
Prévost, C., McNamee, D., Jessup, R. K., Bossaerts, P., & O'Doherty, J. P. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002918 · Cited by 79
Posttraumatic Stress Disorder and the Developing Adolescent Brain
Cisler, J. M., & Herringa, R. J. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2020.06.001 · Cited by 96
Abstract task representations for inference and control
Vaidya, A. R., & Badre, D. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.03.009 · Cited by 87
More Than the Sum of Its Parts: A Role for the Hippocampus in Configural Reinforcement Learning
Duncan, K., Doll, B. B., Daw, N. D., & Shohamy, D. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.042 · Cited by 75
“…predictive of weather outcomes. configurations and elements (Melchers et al., 2008) as well as computational frameworks that emphasize the use of structure in learning (e.g., Gershman et al., 2010), little is known about how the brain supports reinforcement learning when the cues are configural.”
BackgroundMethodInfluentialPutting bandits into context: How function learning supports decision making.
Schulz, E., Konstantinidis, E., & Speekenbrink, M. (2017) · Journal of Experimental Psychology Learning Memory and Cognition · doi:10.1037/xlm0000463 · Cited by 87
A neural network model of when to retrieve and encode episodic memories
Lu, Q., Hasson, U., & Norman, K. A. (2022) · eLife · doi:10.7554/elife.74445 · Cited by 96
“Building on prior work on nonparametric Bayesian inference ( Anderson, 1991 ; Aldous, 1983 ; Pitman, 2006 ) and latent cause modeling ( Gershman et al., 2010 ; Gershman et al., 2015 ), SEM contains specialized computational machinery that allows it to determine which of its stored schemas (each…”
MethodFundamental processes in sensorimotor learning: Reasoning, refinement, and retrieval
Tsay, J. S., Kim, H. E., McDougle, S. D., Taylor, J. A., Haith, A. M., Avraham, G., Krakauer, J. W., Collins, A. G. E., & Ivry, R. B. (2024) · eLife · doi:10.7554/elife.91839 · Cited by 98
Stoianov, I. P., Maisto, D., & Pezzulo, G. (2022) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2022.102329 · Cited by 79
Wang, Q., Jin, J., & Maren, S. (2016) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2016.04.002 · Cited by 69
“, & Quirk (2012). Gating of Fear in Prelimbic Cortex by Hippocampal and Amygdala Inputs. Neuron, 76, 804-812. Tierney, P. L., Dégenètais, E., Thierry, A.-M., Glowinski, J., & Gioanni, Y. (2004). Influence of the hippocampus on interneurons of the rat prefrontal cortex. The European Journal Of Neuroscience, 20, 514-524…”
InfluentialDissociable forms of uncertainty-driven representational change across the human brain
Nassar, M. R., McGuire, J. T., Ritz, H., & Kable, J. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.1713-18.2018 · Cited by 68
“Although previous work has focused primarily on the advantage of such representations for rapid reinstatement of previously learned behaviors (Gershman et al., 2010; Wilson et al., 2014), another advantage of such representations is that they could facilitate rapid disengagement from established behaviors that are no…”
BackgroundEpisodic memory and Pavlovian conditioning: ships passing in the night
Dunsmoor, J. E., & Kroes, M. C. W. (2018) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2018.09.019 · Cited by 82
Bradfield, L. A., & Hart, G. (2020) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2019.11.009 · Cited by 67
“RR) between contexts required the formation of two different states to disambiguate these different contingencies (30, 31) (e.”
BackgroundIndividual learning phenotypes drive collective behavior
Cook, C. N., Lemanski, N. J., Mosqueiro, T. S., Öztürk, C., Gadau, J. R., Pinter‐Wollman, N., & Smith, B. H. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1920554117 · Cited by 70
“This interpretation is an extension of conditioned attention theory (31, 39), which proposes that latent inhibition is induced by allowing animals to focus their attention on important information (30, 31, 40).”
BackgroundRadulescu, A., Shin, Y. S., & Niv, Y. (2021) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-092920-120559 · Cited by 63
“…has helped unify a broad range of phenomena in psychology and neuroscience, most notably fear conditioning, task set learning, categorization, and event segmentation (Anderson 1991; Collins & Frank 2013; Franklin et al. 2020; Gershman et al. 2010, 2017; Sanborn et al. 2010; Shin & DuBrow 2021).”
Self-citationAge-dependent changes in behavioural plasticity: insights from Bayesian models of development
Stamps, J. A., & Krishnan, V. V. (2017) · Animal Behaviour · doi:10.1016/j.anbehav.2017.01.013 · Cited by 67
Sadacca, B. F., Wikenheiser, A. M., & Schoenbaum, G. (2017) · Neuroscience · doi:10.1016/j.neuroscience.2016.04.017 · Cited by 62
“…states similar to the role we describe for OFC here, and Redish and colleagues have a prominent model (Redish et al., 2007), extended by others (Gershman et al., 2010), suggesting low levels of dopamine in the hippocampus might produce the state-creating effects we attribute to rising tonic NE…”
Gamelike features might not improve data
Hawkins, G. E., Rae, B., Nesbitt, K. V., & Brown, S. D. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0264-3 · Cited by 61
Effects of pharmacological manipulations of NMDA-receptors on deliberation in the Multiple-T task
Blumenthal, A., Steiner, A. P., Seeland, K. D., & Redish, A. D. (2011) · Neurobiology of Learning and Memory · doi:10.1016/j.nlm.2011.01.011 · Cited by 63
“, 2010; Langton and Richardson, 2010), which can be seen as an ability to recognize a change in reward-contingency and react to it (Capaldi and Lynch, 1968; Bouton, 2004; Redish et al., 2007; Gershman et al., 2010).”
Finding structure in multi-armed bandits
Schulz, E., Franklin, N. T., & Gershman, S. J. (2018) · Cognitive Psychology · doi:10.1016/j.cogpsych.2019.101261 · Cited by 74
Self-citationCategorization=decision making+generalization
Seger, C. A., & Peterson, E. J. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.03.015 · Cited by 76
“The development of a more realistic approach to modeling states is an active area of research, including methods for creating new states when needed (Gershman et al., 2010; Redish et al., 2007) and ways of disambiguating perceptually overlapping states (Gureckis and Love, 2009).”
MethodBradfield, L. A., & Balleine, B. W. (2017) · Journal of Neuroscience · doi:10.1523/jneurosci.3860-16.2017 · Cited by 65
“…recent findings (Im et al., 2016; Thrailkill et al., 2016), is that internal and external contexts (or states) can be at least partially dissociated by their ability to govern action selection, as predicted by certain reinforcement learning models (Gershman et al., 2010; Gershman and Niv, 2012).”
BackgroundIndividual differences in learning predict the return of fear
Gershman, S. J., & Hartley, C. A. (2015) · Learning & Behavior · doi:10.3758/s13420-015-0176-z · Cited by 64
“We obtained a finer-grained characterization of responses during the acquisition and extinction phases by fitting a computational model of learning (the latent cause model; Gershman et al., 2010; Gershman & Niv 2012) to each participant’s data.”
Self-citationTriple contagion: a two-fears epidemic model
Epstein, J. M., Hatna, E., & Crodelle, J. A. (2021) · Journal of The Royal Society Interface · doi:10.1098/rsif.2021.0186 · Cited by 84
“Both the fear acquisition phase and fear extinction phases can be modelled mathematically [51].”
BackgroundAdaptive learning is structure learning in time
Yu, L. Q., Wilson, R. C., & Nassar, M. R. (2021) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2021.06.024 · Cited by 68
Context-dependent learning and causal structure
Gershman, S. J. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1110-x · Cited by 54
“Of particular relevance is the model developed by Gershman et al. (2010), which explained context-dependent renewal as the consequence of inferring a structure in which the acquisition and extinction trials were generated by distinct latent causes.”
Self-citationPrefrontal mechanisms combining rewards and beliefs in human decision-making
Rouault, M., Drugowitsch, J., & Koechlin, É. (2019) · Nature Communications · doi:10.1038/s41467-018-08121-w · Cited by 67
Wasserman, E. A., Brooks, D. I., & McMurray, B. (2014) · Cognition · doi:10.1016/j.cognition.2014.11.020 · Cited by 60
Context-dependent decision-making: a simple Bayesian model
Lloyd, K., & Leslie, D. S. (2013) · Journal of The Royal Society Interface · doi:10.1098/rsif.2013.0069 · Cited by 52
“Following previous psychological models [10,11,19,20], we assume a sample-based approximation in which contexts are sequentially sampled on a trial-by-trial basis (i.”
BackgroundMethodResultInfluentialMnemonic prediction errors promote detailed memories
Bein, O., Plotkin, N. A., & Davachi, L. (2021) · Learning & Memory · doi:10.1101/lm.053410.121 · Cited by 60
“For example, prediction errors were postulated to cause separation in category learning (McClelland et al. 1995; Love et al. 2004; Davis et al. 2012) and reinforcement learning and state representations (Gershman and Niv 2010; Gershman et al. 2010, 2014, 2017).”
BackgroundExtinction from a rationalist perspective
Gallistel, C. R. (2012) · Behavioural Processes · doi:10.1016/j.beproc.2012.02.008 · Cited by 57
“If the world has many such variables, then when one perceptible change in a state of the world is coincident with another, it becomes probable that there exists an underlying causal connection (cf. Courville et al., 2004; Gershman et al., 2010).”
Transdiagnostic computations of uncertainty: towards a new lens on intolerance of uncertainty
Sandhu, T. R., Xiao, B., & Lawson, R. P. (2023) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2023.105123 · Cited by 51
Hidden state inference requires abstract contextual representations in the ventral hippocampus
Mishchanchuk, K., Gregoriou, G. C., Qü, A. J., Kastler, A., Huys, Q. J. M., Wilbrecht, L., & MacAskill, A. F. (2024) · Science · doi:10.1126/science.adq5874 · Cited by 70
Metadata
- Authors
- Gershman, S. J., Blei, D. M., & Niv, Y.
- Affiliations
- Princeton University (US)
- Venue
- Psychological Review
- Published
- 2010
- Volume
- 117
- Issue
- 1
- Pages
- 197-209
- ISSN
- 0033-295X (ISSN-L)
- 1939-1471
- OpenAlex
- W2086710210
- Cited by
- 386
- References
- 74
- Citations by year
- 2026: 26 · 2025: 29 · 2024: 38 · 2023: 27 · 2022: 24 · 2021: 33 · 2020: 22 · 2019: 32 · 2018: 21 · 2017: 26 · 2016: 29 · 2015: 17 · 2014: 15 · 2013: 16 · 2012: 15
- FWCI
- 8.79
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Psychological Review (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Google and Microsoft Research
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Keywords
- context-dependent learning, extinction, reinforcement learning, state classification, Bayesian inference, latent inhibition, renewal, hippocampus, age differences
- MeSH terms
- Bayes Theorem, Conditioning, Classical, Extinction, Psychological, Hippocampus, Humans, Learning, Models, Psychological, Semantics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.