Bayesian theories of conditioning in a changing world
Abstract
The recent flowering of Bayesian approaches invites the re-examination of classic issues in behavior, even in areas as venerable as Pavlovian conditioning. A statistical account can offer a new, principled interpretation of behavior, and previous experiments and theories can inform many unexplored aspects of the Bayesian enterprise. Here we consider one such issue: the finding that surprising events provoke animals to learn faster. We suggest that, in a statistical account of conditioning, surprise signals change and therefore uncertainty and the need for new learning. We discuss inference in a world that changes and show how experimental results involving surprise can be interpreted from this perspective, and also how, thus understood, these phenomena help constrain statistical theories of animal and human learning.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In Active Inference, generative models are latent cause models that reconstruct the hidden processes jointly responsible for context, sensory and reinforcing events.
“In terms of Active Inference, this implies that the generative models not only encode associations between stimuli or between conditioned stimuli and reward delivery; rather, they are latent cause models that reconstruct the hidden processes jointly responsible for context, sensory and reinforcing events (Courville et al., 2006).”
From Active Inference, homeostatic regulation and adaptive behavioural control (2015) · cites this paper as Courville et al., 2006
Cites
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 7,268 · see citing works
Pearce, J. M., & Hall, G. (1980) · Psychological Review · doi:10.1037/0033-295x.87.6.532 · Cited by 3,023 · see citing works
A theory of attention: Variations in the associability of stimuli with reinforcement
Mackintosh, N. J. (1975) · Psychological Review · doi:10.1037/h0076778 · Cited by 2,733 · see citing works
Conditioned Reflexes: An Investigation of the Physiological Activity of the Cerebral Cortex.
Pavlov, P. I. (1930) · Archives of Neurology And Psychiatry · doi:10.1001/archneurpsyc.1930.02220110256018 · Cited by 2,681 · see citing works
Conditioned Reflexes: an Investigation of the Physiological Activity of the Cerebral Cortex
Denny-Brown, D. E. (1928) · Nature · doi:10.1038/121662a0 · Cited by 1,854 · see citing works
Uncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774 · see citing works
Gallistel, C. R., & Gibbon, J. (2000) · Psychological Review · doi:10.1037/0033-295x.107.2.289 · Cited by 1,122 · see citing works
Latent inhibition: The effect of nonreinforced pre-exposure to the conditional stimulus.
Lubow, R. E., & Moore, A. U. (1959) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0046700 · Cited by 806 · see citing works
Lubow, R. E. (1973) · Psychological Bulletin · doi:10.1037/h0034425 · Cited by 793 · see citing works
The learning curve: implications of a quantitative analysis.
Gallistel, C., Fairhurst, S. P., & Balsam, P. (2004) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.0404965101 · Cited by 587 · see citing works
Learning and selective attention
Dayan, P., Kakade, S., & Montague, P. R. (2000) · Nature Neuroscience · doi:10.1038/81504 · Cited by 495 · see citing works
Le Pelley, M. E. (2004) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/02724990344000141 · Cited by 360 · see citing works
Gallistel, C. R., Mark, T. A., King, A. P., & Latham, P. E. (2001) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037//0097-7403.27.4.354 · Cited by 225 · see citing works
The strength of the orienting response during Pavlovian conditioning.
Kaye, H., & Pearce, J. M. (1984) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.10.1.90 · Cited by 209 · see citing works
Gallistel, C. R., Mark, T. A., King, A. P., et al. (2001) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.27.4.354 · Cited by 174 · see citing works
Latent inhibition of a CS during CS–US pairings.
Hall, G., & Pearce, J. M. (1979) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.5.1.31 · Cited by 173 · see citing works
Gewirtz, J. C., & Davis, M. E. (2000) · Learning & Memory · doi:10.1101/lm.35200 · Cited by 168 · see citing works
Partial reinforcement in autoshaping with pigeons
Gibbon, J., Farrell, L., Locurto, C., Duncan, H. J., & Terrace, H. S. (1980) · Animal Learning & Behavior · doi:10.3758/bf03209729 · Cited by 145 · see citing works
Structure Learning in Human Causal Induction
Tenenbaum, J. B., & Griffiths, T. L. (2000) · Cited by 136 · see citing works
A theory of pavlovian conditioning: The effectiveness of reinforcement and non-reinforcement
Rescorla, R. A. (1972) · Medical Entomology and Zoology · Cited by 108 · see citing works
Restoration of the orienting response to a light by a change in its predictive accuracy
Wilson, P. N., Boumphrey, P., & Pearce, J. M. (2007) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/02724999208250600 · Cited by 103 · see citing works
Acquisition and extinction in autoshaping.
Kakade, S. M., & Dayan, P. (2002) · Psychological Review · doi:10.1037/0033-295x.109.3.533 · Cited by 102 · see citing works
Haselgrove, M., Aydin, A., & Pearce, J. M. (2004) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.30.3.240 · Cited by 85 · see citing works
Holland, P. (1997) · Animal Learning & Behavior · doi:10.3758/bf03209846 · Cited by 84 · see citing works
Overshadowing and latent inhibition counteract each other: Support for the comparator hypothesis.
Blaisdell, A. P., Bristol, A. S., Gunther, L. M., & Miller, R. R. (1998) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.24.3.335 · Cited by 83 · see citing works
Explaining Away in Weight Space
Dayan, P., & Kakade, S. M. (2000) · ScholarlyCommons (University of Pennsylvania) · Cited by 61 · see citing works
WITHIN-SUBJECT PARTIAL REINFORCEMENT EXTINCTION EFFECT IN AUTOSHAPING
Rescorla, R. A. (2010) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/713932693 · Cited by 59 · see citing works
Restoring the Associability of a Pre-Exposed CS by a Surprising Event
Hall, G., & Pearce, J. M. (1982) · The Quarterly Journal of Experimental Psychology Section B · doi:10.1080/14640748208400881 · Cited by 59 · see citing works
Partial reinforcement in appetitive Pavlovian conditioning with rats.
Pearce, J. M., Redhead, E. S., & Aydin, A. (1997) · PubMed · doi:10.1080/713932660 · Cited by 52 · see citing works
Similarity and Discrimination in Classical Conditioning: A Latent Variable Account
Courville, A. C., Daw, N. D., & Touretzky, D. S. (2004) · Cited by 52 · see citing works
Self-citationModel Uncertainty in Classical Conditioning
Courville, A. C., Gordon, G. J., Touretzky, D. S., & Daw, N. D. (2003) · Cited by 51 · see citing works
Self-citationStatistical Models of Conditioning
Dayan, P., & Long, T. W. (1997) · Cited by 34 · see citing works
ACh, Uncertainty, and Cortical Inference
Dayan, P., & Yu, A. J. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0029 · Cited by 26 · see citing works
Partial reinforcement reduces the associative change produced by nonreinforcement.
Rescorla, R. A. (1999) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037//0097-7403.25.4.403 · Cited by 24 · see citing works
Partial reinforcement reduces the associative change produced by nonreinforcement.
Rescorla, R. A. (1999) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.25.4.403 · Cited by 19 · see citing works
Unblocking with Qualitative Change of Unconditioned Stimulus
Blaisdell, A. P., Denniston, J. C., & Miller, R. R. (1997) · Learning and Motivation · doi:10.1006/lmot.1996.0961 · Cited by 18 · see citing works
Modeling Temporal Structure in Classical Conditioning
Courville, A. C., & Touretzky, D. S. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.003.0005 · Cited by 8 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053 · see citing works
Perceiving is believing: a Bayesian approach to explaining the positive symptoms of schizophrenia
Fletcher, P. C., & Frith, C. D. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2536 · Cited by 1,462 · see citing works
Precise minds in uncertain worlds: Predictive coding in autism.
Van de Cruys, S., Evers, K., Van der Hallen, R., Van Eylen, L., Boets, B., de-Wit, L., & Wagemans, J. (2014) · Psychological Review · doi:10.1037/a0037665 · Cited by 885 · see citing works
Associative learning of social value
Behrens, T. E. J., Hunt, L. T., Woolrich, M. W., & Rushworth, M. F. S. (2008) · Nature · doi:10.1038/nature07538 · Cited by 925 · see citing works
Choice, uncertainty and value in prefrontal and cingulate cortex
Rushworth, M. F. S., & Behrens, T. E. J. (2008) · Nature Neuroscience · doi:10.1038/nn2066 · Cited by 866 · see citing works
Spontaneous Cortical Activity Reveals Hallmarks of an Optimal Internal Model of the Environment
Berkes, P., Orbán, G., Lengyel, M., & Fiser, J. (2011) · Science · doi:10.1126/science.1195870 · Cited by 782 · see citing works
Active Inference, homeostatic regulation and adaptive behavioural control
Pezzulo, G., Rigoli, F., & Friston, K. (2015) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2015.09.001 · Cited by 650 · see citing works
Exercising your brain: A review of human brain plasticity and training-induced learning.
Green, C. S., & Bavelier, D. (2008) · Psychology and Aging · doi:10.1037/a0014345 · Cited by 660 · see citing works
Jones, M., & Love, B. C. (2011) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10003134 · Cited by 533 · see citing works
Hayden, B. Y., Heilbronner, S. R., Pearson, J. M., & Platt, M. L. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.4652-10.2011 · Cited by 379 · see citing works
Toward a neurobiology of delusions
Corlett, P. R., Taylor, J. R., Wang, X.-J., Fletcher, P. C., & Krystal, J. H. (2010) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2010.06.007 · Cited by 394 · see citing works
Barto, A., Mirolli, M., & Baldassarre, G. (2013) · Frontiers in Psychology · doi:10.3389/fpsyg.2013.00907 · Cited by 397 · see citing works
From drugs to deprivation: A Bayesian framework for understanding models of psychosis
Corlett, P. R., Frith, C. D., & Fletcher, P. C. (2009) · Psychopharmacology · doi:10.1007/s00213-009-1561-0 · Cited by 367 · see citing works
Dunsmoor, J. E., Niv, Y., Daw, N., & Phelps, E. A. (2015) · Neuron · doi:10.1016/j.neuron.2015.09.028 · Cited by 329 · see citing works
Self-citationEmotional Valence and the Free-Energy Principle
Joffily, M., & Coricelli, G. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003094 · Cited by 322 · see citing works
A computational framework for the study of confidence in humans and animals
Kepecs, A., & Mainen, Z. F. (2012) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2012.0037 · Cited by 305 · see citing works
Adaptive learning under expected and unexpected uncertainty
Soltani, A., & Izquierdo, A. (2019) · Nature reviews. Neuroscience · doi:10.1038/s41583-019-0180-y · Cited by 286 · see citing works
Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings
Payzan-LeNestour, E., & Bossaerts, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001048 · Cited by 267 · see citing works
Gershman, S. J., & Uchida, N. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0220-7 · Cited by 236 · see citing works
Activity patterns of serotonin neurons underlying cognitive flexibility
Matias, S., Lottem, E., Dugué, G. P., & Mainen, Z. F. (2017) · eLife · doi:10.7554/elife.20552 · Cited by 239 · see citing works
Fouragnan, E., Retzler, C., & Philiastides, M. G. (2018) · Human Brain Mapping · doi:10.1002/hbm.24047 · Cited by 176 · see citing works
Dunsmoor, J. E., Campese, V. D., Ceceli, A. O., LeDoux, J. E., & Phelps, E. A. (2015) · Biological Psychiatry · doi:10.1016/j.biopsych.2014.12.008 · Cited by 165 · see citing works
Hippocampal remapping as hidden state inference
Sanders, H., Wilson, M. A., & Gershman, S. J. (2020) · eLife · doi:10.7554/elife.51140 · Cited by 163 · see citing works
Gradual extinction prevents the return of fear: implications for the discovery of state
Gershman, S. J., Jones, C. E., Norman, K. A., Monfils, M.-H., & Niv, Y. (2013) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2013.00164 · Cited by 160 · see citing works
The computational nature of memory modification
Gershman, S. J., Monfils, M.-H., Norman, K. A., & Niv, Y. (2017) · eLife · doi:10.7554/elife.23763 · Cited by 172 · see citing works
Does prediction error drive one-shot declarative learning?
Greve, A., Cooper, E., Kaula, A., Anderson, M. C., & Henson, R. (2017) · Journal of Memory and Language · doi:10.1016/j.jml.2016.11.001 · Cited by 164 · see citing works
Exploring a latent cause theory of classical conditioning
Gershman, S. J., & Niv, Y. (2012) · Learning & Behavior · doi:10.3758/s13420-012-0080-8 · Cited by 149 · see citing works
Brain networks for confidence weighting and hierarchical inference during probabilistic learning
Meyniel, F., & Dehaene, S. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1615773114 · Cited by 138 · see citing works
Discovering latent causes in reinforcement learning
Gershman, S. J., Norman, K. A., & Niv, Y. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.07.007 · Cited by 151 · see citing works
The medial prefrontal cortex shapes dopamine reward prediction errors under state uncertainty
Starkweather, C. K., Gershman, S. J., & Uchida, N. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.036 · Cited by 136 · see citing works
Generalization of value in reinforcement learning by humans
Wimmer, G. E., Daw, N. D., & Shohamy, D. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08017.x · Cited by 111 · see citing works
Self-citationGershman, S. J., Pesaran, B., & Daw, N. D. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2469-09.2009 · Cited by 118 · see citing works
Self-citationHaarsma, J., Fletcher, P. C., Griffin, J. D., Taverne, H. J., Ziauddeen, H., Spencer, T. J., Miller, C., Katthagen, T., Goodyer, I., Diederen, K. M. J., et al. (2021) · Molecular Psychiatry · doi:10.1038/s41380-020-0803-8 · Cited by 108 · see citing works
Belief state representation in the dopamine system
Babayan, B. M., Uchida, N., & Gershman, S. J. (2018) · Nature Communications · doi:10.1038/s41467-018-04397-0 · Cited by 101 · see citing works
Partial reinforcement, extinction, and placebo analgesia
Au Yeung, S. T., Colagiuri, B., Lovibond, P. F., & Colloca, L. (2014) · Pain · doi:10.1016/j.pain.2014.02.022 · Cited by 92 · see citing works
Evidence for model-based computations in the human amygdala during Pavlovian conditioning
Prévost, C., McNamee, D., Jessup, R. K., Bossaerts, P., & O'Doherty, J. P. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002918 · Cited by 79 · see citing works
Frank, B., & McKay, D. (2019) · Cognitive and Behavioral Practice · doi:10.1016/j.cbpra.2018.04.003 · Cited by 64 · see citing works
Radulescu, A., Shin, Y. S., & Niv, Y. (2021) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-092920-120559 · Cited by 63 · see citing works
Ramachandran, S., Meyer, T., & Olson, C. R. (2016) · Journal of Neurophysiology · doi:10.1152/jn.00091.2015 · Cited by 38 · see citing works
Dopamine, inference, and uncertainty
Gershman, S. J. (2017) · Neural Computation · doi:10.1162/neco_a_01023 · Cited by 40 · see citing works
Aubert‐Teillaud, B., Bran, A., & Vaidis, D. C. (2023) · European Journal of Social Psychology · doi:10.1002/ejsp.2981 · Cited by 19 · see citing works
Predictive Representations: Building Blocks of Intelligence
Carvalho, W., Tomov, M. S., de Cothi, W., Barry, C., & Gershman, S. J. (2024) · Neural Computation · doi:10.1162/neco_a_01705 · Cited by 15 · see citing works
A probabilistic successor representation for context-dependent learning
Geerts, J. P., Gershman, S. J., Burgess, N., & Stachenfeld, K. L. (2024) · Psychological Review · doi:10.1037/rev0000414 · Cited by 14 · see citing works
- Authors
- Courville, A. C., Daw, N. D., & Touretzky, D. S.
- Affiliations
- Université de Montréal (CA)
- Gatsby Computational Neuroscience Unit (GB)
- Center for the Neural Basis of Cognition (US)
- Corresponding author
- Courville, Aaron C.
- Venue
- Trends in Cognitive Sciences
- Publisher
- Elsevier BV
- Published
- 2006-07
- Volume
- 10
- Issue
- 7
- Pages
- 294-300
- ISSN
- 1364-6613 (ISSN-L)
- OpenAlex
- W2151196108
- Cited by
- 457
- References
- 37
- Citations by year
- 2026: 13 · 2025: 28 · 2024: 20 · 2023: 24 · 2022: 28 · 2021: 19 · 2020: 15 · 2019: 45 · 2018: 26 · 2017: 31 · 2016: 27 · 2015: 26 · 2014: 30 · 2013: 26 · 2012: 27
- FWCI
- 0.99
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Trends in Cognitive Sciences (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory and Neural Mechanisms
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Keywords
- Bayesian learning, classical conditioning, surprise, uncertainty, statistical learning, dynamic environments, animal learning, human learning
- MeSH terms
- Animals, Association Learning, Bayes Theorem, Behavior, Animal, Conditioning, Classical, Extinction, Psychological, Humans, Inhibition, Psychological, Problem Solving, Reinforcement Schedule, Social Change, Social Environment, Stochastic Processes, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
