The Homeostatic Logic of Reward
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Energy homeostasis depends on behavior to predictively regulate metabolic states within narrow bounds. Here we review three theories of homeostatic control and ask how they provide insight into the circuitry underlying energy homeostasis. We offer two contributions. First, we detail how control theory and reinforcement learning are applied to homeostatic control. We show how these schemes rest on implausible assumptions; either via circular definitions, unprincipled drive functions, or by ignoring environmental volatility. We argue active inference can elude these shortcomings while retaining important features of each model. Second, we review the neural basis of energetic control. We focus on a subset of arcuate subpopulations that project directly to, and are thus in a privileged position to opponently modulate, dopaminergic cells as a function of energetic predictions over a spectrum of time horizons. We discuss how this can be interpreted under these theories, and how this can resolve paradoxes that have arisen. We propose this circuit constitutes a homeostatic-reward interface that underwrites the conjoint optimisation of physiological and behavioural homeostasis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Forward Models for Physiological Motor Control
Miall, R. C., & Wolpert, D. M. (1996) · Neural Networks · doi:10.1016/s0893-6080(96)00035-4 · Cited by 2,260
Berridge, K. C., & Robinson, T. E. (2003) · Trends in Neurosciences · doi:10.1016/s0166-2236(03)00233-9 · Cited by 1,869
A free energy principle for the brain
Friston, K., Kilner, J., & Harrison, L. (2006) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2006.10.001 · Cited by 1,369 · cited 4×
“hidden states in order to maintain physiological integrity (Friston 2012; Friston et al. 2006).”
BackgroundInfluentialSelf-citationAdaptive Coding of Reward Value by Dopamine Neurons
Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005) · Science · doi:10.1126/science.1105370 · Cited by 1,276 · cited 2×
“sensory events in monkeys, where reward magnitude is determined by the volume of 539 nutrients consumed (Tobler et al. 2005; Stauffer et al. 2014; Ballard & Knutson 2009).”
BackgroundAGRP neurons are sufficient to orchestrate feeding behavior rapidly and without training
Aponte, Y., Atasoy, D., & Sternson, S. M. (2011) · Nature Neuroscience · doi:10.1038/nn.2739 · Cited by 1,157
Allostasis: A model of predictive regulation
Sterling, P. (2012) · Physiology & Behavior · doi:10.1016/j.physbeh.2011.06.004 · Cited by 1,073
Deconstruction of a neural circuit for hunger
Atasoy, D., Betley, J. N., Su, H. H., & Sternson, S. M. (2012) · Nature · doi:10.1038/nature11270 · Cited by 969
Neurons for hunger and thirst transmit a negative-valence teaching signal
Betley, J. N., Xu, S., Cao, Z. F. H., Gong, R., Magnus, C. J., Yu, Y., & Sternson, S. M. (2015) · Nature · doi:10.1038/nature14416 · Cited by 704
Small, D. M., Jones-Gotman, M., & Dagher, A. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00253-2 · Cited by 632
An excitatory paraventricular nucleus to AgRP neuron circuit that drives hunger
Krashes, M. J., Shah, B. P., Madara, J. C., Olson, D. P., Strochlic, D. E., Garfield, A. S., Vong, L., Pei, H., Watabe-Uchida, M., Uchida, N., et al. (2014) · Nature · doi:10.1038/nature12956 · Cited by 603 · cited 1×
“Natural deprivation, ghrelin injection, pharmacological or optogenetic activation of AgRP neurons evoke voracious feeding and inhibit POMC neurons, as might be expected with a large deviation from a set point (Betley et al. 2015; Chen et al. 2016; Krashes et al. 2014; Mandelblat-Cerf et al. 2015).”
BackgroundAllostatic Self-efficacy: A Metacognitive Theory of Dyshomeostasis-Induced Fatigue and Depression
Stephan, K. E., Manjaly, Z. M., Mathys, C. D., Weber, L. A. E., Paliwal, S., Gard, T., Tittgemeyer, M., Fleming, S. M., Haker, H., Seth, A. K., et al. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00550 · Cited by 437
A free energy principle for biological systems
Karl, F. (2012) · Entropy · doi:10.3390/e14112100 · Cited by 377 · cited 2×
“One exigent problem (see Friston & Ao 2012 for several others) with reinforcement learning in general is that the definition of reward is behaviour-centric: Agents strive to maximise reward, but reward is defined from observed behaviour.”
BackgroundTakahashi, K. A., & Cone, R. D. (2005) · Endocrinology · doi:10.1210/en.2004-1397 · Cited by 308
Mandelblat-Cerf, Y., Ramesh, R. N., Burgess, C. R., Patella, P., Yang, Z., Lowell, B. B., & Andermann, M. L. (2015) · eLife · doi:10.7554/elife.07122 · Cited by 290
The anatomy of choice: dopamine and decision-making.
Friston, K., Schwartenbeck, P., FitzGerald, T., Moutoussis, M., Behrens, T., & Dolan, R. J. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0481 · Cited by 289
Self-citationA tutorial on the free-energy framework for modelling perception and learning
Bogacz, R. (2017) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2015.11.003 · Cited by 283 · cited 2×
“The interested reader should see Bogacz (2015) for an tutorial based introduction to the technical aspects variational inference in context of perception.”
BackgroundThe dopaminergic midbrain encodes the expected certainty about desired outcomes
Schwartenbeck, P., FitzGerald, T. H. B., Mathys, C., Dolan, R., & Friston, K. (2015) · Cerebral Cortex · doi:10.1093/cercor/bhu159 · Cited by 219
Self-citationHomeostatic reinforcement learning for integrating reward collection and physiological stability
Keramati, M., & Gutkin, B. (2014) · eLife · doi:10.7554/elife.04811 · Cited by 218
Explaining delusions of control: The comparator model 20years on
Frith, C. (2012) · Consciousness and Cognition · doi:10.1016/j.concog.2011.06.010 · Cited by 217
Dopamine Reward Prediction Error Responses Reflect Marginal Utility
Stauffer, W. R., Lak, A., & Schultz, W. (2014) · Current Biology · doi:10.1016/j.cub.2014.08.064 · Cited by 204
Hunger neurons drive feeding through a sustained, positive reinforcement signal.
Chen, Y., Lin, Y.-C., Zimmerman, C. A., Essner, R. A., & Knight, Z. A. (2016) · eLife · doi:10.7554/elife.18640 · Cited by 196
AgRP neurons regulate development of dopamine neuronal plasticity and nonfood-associated behaviors
Dietrich, M. O., Bober, J., Ferreira, J. G., Tellez, L. A., Mineur, Y. S., Souza, D. O., Gao, X.-B., Picciotto, M. R., Araújo, I., Liu, Z.-W., et al. (2012) · Nature Neuroscience · doi:10.1038/nn.3147 · Cited by 158
Dopamine, uncertainty and TD learning
Niv, Y., Duff, M. O., & Dayan, P. (2005) · Behavioral and Brain Functions · doi:10.1186/1744-9081-1-6 · Cited by 153
On hyperpriors and hypopriors: comment on Pellicano and Burr
Friston, K. J., Lawson, R., & Frith, C. D. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2012.11.003 · Cited by 151
Self-citation
Cited by
Computational Models of Interoception and Body Regulation
Petzschner, F. H., Garfinkel, S. N., Paulus, M. P., Koch, C., & Khalsa, S. S. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2020.09.012 · Cited by 227
“[54] (Figure 3B) (see [39,49,55] for a detailed discussion).”
BackgroundCorcoran, A. W., Pezzulo, G., & Hohwy, J. (2020) · Biology & Philosophy · doi:10.1007/s10539-020-09746-2 · Cited by 161
Interoception as modeling, allostasis as control
Sennesh, E., Theriault, J., Brooks, D., van de Meent, J.-W., Barrett, L. F., & Quigley, K. S. (2022) · Biological Psychology · doi:10.1016/j.biopsycho.2021.108242 · Cited by 107
Motivation in the Service of Allostasis: The Role of Anterior Mid-Cingulate Cortex
Touroutoglou, A., Andreano, J. M., Adebayo, M., Lyons, S., & Barrett, L. F. (2019) · Advances in Motivation Science · doi:10.1016/bs.adms.2018.09.002 · Cited by 25
Metadata
- Authors
- Morville, T., Friston, K., Burdakov, D., Siebner, H. R., & Hulme, O. J.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2018-01-05
- OpenAlex
- W2786892827
- Cited by
- 18
- References
- 24
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.