The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.
Homeostatic reinforcement learning for integrating reward collection and physiological stability
Open accessPublished version of a preprint
Abstract
Efficient regulation of internal homeostasis and defending it against perturbations requires adaptive behavioral strategies. However, the computational principles mediating the interaction between homeostatic and associative learning processes remain undefined. Here we use a definition of primary rewards, as outcomes fulfilling physiological needs, to build a normative theory showing how learning motivated behaviors may be modulated by internal states. Within this framework, we mathematically prove that seeking rewards is equivalent to the fundamental objective of physiological stability, defining the notion of physiological rationality of behavior. We further suggest a formal basis for temporal discounting of rewards by showing that discounting motivates animals to follow the shortest path in the space of physiological variables toward the desired setpoint. We also explain how animals learn to act predictively to preclude prospective homeostatic challenges, and several other behavioral patterns. Finally, we suggest a computational role for interaction between hypothalamus and the brain reward system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current models of decision-making do not incorporate all of these aspects, and first attempts of accounting for homeostasis and allostasis in formal models of decision-making have only surfaced relatively recently (e.g., Keramati and Gutkin, 2014; Penny and Stephan, 2014; Pezzulo et al., 2015).”
From Allostatic Self-efficacy: A Metacognitive Theory of Dyshomeostasis-Induced Fatigue and Depression (2016) · cites this paper as Keramati and Gutkin, 2014
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Current models of decision-making do not incorporate all of these aspects, and first attempts of accounting for homeostasis and allostasis in formal models of decision-making have only surfaced relatively recently (e.g., Keramati and Gutkin, 2014; Penny and Stephan, 2014; Pezzulo et al., 2015).”
From Allostatic Self-efficacy: A Metacognitive Theory of Dyshomeostasis-Induced Fatigue and Depression (2016) · cites this paper as Keramati and Gutkin, 2014
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
Prospect theory: An analysis of decision under risk
Kahneman, D. J., & Tversky, A. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511609220.014 · Cited by 32,720
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 3×
“A self-organizing system (i.e. an organism) can be defined as a system that opposes the second law of thermodynamics (Friston, 2010).”
BackgroundMethodInfluentialSakurai, T., Amemiya, A., Ishii, M., Matsuzaki, I., Chemelli, R. M., Tanaka, H., Williams, S. C., Richardson, J. A., Kozlowski, G. P., Wilson, S., et al. (1998) · Cell · doi:10.1016/s0092-8674(00)80949-6 · Cited by 5,501
Skinner, B. F. (1938) · Cited by 3,571
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
A framework for studying the neurobiology of value-based decision making
Rangel, A., Camerer, C., & Montague, P. R. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2357 · Cited by 2,184
Molecular basis of long-term plasticity underlying addiction
Nestler, E. J. (2001) · Nature reviews. Neuroscience · doi:10.1038/35053570 · Cited by 1,893
Organization for physiological homeostasis
Cannon, W. B. (1929) · Physiological Reviews · doi:10.1152/physrev.1929.9.3.399 · Cited by 1,844
Interoceptive inference, emotion, and the embodied self
Seth, A. K. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.09.007 · Cited by 1,816
Principles of Behavior. An Introduction to Behavior Theory
N., E., & Hull, C. L. (1943) · The Journal of Philosophy · doi:10.2307/2019960 · Cited by 1,736
Mowrer, O. H. (1960) · doi:10.1037/10802-000 · Cited by 1,670
Gonzalez, R. C., & Mowrer, O. H. (1962) · The American Journal of Psychology · doi:10.2307/1419634 · Cited by 1,584
Cabanac, M. (1971) · Science · doi:10.1126/science.173.4002.1103 · Cited by 1,521
Behavior Theory and Conditioning
Bitterman, M. E., & Spence, K. W. (1957) · The American Journal of Psychology · doi:10.2307/1419251 · Cited by 1,460
Motivation concepts in behavioral neuroscience
Berridge, K. C. (2004) · Physiology & Behavior · doi:10.1016/j.physbeh.2004.02.004 · Cited by 1,302
Progressive Ratio as a Measure of Reward Strength
Hodos, W. (1961) · Science · doi:10.1126/science.134.3483.943 · Cited by 1,147
Allostasis: A model of predictive regulation
Sterling, P. (2012) · Physiology & Behavior · doi:10.1016/j.physbeh.2011.06.004 · Cited by 1,073
Every good regulator of a system must be a model of that system †
CONANT, R. C., & ROSS ASHBY, W. (1970) · International Journal of Systems Science · doi:10.1080/00207727008920220 · Cited by 1,063
Behavior theory and conditioning.
Spence, K. W. (1956) · doi:10.1037/10029-000 · Cited by 872
Every Good Regulator of a System Must Be a Model of That System
Conant, R. C., & Ashby, W. R. (1991) · Facets of Systems Science · doi:10.1007/978-1-4899-0718-9_37 · Cited by 712
From prediction error to incentive salience: mesolimbic computation of reward motivation.
Berridge, K. C. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.07990.x · Cited by 700
Mas-Colell, A. 1., Whinston, M. D. 1., & Green, J. R. 1. (1995) · Cited by 691
Billings, H. J. (2019) · The Research Repository @ WVU (West Virginia University) · Cited by 665
Excitation of Ventral Tegmental Area Dopaminergic and Nondopaminergic Neurons by Orexins/Hypocretins
Korotkova, T. M., Sergeeva, O. A., Eriksson, K. S., Haas, H. L., & Brown, R. E. (2003) · Journal of Neuroscience · doi:10.1523/jneurosci.23-01-00007.2003 · Cited by 583
An Evolutionary Perspective on Food and Human Taste
Breslin, P. A. S. (2013) · Current Biology · doi:10.1016/j.cub.2013.04.010 · Cited by 551
CHOICE AND DELAY OF REINFORCEMENT1
Chung, S., & Herrnstein, R. J. (1967) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.1967.10-67 · Cited by 540
Narita, M., Nagumo, Y., Hashimoto, S., Narita, M., Khotib, J., Miyatake, M., Sakurai, T., Yanagisawa, M., Nakamachi, T., Shioda, S., et al. (2006) · Journal of Neuroscience · doi:10.1523/jneurosci.2761-05.2006 · Cited by 524
Is dopamine a physiologically relevant mediator of feeding behavior?
Palmiter, R. D. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.06.004 · Cited by 469
Intrinsically Motivated Reinforcement Learning: An Evolutionary Perspective
Singh, S. P., Lewis, R. L., Barto, A. G., & Sorg, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2051031 · Cited by 444
From neuroanatomy to behavior: central integration of peripheral signals regulating feeding behavior
Williams, K. W., & Elmquist, J. K. (2012) · Nature Neuroscience · doi:10.1038/nn.3217 · Cited by 435
Food Reward in the Absence of Taste Receptor Signaling
de Araujo, I. E., Oliveira-Maia, A. J., Sotnikova, T. D., Gainetdinov, R. R., Caron, M. G., Nicolelis, M. A. L., & Simon, S. A. (2008) · Neuron · doi:10.1016/j.neuron.2008.01.032 · Cited by 431
The Role of Learning in the Operation of Motivational Systems
Dickinson, A., & Balleine, B. W. (2002) · Wiley eBooks · doi:10.1002/0471214426.pas0312 · Cited by 387
The eating paradox: How we tolerate food.
Woods, S. C. (1991) · Psychological Review · doi:10.1037/0033-295x.98.4.488 · Cited by 343
Burdakov, D., Gerasimenko, O. V., & Verkhratsky, A. (2005) · Journal of Neuroscience · doi:10.1523/jneurosci.4925-04.2005 · Cited by 337
A Neural Computational Model of Incentive Salience
Zhang, J., Berridge, K. C., Tindell, A. J., Smith, K. S., & Aldridge, J. W. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000437 · Cited by 299
Unraveling the brain regulation of appetite: lessons from genetics
Yeo, G. S. H., & Heisler, L. K. (2012) · Nature Neuroscience · doi:10.1038/nn.3211 · Cited by 295
Speakman, J. R., Levitsky, D. A., Allison, D. B., Bray, M. S., de Castro, J. M., Clegg, D. J., Clapham, J. C., Dulloo, A. G., Gruer, L., Haw, S., et al. (2011) · Disease Models & Mechanisms · doi:10.1242/dmm.008698 · Cited by 290
Regulation of dietary choice by the decision-making circuitry
Rangel, A. (2013) · Nature Neuroscience · doi:10.1038/nn.3561 · Cited by 263
Zhang, M., & Kelley, A. E. (2000) · Neuroscience · doi:10.1016/s0306-4522(00)00198-6 · Cited by 260
Rodgers, R. J., Halford, J. C. G., de Souza, R. L. N., de Souza, A. L. C., Piper, D. C., Arch, J. R. S., Upton, N., Porter, R. A., Johns, A., & Blundell, J. E. (2001) · European Journal of Neuroscience · doi:10.1046/j.0953-816x.2001.01518.x · Cited by 230
High-intensity sweeteners and energy balance.
Swithers, S. E., Martin, A. A., & Davidson, T. L. (2010) · Physiology & Behavior · doi:10.1016/j.physbeh.2009.12.021 · Cited by 206 · cited 1×
“Such association between sensory and post-ingestive properties could have been developed through prior learning (Swithers et al., 2009; Swithers et al., 2010; Beeler et al., 2012) or evolutionary mechanisms (Breslin, 2013).”
Differential effects of morphine on food and water intake in food deprived and freely-feeding rats
Sanger, D. J., & McCarthy, P. S. (1980) · Psychopharmacology · doi:10.1007/bf00433813 · Cited by 205
Cornelius, J. R., Tippmann‐Peikert, M., Slocumb, N. L., Frerichs, C. F., & Silber, M. H. (2010) · SLEEP · doi:10.1093/sleep/33.1.81 · Cited by 199
Conditioning and extinction of tolerance to the hypothermic effect of ethanol in rats.
Mansfield, J. G., & Cunningham, C. L. (1980) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0077824 · Cited by 196
Opiate agonists microinjected into the nucleus accumbens enhance sucrose drinking in rats
Zhang, M., & Kelley, A. E. (1997) · Psychopharmacology · doi:10.1007/s002130050355 · Cited by 191
Solinas, M., & Goldberg, S. R. (2005) · Neuropsychopharmacology · doi:10.1038/sj.npp.1300720 · Cited by 183
Set points, settling points, and the control of body weight☆
Wirtshafter, D., & DAVIS, J. (1977) · Physiology & Behavior · doi:10.1016/0031-9384(77)90162-7 · Cited by 182
Nutrient Selection in the Absence of Taste Receptor Signaling
Ren, X., Ferreira, J. G. P., Zhou, L.-W., Shammah‐Lagnado, S. J., Yeckel, C. W., & de Araújo, I. E. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.5749-09.2010 · Cited by 162
Understanding the mechanisms of food intake and obesity
Rolls, E. T. (2007) · Obesity Reviews · doi:10.1111/j.1467-789x.2007.00321.x · Cited by 151
(2002) · Advances in consciousness research · doi:10.1075/aicr.44 · Cited by 148
Biobehavioral Influences on Energy Intake and Adult Weight Gain
McCrory, M. A., Suen, V. M. M., & Roberts, S. B. (2002) · Journal of Nutrition · doi:10.1093/jn/132.12.3830s · Cited by 139
Morphine enhances hedonic taste palatability in rats
Doyle, T. G., Berridge, K. C., & Gosnell, B. A. (1993) · Pharmacology Biochemistry and Behavior · doi:10.1016/0091-3057(93)90572-b · Cited by 131
Reward effects of food via stomach fistula compared with those of food via mouth.
Miller, N. E., & Kessen, M. L. (1952) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0060113 · Cited by 111
Swithers, S. E., Baker, C. R., & Davidson, T. L. (2009) · Behavioral Neuroscience · doi:10.1037/a0016139 · Cited by 105
Homeostasis: Beyond Curt Richter
Woods, S. C., & Ramsay, D. S. (2007) · Appetite · doi:10.1016/j.appet.2006.09.015 · Cited by 102
Potency of naloxone's anorectic effect in rats is dependent on diet preference
Glass, M. J., Grace, M. K., Cleary, J. P., Billington, C. J., & Levine, A. (1996) · American Journal of Physiology-Regulatory, Integrative and Comparative Physiology · doi:10.1152/ajpregu.1996.271.1.r217 · Cited by 97
Lower pleasantness of palatable foods in nalmefene-treated human volunteers
Yeomans, M. R., & WRIGHT, P. D. (1991) · Appetite · doi:10.1016/0195-6663(91)90062-w · Cited by 96
Naloxone effects on sucrose-motivated behavior
Cleary, J. P., Weldon, D. T., O‘Hare, E., Billington, C. J., & Levine, A. S. (1996) · Psychopharmacology · doi:10.1007/bf02246345 · Cited by 90
Skjoldager, P. D., Pierre, P. J., & Mittleman, G. (1993) · Learning and Motivation · doi:10.1006/lmot.1993.1019 · Cited by 88
Will, M. J., Pratt, W. E., & Kelley, A. E. (2006) · Physiology & Behavior · doi:10.1016/j.physbeh.2006.06.008 · Cited by 87
Homeostasis: Origins of the Concept
Langley, L. L., & Johnson, S. H. (1974) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.3426765 · Cited by 79
Dopaminergic and non-dopaminergic value systems in conditioning and outcome-specific revaluation
Dranias, M. R., Grossberg, S., & Bullock, D. H. (2008) · Brain Research · doi:10.1016/j.brainres.2008.07.013 · Cited by 75
Devine, D. P., Leone, P., & Wise, R. A. (1993) · European Journal of Pharmacology · doi:10.1016/0014-2999(93)90167-g · Cited by 75
Taste uncoupled from nutrition fails to sustain the reinforcing properties of food
Beeler, J. A., McCutcheon, J. E., Cao, Z. F. H., Murakami, M., Alexander, E. R., Roitman, M. F., & Zhuang, X. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08167.x · Cited by 73
Marieb, E. N. 1., & Smith, L. A. (2022) · Cited by 67
Barbano, M. F., Saux, M. L., & Cador, M. (2008) · Psychopharmacology · doi:10.1007/s00213-008-1390-6 · Cited by 56
Is there evidence for a set point that regulates human body weight?
Müller, M. J., Bosy-Westphal, A., & Heymsfield, S. B. (2010) · F1000 Medicine Reports · doi:10.3410/m2-59 · Cited by 49
Orosensory Self‐Stimulation by Sucrose Involves Brain Dopaminergic Mechanismsa
Schneider, L. H. (1989) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1989.tb53252.x · Cited by 48
Mansfield, J. G., Benedict, R. S., & Woods, S. C. (1983) · Psychopharmacology · doi:10.1007/bf00427791 · Cited by 40
Ventral tegmental injections of morphine but not U-50,488H enhance feeding in food-deprived rats
Noel, M. B., & Wise, R. A. (1993) · Brain Research · doi:10.1016/0006-8993(93)91139-j · Cited by 37
Tolerance to hypothermia induced by ethanol depends on specific drug effects
Hjeresen, D. L., Reed, D. R., & Woods, S. C. (1986) · Psychopharmacology · doi:10.1007/bf00175187 · Cited by 27
9. Motivational underpinnings of utility in decision making
Busemeyer, J. R., Townsend, J. T., & Stout, J. C. (2002) · Advances in consciousness research · doi:10.1075/aicr.44.09bus · Cited by 27
Separation of satiating and rewarding consequences of drinking
McFarland, D. J. (1969) · Physiology & Behavior · doi:10.1016/0031-9384(69)90054-7 · Cited by 26
Human Anatomy & Physiology 9th Edition
Elaine, N. M., & Katja, H. (2013) · Cited by 17
Artificial motives: A review of motivation in artificial creatures
Savage, T. (2000) · Connection Science · doi:10.1080/095400900750060131 · Cited by 16
Woods, S. C., & Seeley, R. J. (2002) · Wiley eBooks · doi:10.1002/0471214426.pas0315 · Cited by 12
Flagged first, then by citations
Research works printed but not linked: 3
- Bernard, 1957, Lectures on the physiological properties and the pathological alternations of the liquids of the organism: Third lecture, 89
- Marr, 1982, Vision
- Sibly, 1974, State Space Approach to Motivation, Motivational Control System Analysis
Cited by
Sorted by citations
Allostatic Self-efficacy: A Metacognitive Theory of Dyshomeostasis-Induced Fatigue and Depression
Stephan, K. E., Manjaly, Z. M., Mathys, C. D., Weber, L. A. E., Paliwal, S., Gard, T., Tittgemeyer, M., Fleming, S. M., Haker, H., Seth, A. K., et al. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00550 · Cited by 437
“Current models of decision-making do not incorporate all of these aspects, and first attempts of accounting for homeostasis and allostasis in formal models of decisionmaking have only surfaced relatively recently (e.g., Keramati and Gutkin, 2014; Penny and Stephan, 2014; Pezzulo et al., 2015).”
BackgroundAtzil, S., Gao, W., Fradkin, I., & Barrett, L. F. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0384-6 · Cited by 415
Emotion words, emotion concepts, and emotional development in children: A constructionist hypothesis
Hoemann, K., Xu, F., & Barrett, L. F. (2019) · Developmental Psychology · doi:10.1037/dev0000686 · Cited by 354
Computational Models of Interoception and Body Regulation
Petzschner, F. H., Garfinkel, S. N., Paulus, M. P., Koch, C., & Khalsa, S. S. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2020.09.012 · Cited by 227
Reinforcement learning across development: What insights can we draw from a decade of research?
Nussenbaum, K., & Hartley, C. A. (2019) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2019.100733 · Cited by 225
Metadata
- Authors
- Keramati, M., & Gutkin, B.
- Affiliations
- Inserm (FR)
- National Research University Higher School of Economics (RU)
- Corresponding author
- Keramati, Mehdi
- Gutkin, Boris
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2014-12-02
- Volume
- 3
- ISSN
- 2050-084X (ISSN-L)
- PMC
- PMC4270100
- OpenAlex
- W2142110758
- Cited by
- 218
- References
- 80
- Citations by year
- 2026: 23 · 2025: 35 · 2024: 29 · 2023: 15 · 2022: 37 · 2021: 23 · 2020: 16 · 2019: 16 · 2018: 10 · 2017: 8 · 2016: 5 · 2015: 1
- FWCI
- 2.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- eLife (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- City Research Online (City University London) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Gatsby Charitable Foundation
- National Research University Higher School of Economics (Basic Research Program)
- Institut national de la santé et de la recherche médicale (INSERM U960, France)
- Center for Research and Interdisciplinary (Frontiers du Vivant)
- Agence Nationale de la Recherche (ANR-10-LABX-0087 IEC, France, ANR-10-IDEX-0001-02 PSL, France)
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- physiological stability, primary reward, normative theory, delay discounting, predictive learning, hypothalamus, brain reward system, adaptive behavior
- MeSH terms
- Animals, Behavior, Computer Simulation, Conditioning, Psychological, Extinction, Psychological, Homeostasis, Learning, Models, Neurological, Reinforcement, Psychology, Reward, Satiety Response
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
