Cited by 218Neurotransmitter Receptor Influence on BehaviorNeural and Behavioral Psychology StudiesReinforcement Learning in Robotics

The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index.

Homeostatic reinforcement learning for integrating reward collection and physiological stability

Open access

Keramati, Mehdi, Gutkin, Boris·2014·eLife

DOI 10.7554/elife.04811PMC PMC4270100

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Efficient regulation of internal homeostasis and defending it against perturbations requires adaptive behavioral strategies. However, the computational principles mediating the interaction between homeostatic and associative learning processes remain undefined. Here we use a definition of primary rewards, as outcomes fulfilling physiological needs, to build a normative theory showing how learning motivated behaviors may be modulated by internal states. Within this framework, we mathematically prove that seeking rewards is equivalent to the fundamental objective of physiological stability, defining the notion of physiological rationality of behavior. We further suggest a formal basis for temporal discounting of rewards by showing that discounting motivates animals to follow the shortest path in the space of physiological variables toward the desired setpoint. We also explain how animals learn to act predictively to preclude prospective homeostatic challenges, and several other behavioral patterns. Finally, we suggest a computational role for interaction between hypothalamus and the brain reward system.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.7554/elife.04811

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Figure 1. Schematics of the model in an exemplary two-dimensional homeostatic space. Upon performing an action, the animal receives an outcome Kt from the environment. The rewarding value of this outcome depends on its ability to make the internal state, Ht, closer to the homeostatic setpoint, H^(*), and thus reduce the drive level (the vertical axis). This experienced reward, denoted by r(Ht,Kt), is then learned by an RL algorithm. Here a model-free RL algorithm is shown in which a reward prediction error signal is computed by comparing the realized reward and the expected rewarding value of the performed response. This signal is then used to update the subjective value attributed to the corresponding response. Subjective values of alternative choices bias the action selection process. DOI: http://dx.doi.org/10.7554/eLife.04811.003CC BY · PubMed Central

Figure 2. Experimental results (adapted from Mansfield & Cunningham, 1980) on the acquisition and extinction of conditioned tolerance response to ethanol. (A) In each block (day) of the experiment, the animal received ethanol injection after the presentation of the stimulus. (B) The change in the body temperature was measured 30, 60, 90, and 120 min after ethanol administration. Initially, the hypothermic effect of ethanol decreased the body temperature of animals. After several training days, however, animals learned to activate a tolerance response upon observing the stimulus, resulting in smaller deviations from the temperature setpoint. If the stimulus was not followed by ethanol injection, as in the first day of extinction (E1), the activation of the conditioned tolerance response resulted in an increase in body temperature. The tolerance response was weakened after several (four) extinction sessions, resulting in increased deviation from the setpoint in the first day of re-acquisition (R1), where presentation of the cue was again followed by ethanol injection. DOI: http://dx.doi.org/10.7554/eLife.04811.004CC BY · PubMed Central

Figure 3. Simulation result on anticipatory responding. (A) In every trial, the simulated agent can choose between initiating a tolerance response and doing nothing, upon observing the stimulus. Regardless of the agent's choice, ethanol is administered after 1 hr, followed by four temperature measurements every 30 min. (B) Dynamics of temperature upon ethanol injection. (C) Learning curve for choosing the ‘tolerance’ response. (D) Dynamics of temperature upon initiating the tolerance response. (E) Temperature profile during several simulated trails. (F) Dynamics of temperature upon initiating the tolerance response, followed by ethanol administration. Plots c and e are averaged over 500 simulated agents. DOI: http://dx.doi.org/10.7554/eLife.04811.005CC BY · PubMed Central

Figure 4. Schematic illustration of the behavioral properties of the drive function. (A) excitatory effect of the dose of outcome on its rewarding value. (B, C) excitatory effect of deprivation level on the rewarding value of outcomes: Increased deprivation increases the rewarding value of reducing drive (B), and increases the punishing value of increasing drive (C). (D) inhibitory effect of irrelevant drives on the rewarding value of outcomes. DOI: http://dx.doi.org/10.7554/eLife.04811.007CC BY · PubMed Central

Figure 5. Risk aversion simulation. In a conditioned place preference paradigm, the agent's presence in the left and the right compartments has equal expected payoffs, but different levels of risk (A) Panel (B) shows the Markov decision process of the same task. In fact, in every trial, the agent chooses whether to stay it the current compartment, or transit to the other one. The average input of energy per trial, regardless of the animal's choice, is set such that it is equal to the animal's normal energy expenditure. Thus, the internal state stays close to its initial level, which is equal to the setpoint here (D). The model learns to prefer the non-risky over the risky compartments (C) in order to avoid severe deviations from the setpoint. DOI: http://dx.doi.org/10.7554/eLife.04811.008CC BY · PubMed Central

Figure 6. Simulations showing that the model avoids extreme deviations. Starting from 30, the agent can either decrease or increase its internal state by one unit in each trial. (A) The number of visits at each internal state after 106 trials. (B) The drive function in the one-dimensional homeostatic space. (setpoint = 0). The mean (C) and standard deviation (D) of the internal state of 105 agents, along 1500 trials. DOI: http://dx.doi.org/10.7554/eLife.04811.010CC BY · PubMed Central

Figure 6—figure supplement 1. The Markov Decision Process used for simulation results presented in Figure 6 and Figure 6—figure supplements 2–7. DOI: http://dx.doi.org/10.7554/eLife.04811.012CC BY · PubMed Central

Figure 6—figure supplement 2. Value function (A) and choice preferences (B) for state–action pairs after simulating one agent for 106 trials (As in Figure 6). The parameters of the model where as follows: α = 0.4, β = 0.05, γ = 0.9, n = 4. DOI: http://dx.doi.org/10.7554/eLife.04811.013CC BY · PubMed Central

Figure 6—figure supplement 3. Simulation results replicating Figure 6, with the difference that the initial internal state was zero. (A) The number of visits at each internal state after 106 trials. (B) The drive function in the one-dimensional homeostatic space (setpoint=0). Value function (C) and choice preferences (D) for state-action pairs after simulating one agent for 106 trials. The mean (E) and standard deviation (F) of the internal state of 105 agents, along 1500 trials. DOI: http://dx.doi.org/10.7554/eLife.04811.014CC BY · PubMed Central

Figure 6—figure supplement 4. Simulation results replicating Figure 6, with the difference that the initial internal state was zero, and the rate of exploration, β, was 0.03. (A) The number of visits at each internal state after 106 trials. (B) The drive function in the one-dimensional homeostatic space (setpoint=0). Value function (C) and choice preferences (D) for state-action pairs after simulating one agent for 106 trials. The mean (E) and standard deviation (F) of the internal state of 105 agents, along 1500 trials. DOI: http://dx.doi.org/10.7554/eLife.04811.015CC BY · PubMed Central

Figure 6—figure supplement 5. Simulation results replicating Figure 6, with the difference that the initial internal state was zero, and also m = n = 1. (A) The number of visits at each internal state after 106 trials. (B) The drive function in the one-dimensional homeostatic space (setpoint=0). Value function (C) and choice preferences (D) for state-action pairs after simulating one agent for 106 trials. The mean (E) and standard deviation (F) of the internal state of 105 agents, along 1500 trials. DOI: http://dx.doi.org/10.7554/eLife.04811.016CC BY · PubMed Central

Figure 6—figure supplement 6. Simulation results replicating Figure 6, with the difference that the initial internal state was zero, and the discount factor, γ, was zero. (A) The number of visits at each internal state after 106 trials. (B) The drive function in the one-dimensional homeostatic space (setpoint=0). Value function (C) and choice preferences (D) for state-action pairs after simulating one agent for 106 trials. The mean (E) and standard deviation (F) of the internal state of 105 agents, along 1500 trials. DOI: http://dx.doi.org/10.7554/eLife.04811.017CC BY · PubMed Central

Figure 6—figure supplement 7. Simulation results replicating Figure 6, with the difference that the initial internal state was zero, and the discount factor, γ, was one (no discounting). (A) The number of visits at each internal state after 106 trials. (B) The drive function in the one-dimensional homeostatic space (setpoint=0). Value function (C) and choice preferences (D) for state-action pairs after simulating one agent for 106 trials. The mean (E) and standard deviation (F) of the internal state of 105 agents, along 1500 trials. DOI: http://dx.doi.org/10.7554/eLife.04811.018CC BY · PubMed Central

Figure 7. Experimental results (adapted from McFarland, 1969) on learning the reinforcing effect of oral vs. intragastric delivery of water. Thirsty animals were initially trained to peck at a green key to receive water orally. In the next phase, pecking at the green key had no consequence, while pecking at a novel yellow key resulted in oral delivery of water in one group (A), and intragastric injection of the same amount of water through a fistula in a second group (B). In the first group, responding was rapidly transferred from the green to the yellow key, and then suppressed. In the fistula group, the yellow key was not reinforced. DOI: http://dx.doi.org/10.7554/eLife.04811.019CC BY · PubMed Central

Figure 8. Simulation results replicating the data from McFarland (1969) on learning the reinforcing effect of oral vs. intragastric delivery of water. As in the experiment, two groups of simulated agents were pre-trained to respond on the green key to receive oral delivery of water. During the test phase, the green key had no consequence, whereas a novel yellow key resulted in oral delivery in one group (A) and intragastric injection in the second group (B). All agents started this phase in a thirsty state (initial internal state = 0; setpoint = 0). In the oral group, responding transferred rapidly from the green to the yellow key and was then suppressed (A) as the internal state approached the setpoint (E). This transfer is due to gradually updating the subjective probability of receiving water outcome upon responding on either key (C). In the fistula group, as the water was not sensed, the outcome expectation converged to zero for both keys (D) and thus, responding was extinguished (B). As a result, the internal state changed only slightly (F). DOI: http://dx.doi.org/10.7554/eLife.04811.020CC BY · PubMed Central

Figure 8—figure supplement 1. A model-based homeostatic RL system. Upon performing an action in a certain state, the agent receives an outcome, Kt, which results in the internal state to shift from Ht to Ht+Kt. At the same time, sensory properties of the outcome are sensed by the agent. Based on this information, the agent updates the state-action-outcome associations. In fact, the agent learns to predict the sensory properties, K^t, of the outcome that is expected to be received upon performing a certain action. Having learned these associations, the agent can estimate the rewarding value of different options. That is, when the agent is in a certain state, it predicts the outcome K^t, expected to result from each behavioral policy. Based on K^t and the internal state Ht, the agent can approximate the drive-reduction reward. DOI: http://dx.doi.org/10.7554/eLife.04811.022CC BY · PubMed Central

Figure 8—figure supplement 2. The Markov Decision Process used for simulating the reinforcing vs. satiation effects of water. At each time point, the agent can choose between doing nothing (nul) or pecking at either the green or the yellow key. DOI: http://dx.doi.org/10.7554/eLife.04811.023CC BY · PubMed Central

Figure 9. Simulation results of the satiation test. Left column shows results for the case where water was received only orally. Rate of responding drops rapidly (A) as the internal state approaches the setpoint (E). Also, the agent learns rapidly that upon every key pecking, it receives 1.0 unit of water (C). On the right column, upon every key-peck, 0.5 unit of water is received orally, and 0.5 unit is received via the fistula. As only oral delivery is sensed by the agent, the subjective outcome-magnitude converges to 0.5 (D). As a result, the reinforcing value of key-pecking is less than that of the oral case and thus, the rate of responding is lower (B). This in turn results in slower convergence of the internal state to the setpoint (F). The MDP and the free parameters used for simulation are the same as in Figure 8. DOI: http://dx.doi.org/10.7554/eLife.04811.024CC BY · PubMed Central

Figure 10. Simulating over-eating of hyperpalatable vs. normal food. (A) The simulated agent can consume normal (T = 0) or hyperpalatable (T > 0) food. The nutritional content, K, of both foods are equal. In the single-option task (C, D), one group of animals can only choose between normal food and nothing (nul), whereas the other group can choose between hyperpalatable food and nothing. Starting the task in a deprived state (initial internal state=−50), the internal state of the second, but not the first, group converges to a level above the setpoint (C) and the total consumption of food is higher in this group (D). In the multiple-choice task, the agents can choose between normal food, hyperpalatable food, and nothing (B). Results show that the hyperpalatable food is preferred over the normal food (E) and the internal state is defended at a level beyond the setpoint (F). See Figure 10—source data 1 for simulation details. DOI: http://dx.doi.org/10.7554/eLife.04811.025CC BY · PubMed Central

Figure 11. Behavioral predictions of the model. (A) Differential predictions of the multiplicative (linear) and drive-reduction (non-linear) forms of reward. In our model, assuming that the internal state is at ht (A), outcomes larger than h^(*)−ht result in overshooting the setpoint and thus a declining trend of the rewarding value (B). Previous models, however, predict the rewarding value to increase linearly as the outcome increases in magnitude. (C) Our model predicts that when given a choice between two options with equal net effects on the internal state, animals choose the option that first results in reducing the homeostatic deviation and then is followed by an increase in deviation (green), as compared to a reversed-order option (red). DOI: http://dx.doi.org/10.7554/eLife.04811.027CC BY · PubMed Central

Figure 12. Simulation results, predicting a transitory burst of responding upon reducing the dose of outcome. Our model (left column) and negative-feedback models (right column) are simulated is a process where responding yields big and small outcomes, during the first and second hours of the experiment, respectively. In our model, the objective is stay as close as possible to the setpoint (A), whereas in previous homeostatic regulation models, the objective is to stay above the setpoint (D). Thus, our model predicts a short-term burst of responding after the dose reduction, followed by regular and escalated response rate (B, C). Classical HR models, however, predict an immediate transition from a steady low to a steady high response rate (E, F). See Figure 12—figure supplements 1 and Figure 12—source data 1 for simulation details. DOI: http://dx.doi.org/10.7554/eLife.04811.028CC BY · PubMed Central

Figure 12—figure supplement 1. The Markov Decision Process used for the within-session dose-change simulation. DOI: http://dx.doi.org/10.7554/eLife.04811.030CC BY · PubMed Central

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
Prospect theory: An analysis of decision under risk
Kahneman, D. J., & Tversky, A. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511609220.014 · Cited by 32,720
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,268
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 3×
“A self-organizing system (i.e. an organism) can be defined as a system that opposes the second law of thermodynamics (Friston, 2010).”
BackgroundMethodInfluential
Orexins and orexin receptors: a family of hypothalamic neuropeptides and G protein-coupled receptors that regulate feeding behavior
Sakurai, T., Amemiya, A., Ishii, M., Matsuzaki, I., Chemelli, R. M., Tanaka, H., Williams, S. C., Richardson, J. A., Kozlowski, G. P., Wilson, S., et al. (1998) · Cell · doi:10.1016/s0092-8674(00)80949-6 · Cited by 5,501
Behavior of Organisms
Skinner, B. F. (1938) · Cited by 3,571
Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
A framework for studying the neurobiology of value-based decision making
Rangel, A., Camerer, C., & Montague, P. R. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2357 · Cited by 2,184

Show all 80 references

Research works printed but not linked: 3

Bernard, 1957, Lectures on the physiological properties and the pathological alternations of the liquids of the organism: Third lecture, 89
Marr, 1982, Vision
Sibly, 1974, State Space Approach to Motivation, Motivational Control System Analysis

Sorted by citations

Allostatic Self-efficacy: A Metacognitive Theory of Dyshomeostasis-Induced Fatigue and Depression
Stephan, K. E., Manjaly, Z. M., Mathys, C. D., Weber, L. A. E., Paliwal, S., Gard, T., Tittgemeyer, M., Fleming, S. M., Haker, H., Seth, A. K., et al. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00550 · Cited by 437
“Current models of decision-making do not incorporate all of these aspects, and first attempts of accounting for homeostasis and allostasis in formal models of decisionmaking have only surfaced relatively recently (e.g., Keramati and Gutkin, 2014; Penny and Stephan, 2014; Pezzulo et al., 2015).”
Background
Growing a social brain
Atzil, S., Gao, W., Fradkin, I., & Barrett, L. F. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0384-6 · Cited by 415
Emotion words, emotion concepts, and emotional development in children: A constructionist hypothesis
Hoemann, K., Xu, F., & Barrett, L. F. (2019) · Developmental Psychology · doi:10.1037/dev0000686 · Cited by 354
Computational Models of Interoception and Body Regulation
Petzschner, F. H., Garfinkel, S. N., Paulus, M. P., Koch, C., & Khalsa, S. S. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2020.09.012 · Cited by 227
Reinforcement learning across development: What insights can we draw from a decade of research?
Nussenbaum, K., & Hartley, C. A. (2019) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2019.100733 · Cited by 225

See all 218

Authors: Keramati, M., & Gutkin, B.
Affiliations: Inserm (FR)
National Research University Higher School of Economics (RU)
Corresponding author: Keramati, Mehdi
Gutkin, Boris
ORCID: Keramati, Mehdi
Gutkin, Boris
Venue: eLife
Publisher: eLife Sciences Publications, Ltd
Published: 2014-12-02
Volume: 3
ISSN: 2050-084X (ISSN-L)
DOI: 10.7554/elife.04811
PMC: PMC4270100
OpenAlex: W2142110758
Cited by: 218
References: 80
Citations by year: 2026: 23 · 2025: 35 · 2024: 29 · 2023: 15 · 2022: 37 · 2021: 23 · 2020: 16 · 2019: 16 · 2018: 10 · 2017: 8 · 2016: 5 · 2015: 1
FWCI: 2.13
Type: journal article
Language: en
Open access: gold
doi.org
Licence: cc-by
creativecommons.org/licenses/by/4.0/ (vor)
creativecommons.org/licenses/by/4.0/ (am)
creativecommons.org/licenses/by/4.0/ (tdm)
Locations: eLife (publishedVersion, cc-by)
bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
PubMed Central (publishedVersion, cc-by)
PubMed (publishedVersion)
City Research Online (City University London) (acceptedVersion)
DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders: Gatsby Charitable Foundation
National Research University Higher School of Economics (Basic Research Program)
Institut national de la santé et de la recherche médicale (INSERM U960, France)
Center for Research and Interdisciplinary (Frontiers du Vivant)
Agence Nationale de la Recherche (ANR-10-LABX-0087 IEC, France, ANR-10-IDEX-0001-02 PSL, France)
Topics: Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords: physiological stability, primary reward, normative theory, delay discounting, predictive learning, hypothalamus, brain reward system, adaptive behavior
MeSH terms: Animals, Behavior, Computer Simulation, Conditioning, Psychological, Extinction, Psychological, Homeostasis, Learning, Models, Neurological, Reinforcement, Psychology, Reward, Satiety Response

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Homeostatic reinforcement learning for integrating reward collection and physiological stability

Abstract

Overview

Integrity signals

Figures

Verification

Claims the paper makes

Claims other papers make about this one

Self-check

Citations

Cites

Cited by

Related

Metadata

Record

About these checks