Homeostatic reinforcement learning for integrating reward collection and physiological stability
Open accessPublished version of a preprint
Abstract
Efficient regulation of internal homeostasis and defending it against perturbations requires adaptive behavioral strategies. However, the computational principles mediating the interaction between homeostatic and associative learning processes remain undefined. Here we use a definition of primary rewards, as outcomes fulfilling physiological needs, to build a normative theory showing how learning motivated behaviors may be modulated by internal states. Within this framework, we mathematically prove that seeking rewards is equivalent to the fundamental objective of physiological stability, defining the notion of physiological rationality of behavior. We further suggest a formal basis for temporal discounting of rewards by showing that discounting motivates animals to follow the shortest path in the space of physiological variables toward the desired setpoint. We also explain how animals learn to act predictively to preclude prospective homeostatic challenges, and several other behavioral patterns. Finally, we suggest a computational role for interaction between hypothalamus and the brain reward system.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Current models of decision-making do not incorporate all of the four aspects of homeostatic utility.
“Current models of decision-making do not incorporate all of these aspects, and first attempts of accounting for homeostasis and allostasis in formal models of decision-making have only surfaced relatively recently (e.g., Keramati and Gutkin, 2014; Penny and Stephan, 2014; Pezzulo et al., 2015).”
From Allostatic Self-efficacy: A Metacognitive Theory of Dyshomeostasis-Induced Fatigue and Depression (2016) · cites this paper as Keramati and Gutkin, 2014
The Introduction (page unavailable) identifies gaps in earlier homeostatic-regulation models and presents this paper as a formal model integrating homeostasis with reinforcement learning. It does not identify the citing paper’s four aspects of homeostatic utility or establish that current decision-making models omit all four. The Discussion, ‘Limitations and future directions’ (page unavailable), also says this model assumes a fixed setpoint rather than an adaptive one. (LLM, conf=0.84)
Experimental — accuracy not yet measuredFirst attempts to account for homeostasis and allostasis in formal models of decision-making have only surfaced relatively recently.
“Current models of decision-making do not incorporate all of these aspects, and first attempts of accounting for homeostasis and allostasis in formal models of decision-making have only surfaced relatively recently (e.g., Keramati and Gutkin, 2014; Penny and Stephan, 2014; Pezzulo et al., 2015).”
From Allostatic Self-efficacy: A Metacognitive Theory of Dyshomeostasis-Induced Fatigue and Depression (2016) · cites this paper as Keramati and Gutkin, 2014
The Introduction describes a formal model linking homeostatic regulation to reinforcement learning, so this 2014 paper is an example of the kind of recent work cited. But the Discussion (Relationship to other theoretical models) treats allostasis as a distinct approach, and Limitations and future directions says this model assumes a fixed setpoint. It therefore does not fully support presenting this paper as a formal model of both homeostasis and allostasis. Page numbers are unavailable in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524 · see citing works
Prospect theory: An analysis of decision under risk
Kahneman, D. J., & Tversky, A. (1988) · Cambridge University Press eBooks · doi:10.1017/cbo9780511609220.014 · Cited by 32,700 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225 · see citing works
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. (2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 25,263 · see citing works
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388 · see citing works
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 3× · see citing works
“A self-organizing system (i.e. an organism) can be defined as a system that opposes the second law of thermodynamics (Friston, 2010).”
BackgroundMethodInfluentialSakurai, T., Amemiya, A., Ishii, M., Matsuzaki, I., Chemelli, R. M., Tanaka, H., Williams, S. C., Richardson, J. A., Kozlowski, G. P., Wilson, S., et al. (1998) · Cell · doi:10.1016/s0092-8674(00)80949-6 · Cited by 5,501 · see citing works
Skinner, B. F. (1938) · Cited by 3,184 · see citing works
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533 · see citing works
A framework for studying the neurobiology of value-based decision making
Rangel, A., Camerer, C., & Montague, P. R. (2008) · Nature Reviews Neuroscience · doi:10.1038/nrn2357 · Cited by 2,184 · see citing works
Molecular basis of long-term plasticity underlying addiction
Nestler, E. J. (2001) · Nature reviews. Neuroscience · doi:10.1038/35053570 · Cited by 1,857 · see citing works
Organization for physiological homeostasis
Cannon, W. B. (1929) · Physiological Reviews · doi:10.1152/physrev.1929.9.3.399 · Cited by 1,844 · see citing works
Interoceptive inference, emotion, and the embodied self
Seth, A. K. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.09.007 · Cited by 1,816 · see citing works
Principles of Behavior. An Introduction to Behavior Theory
N., E., & Hull, C. L. (1943) · The Journal of Philosophy · doi:10.2307/2019960 · Cited by 1,731 · see citing works
Mowrer, O. H. (1960) · doi:10.1037/10802-000 · Cited by 1,670 · see citing works
Gonzalez, R. C., & Mowrer, O. H. (1962) · The American Journal of Psychology · doi:10.2307/1419634 · Cited by 1,584 · see citing works
Cabanac, M. (1971) · Science · doi:10.1126/science.173.4002.1103 · Cited by 1,521 · see citing works
Behavior Theory and Conditioning
Bitterman, M. E., & Spence, K. W. (1957) · The American Journal of Psychology · doi:10.2307/1419251 · Cited by 1,460 · see citing works
Motivation concepts in behavioral neuroscience
Berridge, K. C. (2004) · Physiology & Behavior · doi:10.1016/j.physbeh.2004.02.004 · Cited by 1,302 · see citing works
Progressive Ratio as a Measure of Reward Strength
Hodos, W. (1961) · Science · doi:10.1126/science.134.3483.943 · Cited by 1,147 · see citing works
Allostasis: A model of predictive regulation
Sterling, P. (2012) · Physiology & Behavior · doi:10.1016/j.physbeh.2011.06.004 · Cited by 1,073 · see citing works
Every good regulator of a system must be a model of that system †
CONANT, R. C., & ROSS ASHBY, W. (1970) · International Journal of Systems Science · doi:10.1080/00207727008920220 · Cited by 1,063 · see citing works
Behavior theory and conditioning.
Spence, K. W. (1956) · doi:10.1037/10029-000 · Cited by 872 · see citing works
Every Good Regulator of a System Must Be a Model of That System
Conant, R. C., & Ashby, W. R. (1991) · Facets of Systems Science · doi:10.1007/978-1-4899-0718-9_37 · Cited by 712 · see citing works
From prediction error to incentive salience: mesolimbic computation of reward motivation.
Berridge, K. C. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.07990.x · Cited by 700 · see citing works
Mas-Colell, A. 1., Whinston, M. D. 1., & Green, J. R. 1. (1995) · Cited by 691 · see citing works
Billings, H. J. (2019) · The Research Repository @ WVU (West Virginia University) · Cited by 665 · see citing works
Excitation of Ventral Tegmental Area Dopaminergic and Nondopaminergic Neurons by Orexins/Hypocretins
Korotkova, T. M., Sergeeva, O. A., Eriksson, K. S., Haas, H. L., & Brown, R. E. (2003) · Journal of Neuroscience · doi:10.1523/jneurosci.23-01-00007.2003 · Cited by 583 · see citing works
CHOICE AND DELAY OF REINFORCEMENT1
Chung, S., & Herrnstein, R. J. (1967) · Journal of the Experimental Analysis of Behavior · doi:10.1901/jeab.1967.10-67 · Cited by 540 · see citing works
Narita, M., Nagumo, Y., Hashimoto, S., Narita, M., Khotib, J., Miyatake, M., Sakurai, T., Yanagisawa, M., Nakamachi, T., Shioda, S., et al. (2006) · Journal of Neuroscience · doi:10.1523/jneurosci.2761-05.2006 · Cited by 516 · see citing works
An Evolutionary Perspective on Food and Human Taste
Breslin, P. A. S. (2013) · Current Biology · doi:10.1016/j.cub.2013.04.010 · Cited by 512 · see citing works
Is dopamine a physiologically relevant mediator of feeding behavior?
Palmiter, R. D. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.06.004 · Cited by 469 · see citing works
Food Reward in the Absence of Taste Receptor Signaling
de Araujo, I. E., Oliveira-Maia, A. J., Sotnikova, T. D., Gainetdinov, R. R., Caron, M. G., Nicolelis, M. A. L., & Simon, S. A. (2008) · Neuron · doi:10.1016/j.neuron.2008.01.032 · Cited by 431 · see citing works
From neuroanatomy to behavior: central integration of peripheral signals regulating feeding behavior
Williams, K. W., & Elmquist, J. K. (2012) · Nature Neuroscience · doi:10.1038/nn.3217 · Cited by 421 · see citing works
Intrinsically Motivated Reinforcement Learning: An Evolutionary Perspective
Singh, S. P., Lewis, R. L., Barto, A. G., & Sorg, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2051031 · Cited by 418 · see citing works
The Role of Learning in the Operation of Motivational Systems
Dickinson, A., & Balleine, B. W. (2002) · Wiley eBooks · doi:10.1002/0471214426.pas0312 · Cited by 387 · see citing works
Burdakov, D., Gerasimenko, O. V., & Verkhratsky, A. (2005) · Journal of Neuroscience · doi:10.1523/jneurosci.4925-04.2005 · Cited by 337 · see citing works
The eating paradox: How we tolerate food.
Woods, S. C. (1991) · Psychological Review · doi:10.1037/0033-295x.98.4.488 · Cited by 337 · see citing works
Speakman, J. R., Levitsky, D. A., Allison, D. B., Bray, M. S., de Castro, J. M., Clegg, D. J., Clapham, J. C., Dulloo, A. G., Gruer, L. D., Haw, S. J., et al. (2011) · Disease Models & Mechanisms · doi:10.1242/dmm.008698 · Cited by 316 · see citing works
A Neural Computational Model of Incentive Salience
Zhang, J., Berridge, K. C., Tindell, A. J., Smith, K. S., & Aldridge, J. W. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000437 · Cited by 299 · see citing works
Unraveling the brain regulation of appetite: lessons from genetics
Yeo, G. S. H., & Heisler, L. K. (2012) · Nature Neuroscience · doi:10.1038/nn.3211 · Cited by 281 · see citing works
Zhang, M., & Kelley, A. E. (2000) · Neuroscience · doi:10.1016/s0306-4522(00)00198-6 · Cited by 260 · see citing works
Regulation of dietary choice by the decision-making circuitry
Rangel, A. (2013) · Nature Neuroscience · doi:10.1038/nn.3561 · Cited by 246 · see citing works
Rodgers, R. J., Halford, J. C. G., de Souza, R. L. N., de Souza, A. L. C., Piper, D. C., Arch, J. R. S., Upton, N., Porter, R. A., Johns, A., & Blundell, J. E. (2001) · European Journal of Neuroscience · doi:10.1046/j.0953-816x.2001.01518.x · Cited by 230 · see citing works
High-intensity sweeteners and energy balance.
Swithers, S. E., Martin, A. A., & Davidson, T. L. (2010) · Physiology & Behavior · doi:10.1016/j.physbeh.2009.12.021 · Cited by 206 · cited 1× · see citing works
“Such association between sensory and post-ingestive properties could have been developed through prior learning (Swithers et al., 2009; Swithers et al., 2010; Beeler et al., 2012) or evolutionary mechanisms (Breslin, 2013).”
Differential effects of morphine on food and water intake in food deprived and freely-feeding rats
Sanger, D. J., & McCarthy, P. S. (1980) · Psychopharmacology · doi:10.1007/bf00433813 · Cited by 205 · see citing works
Cornelius, J. R., Tippmann‐Peikert, M., Slocumb, N. L., Frerichs, C. F., & Silber, M. H. (2010) · SLEEP · doi:10.1093/sleep/33.1.81 · Cited by 199 · see citing works
Conditioning and extinction of tolerance to the hypothermic effect of ethanol in rats.
Mansfield, J. G., & Cunningham, C. L. (1980) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0077824 · Cited by 199 · see citing works
Opiate agonists microinjected into the nucleus accumbens enhance sucrose drinking in rats
Zhang, M., & Kelley, A. E. (1997) · Psychopharmacology · doi:10.1007/s002130050355 · Cited by 191 · see citing works
Set points, settling points, and the control of body weight☆
Wirtshafter, D., & DAVIS, J. (1977) · Physiology & Behavior · doi:10.1016/0031-9384(77)90162-7 · Cited by 182 · see citing works
Solinas, M., & Goldberg, S. R. (2005) · Neuropsychopharmacology · doi:10.1038/sj.npp.1300720 · Cited by 174 · see citing works
Nutrient Selection in the Absence of Taste Receptor Signaling
Ren, X., Ferreira, J. G. P., Zhou, L.-W., Shammah‐Lagnado, S. J., Yeckel, C. W., & de Araújo, I. E. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.5749-09.2010 · Cited by 162 · see citing works
(2002) · Advances in consciousness research · doi:10.1075/aicr.44 · Cited by 148 · see citing works
Understanding the mechanisms of food intake and obesity
Rolls, E. T. (2007) · Obesity Reviews · doi:10.1111/j.1467-789x.2007.00321.x · Cited by 148 · see citing works
Biobehavioral Influences on Energy Intake and Adult Weight Gain
McCrory, M. A., Suen, V. M. M., & Roberts, S. B. (2002) · Journal of Nutrition · doi:10.1093/jn/132.12.3830s · Cited by 136 · see citing works
Morphine enhances hedonic taste palatability in rats
Doyle, T. G., Berridge, K. C., & Gosnell, B. A. (1993) · Pharmacology Biochemistry and Behavior · doi:10.1016/0091-3057(93)90572-b · Cited by 131 · see citing works
Reward effects of food via stomach fistula compared with those of food via mouth.
Miller, N. E., & Kessen, M. L. (1952) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0060113 · Cited by 111 · see citing works
Swithers, S. E., Baker, C. R., & Davidson, T. L. (2009) · Behavioral Neuroscience · doi:10.1037/a0016139 · Cited by 105 · see citing works
Homeostasis: Beyond Curt Richter
Woods, S. C., & Ramsay, D. S. (2007) · Appetite · doi:10.1016/j.appet.2006.09.015 · Cited by 100 · see citing works
Potency of naloxone's anorectic effect in rats is dependent on diet preference
Glass, M. J., Grace, M. K., Cleary, J. P., Billington, C. J., & Levine, A. (1996) · American Journal of Physiology-Regulatory, Integrative and Comparative Physiology · doi:10.1152/ajpregu.1996.271.1.r217 · Cited by 97 · see citing works
Lower pleasantness of palatable foods in nalmefene-treated human volunteers
Yeomans, M. R., & WRIGHT, P. D. (1991) · Appetite · doi:10.1016/0195-6663(91)90062-w · Cited by 96 · see citing works
Naloxone effects on sucrose-motivated behavior
Cleary, J. P., Weldon, D. T., O‘Hare, E., Billington, C. J., & Levine, A. S. (1996) · Psychopharmacology · doi:10.1007/bf02246345 · Cited by 90 · see citing works
Skjoldager, P. D., Pierre, P. J., & Mittleman, G. (1993) · Learning and Motivation · doi:10.1006/lmot.1993.1019 · Cited by 88 · see citing works
Will, M. J., Pratt, W. E., & Kelley, A. E. (2006) · Physiology & Behavior · doi:10.1016/j.physbeh.2006.06.008 · Cited by 87 · see citing works
Homeostasis: Origins of the Concept
Langley, L. L., & Johnson, S. H. (1974) · Journal of Dynamic Systems Measurement and Control · doi:10.1115/1.3426765 · Cited by 79 · see citing works
Devine, D. P., Leone, P., & Wise, R. A. (1993) · European Journal of Pharmacology · doi:10.1016/0014-2999(93)90167-g · Cited by 75 · see citing works
Taste uncoupled from nutrition fails to sustain the reinforcing properties of food
Beeler, J. A., McCutcheon, J. E., Cao, Z. F. H., Murakami, M., Alexander, E. R., Roitman, M. F., & Zhuang, X. (2012) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2012.08167.x · Cited by 73 · see citing works
Dopaminergic and non-dopaminergic value systems in conditioning and outcome-specific revaluation
Dranias, M. R., Grossberg, S., & Bullock, D. H. (2008) · Brain Research · doi:10.1016/j.brainres.2008.07.013 · Cited by 72 · see citing works
Is there evidence for a set point that regulates human body weight?
Müller, M. J., Bosy‐Westphal, A., & Heymsfield, S. B. (2010) · F1000 Medicine Reports · doi:10.3410/m2-59 · Cited by 72 · see citing works
Marieb, E. N. 1., & Smith, L. A. (2022) · Cited by 67 · see citing works
Barbano, M. F., Saux, M. L., & Cador, M. (2008) · Psychopharmacology · doi:10.1007/s00213-008-1390-6 · Cited by 56 · see citing works
Orosensory Self‐Stimulation by Sucrose Involves Brain Dopaminergic Mechanismsa
Schneider, L. H. (1989) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1989.tb53252.x · Cited by 48 · see citing works
Mansfield, J. G., Benedict, R. S., & Woods, S. C. (1983) · Psychopharmacology · doi:10.1007/bf00427791 · Cited by 40 · see citing works
Ventral tegmental injections of morphine but not U-50,488H enhance feeding in food-deprived rats
Noel, M. B., & Wise, R. A. (1993) · Brain Research · doi:10.1016/0006-8993(93)91139-j · Cited by 37 · see citing works
9. Motivational underpinnings of utility in decision making
Busemeyer, J. R., Townsend, J. T., & Stout, J. C. (2002) · Advances in consciousness research · doi:10.1075/aicr.44.09bus · Cited by 27 · see citing works
Separation of satiating and rewarding consequences of drinking
McFarland, D. J. (1969) · Physiology & Behavior · doi:10.1016/0031-9384(69)90054-7 · Cited by 26 · see citing works
Tolerance to hypothermia induced by ethanol depends on specific drug effects
Hjeresen, D. L., Reed, D. R., & Woods, S. C. (1986) · Psychopharmacology · doi:10.1007/bf00175187 · Cited by 25 · see citing works
Human Anatomy & Physiology 9th Edition
Elaine, N. M., & Katja, H. (2013) · Cited by 17 · see citing works
Artificial motives: A review of motivation in artificial creatures
Savage, T. (2000) · Connection Science · doi:10.1080/095400900750060131 · Cited by 16 · see citing works
Woods, S. C., & Seeley, R. J. (2002) · Wiley eBooks · doi:10.1002/0471214426.pas0315 · Cited by 12 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Dickinson A Balleine BW 2002 The role of learning in motivation Gallistel CR Volume 3 of Steven's Handbook of experimental psychology: learning, motivation, and Emotion 3rd edition New York Wiley 497 533
- Marr D 1982 Vision Cambridge, Massachusetts MIT Press
- Sibly RM McFarland DJ 1974 State Space Approach to Motivation, Motivational Control System Analysis Academic Press
Cited by
Sorted by citations
Allostatic Self-efficacy: A Metacognitive Theory of Dyshomeostasis-Induced Fatigue and Depression
Stephan, K. E., Manjaly, Z. M., Mathys, C. D., Weber, L. A. E., Paliwal, S., Gard, T., Tittgemeyer, M., Fleming, S. M., Haker, H., Seth, A. K., et al. (2016) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2016.00550 · Cited by 437 · see citing works
“Current models of decision-making do not incorporate all of these aspects, and first attempts of accounting for homeostasis and allostasis in formal models of decisionmaking have only surfaced relatively recently (e.g., Keramati and Gutkin, 2014; Penny and Stephan, 2014; Pezzulo et al., 2015).”
BackgroundAtzil, S., Gao, W., Fradkin, I., & Barrett, L. F. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0384-6 · Cited by 415 · see citing works
Emotion words, emotion concepts, and emotional development in children: A constructionist hypothesis
Hoemann, K., Xu, F., & Barrett, L. F. (2019) · Developmental Psychology · doi:10.1037/dev0000686 · Cited by 354 · see citing works
Computational Models of Interoception and Body Regulation
Petzschner, F. H., Garfinkel, S. N., Paulus, M. P., Koch, C., & Khalsa, S. S. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2020.09.012 · Cited by 227 · see citing works
Reinforcement learning across development: What insights can we draw from a decade of research?
Nussenbaum, K., & Hartley, C. A. (2019) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2019.100733 · Cited by 203 · see citing works
Estimation of Current and Future Physiological States in Insular Cortex
Livneh, Y., Sugden, A. U., Madara, J. C., Essner, R. A., Flores, V. I., Sugden, L. A., Resch, J. M., Lowell, B. B., & Andermann, M. L. (2020) · Neuron · doi:10.1016/j.neuron.2019.12.027 · Cited by 209 · see citing works
Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., Deane, G., Fleming, S. M., Frith, C., Ji, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08708 · Cited by 196 · see citing works
Petzschner, F. H., Weber, L. A. E., Gard, T., & Stephan, K. E. (2017) · Biological Psychiatry · doi:10.1016/j.biopsych.2017.05.012 · Cited by 181 · see citing works
Homeostasis and soft robotics in the design of feeling machines
Man, K., & Damásio, A. R. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0103-7 · Cited by 174 · see citing works
The brain’s reward system in health and disease
Lewis, R. G., Florio, E., Punzo, D., & Borrelli, E. (2021) · Advances in experimental medicine and biology · doi:10.1007/978-3-030-81147-1_4 · Cited by 170 · see citing works
Pain: A Precision Signal for Reinforcement Learning and Control
Seymour, B. (2019) · Neuron · doi:10.1016/j.neuron.2019.01.055 · Cited by 164 · see citing works
Corcoran, A. W., Pezzulo, G., & Hohwy, J. (2020) · Biology & Philosophy · doi:10.1007/s10539-020-09746-2 · Cited by 161 · see citing works
Long, M., Verbeke, W., Ein‐Dor, T., & Vrtička, P. (2020) · Cortex · doi:10.1016/j.cortex.2020.01.010 · Cited by 160 · see citing works
Juechems, K., & Summerfield, C. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.07.012 · Cited by 147 · see citing works
The whole prefrontal cortex is premotor cortex
Fine, J. M., & Hayden, B. Y. (2021) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2020.0524 · Cited by 139 · see citing works
Consciousness matters: phenomenal experience has functional value
Cleeremans, A., & Tallon‐Baudry, C. (2022) · Neuroscience of Consciousness · doi:10.1093/nc/niac007 · Cited by 158 · see citing works
Vagus nerve stimulation boosts the drive to work for rewards
Neuser, M. P., Teckentrup, V., Kühnel, A., Hallschmid, M., Walter, M., & Kroemer, N. B. (2020) · Nature Communications · doi:10.1038/s41467-020-17344-9 · Cited by 105 · see citing works
Interoception as modeling, allostasis as control
Sennesh, E., Theriault, J., Brooks, D., van de Meent, J.-W., Barrett, L. F., & Quigley, K. S. (2022) · Biological Psychology · doi:10.1016/j.biopsycho.2021.108242 · Cited by 107 · see citing works
Livneh, Y., & Andermann, M. L. (2021) · Neuron · doi:10.1016/j.neuron.2021.08.036 · Cited by 106 · see citing works
Dopamine role in learning and action inference
Bogacz, R. (2020) · eLife · doi:10.7554/elife.53262 · Cited by 79 · see citing works
Context-dependent outcome encoding in human reinforcement learning
Palminteri, S., & Lebreton, M. P. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2021.06.006 · Cited by 78 · see citing works
Emotions as ComputationsPreprint
Emanuel, A., & Eldar, E. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104977 · Cited by 83 · see citing works
An Analysis of Decision under Risk in Rats
Constantinople, C. M., Piet, A. T., & Brody, C. D. (2019) · Current Biology · doi:10.1016/j.cub.2019.05.013 · Cited by 75 · see citing works
Modelling Stock Markets by Multi-agent Reinforcement Learning
Lussange, J., Lazarevich, I., Bourgeois-Gironde, S., Palminteri, S., & Gutkin, B. (2020) · Computational Economics · doi:10.1007/s10614-020-10038-w · Cited by 81 · see citing works
Self-citationMulti-task reinforcement learning in humans
Tomov, M. S., Schulz, E., & Gershman, S. J. (2021) · Nature Human Behaviour · doi:10.1038/s41562-020-01035-y · Cited by 60 · see citing works
A goal-centric outlook on learning
Molinaro, G., & Collins, A. G. E. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.08.011 · Cited by 63 · see citing works
Arnaldo, I., Corcoran, A. W., Friston, K. J., & Ramstead, M. J. D. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104590 · Cited by 56 · see citing works
Valuing what happens: a biogenic approach to valence and (potentially) affect
Lyon, P., & Kuchling, F. (2021) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2019.0752 · Cited by 56 · see citing works
Bronleigh, M., Baumann, O., & Stapleton, P. (2022) · Eating and Weight Disorders - Studies on Anorexia Bulimia and Obesity · doi:10.1007/s40519-022-01390-x · Cited by 54 · see citing works
Reinforcement learning with Marr
Niv, Y., & Langdon, A. (2016) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2016.04.005 · Cited by 49 · see citing works
Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation
Kato, A., & Morita, K. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005145 · Cited by 57 · see citing works
Azzalini, D., Buot, A., Palminteri, S., & Tallon-Baudry, C. (2021) · The Journal of Neuroscience · doi:10.1523/jneurosci.1932-20.2021 · Cited by 47 · see citing works
Active inference models do not contradict folk psychology
Smith, R. S., Ramstead, M. J. D., & Kiefer, A. B. (2022) · Synthese · doi:10.1007/s11229-022-03480-w · Cited by 50 · see citing works
Cocaine addiction as a homeostatic reinforcement learning disorder.
Keramati, M., Durand, A., Girardeau, P., Gutkin, B., & Ahmed, S. H. (2017) · Psychological Review · doi:10.1037/rev0000046 · Cited by 44 · see citing works
Self-citationChakravarthy, V. S., Balasubramani, P. P., Mandali, A., Jahanshahi, M., & Moustafa, A. A. (2018) · Physiology & Behavior · doi:10.1016/j.physbeh.2018.06.032 · Cited by 43 · see citing works
Stimulation of the vagus nerve reduces learning in a go/no-go reinforcement learning task
Kühnel, A., Teckentrup, V., Neuser, M. P., Huys, Q. J. M., Burrasch, C., Walter, M., & Kroemer, N. B. (2020) · European Neuropsychopharmacology · doi:10.1016/j.euroneuro.2020.03.023 · Cited by 43 · see citing works
A dopamine mechanism for reward maximization
Schultz, W. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2316658121 · Cited by 44 · see citing works
A Network for Computing Value Equilibrium in the Human Medial Prefrontal Cortex
Juechems, K., Balaguer, J., Castañón, S. H., Ruz, M., O’Reilly, J. X., & Summerfield, C. (2019) · Neuron · doi:10.1016/j.neuron.2018.12.029 · Cited by 41 · see citing works
Post-injury pain and behaviour: a control theory perspective
Seymour, B., Crook, R. J., & Chen, Z. S. (2023) · Nature reviews. Neuroscience · doi:10.1038/s41583-023-00699-5 · Cited by 41 · see citing works
A rubric for human-like agents and NeuroAI
Momennejad, I. (2022) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2021.0446 · Cited by 41 · see citing works
Goals, usefulness and abstraction in value-based choice
Martino, B. D., & Cortese, A. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.11.001 · Cited by 41 · see citing works
The affective gradient hypothesis: An affect-centered account of motivated behavior
Shenhav, A. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.08.003 · Cited by 43 · see citing works
Adaptive Empathy: A Model for Learning Empathic Responses in Response to Feedback
Shamay-Tsoory, S. G., & Hertz, U. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916211031926 · Cited by 33 · see citing works
The function of consciousness is to generate experiencePreprint
Cleeremans, A., & Tallon‐Baudry, C. (2021) · doi:10.31234/osf.io/jfpw2 · Cited by 34 · see citing works
Computational Mechanisms of Addiction: Recent Evidence and Its Relevance to Addiction Medicine
Smith, R. S., Taylor, S., & Bilek, E. (2021) · Current Addiction Reports · doi:10.1007/s40429-021-00399-z · Cited by 32 · see citing works
Neurocomputational theories of homeostatic control.
Hulme, O., Morville, T., & Gutkin, B. (2019) · Physics of Life Reviews · doi:10.1016/j.plrev.2019.07.005 · Cited by 30 · see citing works
Self-citationLearning Outside the Brain: Integrating Cognitive Science and Systems Biology
Gunawardena, J. (2022) · Proceedings of the IEEE · doi:10.1109/jproc.2022.3162791 · Cited by 31 · see citing works
A parabrachial hub for need-state control of enduring pain
Goldstein, N., Maes, A., Allen, H. N., Nelson, T. S., Kruger, K. A., Kindel, M., Yeung, A. T. M., Smith, N. K., Carty, J. R. E., Boccia, L., et al. (2025) · Nature · doi:10.1038/s41586-025-09602-x · Cited by 35 · see citing works
Intrinsic rewards explain context-sensitive valuation in reinforcement learning
Molinaro, G., & Collins, A. G. E. (2023) · PLoS Biology · doi:10.1371/journal.pbio.3002201 · Cited by 29 · see citing works
Maintaining Homeostasis by Decision-Making
Korn, C. W., & Bach, D. R. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004301 · Cited by 26 · see citing works
The computational structure of consummatory anhedonia
Hall, A., Browning, M. D., & Huys, Q. J. M. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.006 · Cited by 29 · see citing works
Dynamics of glia and neurons regulate homeostatic rest, sleep and feeding behavior in Drosophila
Flores-Valle, A., Vishniakou, I., & Seelig, J. D. (2025) · Nature Neuroscience · doi:10.1038/s41593-025-01942-1 · Cited by 33 · see citing works
To Do or Not to Do: Dopamine, Affordability and the Economics of Opportunity
Beeler, J. A., & Mourra, D. (2018) · Frontiers in Integrative Neuroscience · doi:10.3389/fnint.2018.00006 · Cited by 27 · see citing works
A mathematical model of reward-mediated learning in drug addiction
Chou, T., & D'Orsogna, M. R. (2022) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/5.0082997 · Cited by 23 · see citing works
“Liking” as an early and editable draft of long-run affective value
Dayan, P. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001476 · Cited by 26 · see citing works
Tian, L., Oviatt, S. L., Muszyński, M., Chamberlain, B. C., Healey, J., & Sano, A. (2022) · ACM eBooks · doi:10.1145/3502398 · Cited by 23 · see citing works
Modeling the effects of motivation on choice and learning in the basal ganglia
van Swieten, M. M. H., & Bogacz, R. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007465 · Cited by 22 · see citing works
A Computational View on the Nature of Reward and Value in Anhedonia
Huys, Q. J. M., & Browning, M. D. (2021) · Current topics in behavioral neurosciences · doi:10.1007/7854_2021_290 · Cited by 23 · see citing works
Neural Networks With Motivation
Shuvaev, S. A., Tran, N. B., Stephenson‐Jones, M., Li, B., & Koulakov, A. A. (2021) · UCL Discovery (University College London) · Cited by 20 · see citing works
Exploring internal body heat balance to understand thermal sensation
Kingma, B., Schweiker, M., Wagner, A., & Lichtenbelt, W. D. V. M. (2017) · Building Research & Information · doi:10.1080/09613218.2017.1299996 · Cited by 20 · see citing works
The interoceptive origin of reinforcement learning
Weber, L. A. E., Yee, D. M., Small, D. M., & Petzschner, F. H. (2025) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2025.05.008 · Cited by 26 · see citing works
Having multiple selves helps learning agents explore and adapt in complex changing worlds
Dulberg, Z., Dubey, R., Berwian, I. M., & Cohen, J. D. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2221180120 · Cited by 19 · see citing works
Artificial consciousness: the missing ingredient for ethical AI?
Chella, A. (2023) · Frontiers in Robotics and AI · doi:10.3389/frobt.2023.1270460 · Cited by 19 · see citing works
Badiani, A., Caprioli, D., & Pirro, S. D. (2019) · Psychopharmacology · doi:10.1007/s00213-019-05318-9 · Cited by 20 · see citing works
Huang, F.-Y., Sutcliffe, M. P. F., & Fabian Grabenhorst, B. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2101954118 · Cited by 19 · see citing works
Interrupting behaviour: Minimizing decision costs via temporal commitment and low-level interrupts
Lloyd, K., & Dayan, P. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005916 · Cited by 21 · see citing works
What Is Social about Autism? The Role of Allostasis-Driven Learning
Djerassi, M., Ophir, S., & Atzil, S. (2021) · Brain Sciences · doi:10.3390/brainsci11101269 · Cited by 20 · see citing works
Misdeed of the need: towards computational accounts of transition to addiction
Keramati, M., Ahmed, S. H., & Gutkin, B. S. (2017) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2017.08.014 · Cited by 17 · see citing works
Self-citationThe Homeostatic Logic of RewardPreprint
Morville, T., Friston, K. J., Burdakov, D., Siebner, H. R., & Hulme, O. J. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/242974 · Cited by 19 · see citing works
Old Strategies, New Environments: Reinforcement Learning on Social Media
Turner, G., Ferguson, A. M., Katiyar, T., Palminteri, S., & Orben, A. C. (2024) · Biological Psychiatry · doi:10.1016/j.biopsych.2024.12.012 · Cited by 26 · see citing works
Mitchell, D., Maloney, S. K., Snelling, E. P., Hetem, R. S., & Fuller, A. (2025) · Acta Physiologica · doi:10.1111/apha.70063 · Cited by 17 · see citing works
Physiological Needs: Sensations and Predictions in the Insular Cortex
Prilutski, Y., & Livneh, Y. (2022) · Physiology · doi:10.1152/physiol.00019.2022 · Cited by 17 · see citing works
Chalmers, E., Luczak, A., & Gruber, A. J. (2016) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2016.00128 · Cited by 16 · see citing works
Bronfman, Z. Z., Ginsburg, S., & Jablonka, E. (2021) · Journal of Artificial Intelligence and Consciousness · doi:10.1142/s2705078521500168 · Cited by 12 · see citing works
Neurobiology of Thirst and Hunger Drives
Encarnacion-Rivera, L., Deisseroth, K. A., & Luo, L. (2025) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-112723-032328 · Cited by 16 · see citing works
High‐fat diet‐induced elevation of body weight set point in male mice
Fang, L. Z., Vidal, J. A. L., Hawlader, O., & Hirasawa, M. (2023) · Obesity · doi:10.1002/oby.23650 · Cited by 15 · see citing works
Emergence of integrated behaviors through direct optimization for homeostasis
Yoshida, N., Daikoku, T., Nagai, Y., & Kuniyoshi, Y. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106379 · Cited by 15 · see citing works
Dahl, R. E., Armstrong‐Carter, E., & van den Bos, W. (2024) · Developmental Cognitive Neuroscience · doi:10.1016/j.dcn.2024.101430 · Cited by 16 · see citing works
Reward Bases: A simple mechanism for adaptive acquisition of multiple reward types
Millidge, B., Song, Y., Lak, A., Walton, M. E., & Bogacz, R. (2024) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012580 · Cited by 20 · see citing works
The topological space of subjective experience
Tallon‐Baudry, C. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.09.002 · Cited by 15 · see citing works
Drive competition underlies effective allostatic orchestration
Rosado, O. G., Amil, A. F., Freire, I. T., & Verschure, P. F. M. J. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.1052998 · Cited by 12 · see citing works
Torres–Martínez, S. (2025) · AI & Society · doi:10.1007/s00146-024-02165-x · Cited by 14 · see citing works
Hypothalamic-Extended Amygdala Circuit Regulates Temporal Discounting
Li, H. E., Rossi, M. A., Watson, G. D. R., Moore, H. G., Cai, M. T., Kim, N., Vokt, K. A., Lu, D., Bartholomew, R. A., Hughes, R. N., et al. (2021) · Journal of Neuroscience · doi:10.1523/jneurosci.1836-20.2020 · Cited by 11 · see citing works
Context-dependent outcome encoding in human reinforcement learningPreprint
Palminteri, S., & Lebreton, M. P. (2021) · doi:10.31234/osf.io/4qh2d · Cited by 12 · see citing works
Neural and Computational Mechanisms of Motivation and Decision-making
Yee, D. M. (2024) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_02258 · Cited by 11 · see citing works
Duriez, A., Bergerot, C., Cone, J. J., Roitman, M. F., & Gutkin, B. S. (2023) · Nutrients · doi:10.3390/nu15041015 · Cited by 12 · see citing works
Self-citationA Control Theory Model of Smoking
Bobashev, G., Holloway, J. W., Solano, E., & Gutkin, B. S. (2017) · doi:10.3768/rtipress.2017.op.0040.1706 · Cited by 10 · see citing works
Self-citationA probabilistic model of relapse in drug addiction
Mao, S., Chou, T., & D'Orsogna, M. R. (2024) · Mathematical Biosciences · doi:10.1016/j.mbs.2024.109184 · Cited by 10 · see citing works
Interoception in pregnancy: Implications for peripartum depression
Savoca, P. W., Glynn, L. M., Fox, M. M., Richards, M., & Callaghan, B. L. (2024) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2024.105874 · Cited by 11 · see citing works
Computational Mechanisms of Osmoregulation: A Reinforcement Learning Model for Sodium Appetite
Uchida, Y., Hikida, T., & Yamashita, Y. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.857009 · Cited by 9 · see citing works
31st Annual Computational Neuroscience Meeting: CNS*2022
Corsi, M., Sorrentino, P., Schwartz, D., George, N., Hugueville, L., Kahn, A. E., Dupont, S., Bassett, D. S., Jirsa, V., & Fallani, F. D. V. (2023) · Journal of Computational Neuroscience · doi:10.1007/s10827-022-00841-9 · Cited by 10 · see citing works
Ghrelin is associated with an elevated mood after an overnight fast in depression
Fahed, R., Schulz, C., Klaus, J., Ellinger, S., Walter, M., & Kroemer, N. B. (2024) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2024.04.053 · Cited by 10 · see citing works
Computational Neural Mechanisms of Goal-Directed Planning and Problem Solving
Fine, J. M., Zarr, N., & Brown, J. W. (2020) · Computational Brain & Behavior · doi:10.1007/s42113-020-00095-7 · Cited by 9 · see citing works
A normative framework dissociates need and motivation in hypothalamic neurons
Kim, K. S., Lee, Y. H., Kim, Y., Yun, J. W., Song, H. Y., Park, J. S., Jung, S.-H., Sohn, J., Kim, K. W., Kim, H. R., et al. (2024) · Science Advances · doi:10.1126/sciadv.ado1820 · Cited by 9 · see citing works
Nutrient-Sensitive Reinforcement Learning in Monkeys
Huang, F.-Y., & Fabian Grabenhorst, B. (2023) · Journal of Neuroscience · doi:10.1523/jneurosci.0752-22.2022 · Cited by 9 · see citing works
The nervous system as a solution for implementing closed negative feedback control loops
Sosa, R., & Alcalá, E. (2022) · Journal of the Experimental Analysis of Behavior · doi:10.1002/jeab.736 · Cited by 10 · see citing works
The Secret of Human Existence Homeostasis: Spiritual Intelligence is the Hope of All Humanity
Alkhatib, A. J. (2019) · Open Access Journal of Addiction and Psychology · doi:10.33552/oajap.2019.02.000537 · Cited by 8 · see citing works
Smith, B. J., & Read, S. J. (2021) · Cognitive Affective & Behavioral Neuroscience · doi:10.3758/s13415-021-00953-2 · Cited by 9 · see citing works
Predictive Processing of Interoception, Decision-Making, and Allostasis
Ohira, H. (2020) · Psihologijske teme · doi:10.31820/pt.29.1.1 · Cited by 8 · see citing works
The Emotional Divergent–Convergent Thinking Program (EDICOP): Design, Implementation, and Results
Soroa, G., Aritzeta, A., Muela, A., Balluerka, N., Gorostiaga, A., & Aliri, J. (2020) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe10040074 · Cited by 8 · see citing works
- Authors
- Keramati, M., & Gutkin, B.
- Affiliations
- Inserm (FR)
- National Research University Higher School of Economics (RU)
- Corresponding author
- Keramati, Mehdi
- Gutkin, Boris
- Venue
- eLife
- Publisher
- eLife Sciences Publications, Ltd
- Published
- 2014-12-02
- Volume
- 3
- ISSN
- 2050-084X (ISSN-L)
- PMC
- PMC4270100
- OpenAlex
- W2142110758
- Cited by
- 218
- References
- 80
- Citations by year
- 2026: 23 · 2025: 35 · 2024: 29 · 2023: 15 · 2022: 37 · 2021: 23 · 2020: 16 · 2019: 16 · 2018: 10 · 2017: 8 · 2016: 5 · 2015: 1
- FWCI
- 2.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- eLife (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- City Research Online (City University London) (acceptedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Gatsby Charitable Foundation
- National Research University Higher School of Economics (Basic Research Program)
- Institut national de la santé et de la recherche médicale (INSERM U960, France)
- Center for Research and Interdisciplinary (Frontiers du Vivant)
- Agence Nationale de la Recherche (ANR-10-LABX-0087 IEC, France, ANR-10-IDEX-0001-02 PSL, France)
- Topics
- Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurotransmitter Receptor Influence on Behavior
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- physiological stability, primary reward, normative theory, delay discounting, predictive learning, hypothalamus, brain reward system, adaptive behavior
- MeSH terms
- Animals, Behavior, Computer Simulation, Conditioning, Psychological, Extinction, Psychological, Homeostasis, Learning, Models, Neurological, Reinforcement, Psychology, Reward, Satiety Response
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
