The DOI matches its DataCite record.
Modularity benefits reinforcement learning agents with competing homeostatic drives
Open accessAbstract
The problem of balancing conflicting needs is fundamental to intelligence. Standard reinforcement learning algorithms maximize a scalar reward, which requires combining different objective-specific rewards into a single number. Alternatively, different objectives could also be combined at the level of action value, such that specialist modules responsible for different objectives submit different action suggestions to a decision process, each based on rewards that are independent of one another. In this work, we explore the potential benefits of this alternative strategy. We investigate a biologically relevant multi-objective problem, the continual homeostasis of a set of variables, and compare a monolithic deep Q-network to a modular network with a dedicated Q-learner for each variable. We find that the modular agent: a) requires minimal exogenously determined exploration; b) has improved sample efficiency; and c) is more robust to out-of-domain perturbation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
To solve the nl-MOMDP task, the bee must use sensory inputs, representations of internal state and stored knowledge of value to output pairs of actions and corresponding expected value vectors (Q-values).
“First, the bee must use sensory inputs, representations of internal state and stored knowledge of value to output pairs of actions and corresponding expected value vectors (Q-values [22,28]).”
From Phenomenal interface theory: a model for basal consciousness (2025) · cites this paper as [22]
The Environment section (p. 2) describes perceptual input and internal-state variables, and Models (p. 3) describes learned Q-values for actions from separate modules. But the paper studies an artificial agent in a toy grid world, not a bee solving an nl-MOMDP task, and it does not specify the claimed output of action–expected-value-vector pairs for that task. (LLM, conf=0.85)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 9
- [2][2] Mehdi Keramati and Boris Gutkin. 'Homeostatic reinforcement learning for integrating reward collection and physiological stability'. In: Elife 3 (2014), e04811. -
- [3][3] Diederik M Roijers et al. 'A survey of multi-objective sequential decision-making'. In: Journal of Artificial Intelligence Research 48 (2013), pp. 67-113. -
- [5][5] Stuart J Russell and Andrew Zimdars. 'Q-decomposition for reinforcement learning agents'. In: Proceedings of the 20th International Conference on Machine Learning (ICML-03) . 2003, pp. 656-663. -
- [6][6] Harm Van Seijen et al. 'Hybrid reward architecture for reinforcement learning'. In: Advances in Neural Information Processing Systems 30 (2017). -
- [7][7] Volodymyr Mnih et al. 'Playing atari with deep reinforcement learning'. In: arXiv preprint arXiv:1312.5602 (2013). - · arXiv
- [8][8] Robert A Jacobs et al. 'Adaptive mixtures of local experts'. In: Neural computation 3.1 (1991), pp. 79-87. -
- [9][9] Tianpei Yang et al. 'Exploration in deep reinforcement learning: a comprehensive survey'. In: arXiv:2109.06668 (2021). - · arXiv
- [10][10] Samuel M McClure, Mark S Gilzenrat, and Jonathan D Cohen. 'An exploration-exploitation model based on norepinepherine and dopamine activity'. In: Advances in neural information processing systems 18 (2005). -
- [11][11] Anna Freud. The ego and the mechanisms of defence . Routledge, 2018.
Cited by
Sorted by citations
Phenomenal interface theory: a model for basal consciousness
Klein, C., & Barron, A. B. (2025) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2024.0301 · Cited by 8
Metadata
- Authors
- Dulberg, Z., Dubey, R., Berwian, I. M., & Cohen, J. D.
- Affiliations
- Princeton University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2204.06608
- OpenAlex
- W4224002285
- Cited by
- 3
- References
- 0
- Citations by year
- 2025: 1 · 2023: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- John Templeton Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Advanced Multi-Objective Optimization Algorithms
- Keywords
- modular reinforcement learning, multi-objective reinforcement learning, homeostasis, deep Q-network, exploration, sample efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
