The DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.
Emotion in reinforcement learning agents and robots: a survey
Open accessAbstract
This article provides the first survey of computational models of emotion in reinforcement learning (RL) agents. The survey focuses on agent/robot emotions, and mostly ignores human user emotions. Emotions are recognized as functional in decision-making by influencing motivation and action selection. Therefore, computational emotion models are usually grounded in the agent’s decision making architecture, of which RL is an important subclass. Studying emotions in RL-based agents is useful for three research fields. For machine learning (ML) researchers, emotion models may improve learning efficiency. For the interactive ML and human–robot interaction community, emotions can communicate state and enhance user investment. Lastly, it allows affective modelling researchers to investigate their emotion theories in a successful AI agent class. This survey provides background on emotion theory and RL. It systematically addresses (1) from what underlying dimensions (e.g. homeostasis, appraisal) emotions can be derived and how these can be modelled in RL-agents, (2) what types of emotions have been derived from these dimensions, and (3) how these emotions may either influence the learning efficiency of the agent or be useful as social signals. We also systematically compare evaluation criteria, and draw connections to important RL sub-domains like (intrinsic) motivation and model-based RL. In short, this survey provides both a practical overview for engineers wanting to implement emotions in their RL agents, and identifies challenges and directions for future emotion-RL research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
The Cognitive Structure of Emotions
Ortony, A., Clore, G. L., & Collins, A. (1988) · doi:10.1017/cbo9780511571299 · Cited by 6,208
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227
A survey of socially interactive robots
Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(02)00372-x · Cited by 3,162
Russell, J. A., & Barrett, L. F. (1999) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.76.5.805 · Cited by 3,102
Appraisal Processes in Emotion: Theory, Methods, Research
(2001) · doi:10.1093/oso/9780195130072.001.0001 · Cited by 2,384
Relations among emotion, appraisal, and emotional action readiness
Frijda, N. H., Kuipers, P., & ter Schure, E. (1989) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.57.2.212 · Cited by 2,379
Research works printed but not linked: 22
- Antos, D., & Pfeffer, A. (2011). Using emotions to enhance decision-making. In Proceedings of the international joint conference on artificial intelligence (IJCAI) (Vol. 22, p. 24).
- Belavkin, R. V. (2004). On relation between emotion and entropy. In Proceedings of the AISB’04 symposium on emotion, cognition and affective computing (pp. 1–8). AISB Press.
- Bozinovski, S. (1982). A self-learning system using secondary reinforcement. In E. Trappl (Ed.), Cybernetics and Systems Research (pp. 397–402). North-Holland, Amsterdam.
- Damasio, A. R. (1994). Descartes’ error: Emotion, reason and the human brain. New York: Grosset/Putnam.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Cambridge: MIT Press.
- Gratch, J., & Marsella, S. (2014). Appraisal models. Oxford: Oxford University Press.
- Joffily, M., & Coricelli, G. (2013). Emotional valence and the free-energy principle. PLOS Computational Biology, 9(6), e1003094. doi: 10.1371/journal.pcbi.10030946 . · doi:10.1371/journal.pcbi.10030946
- Lahnstein, M. (2005). The emotive episode is a composition of anticipatory and reactive evaluations. In Proceedings of the AISB’05 symposium on agents that want and like (pp. 62–69).
- Laird, J. E. (2008). Extending the Soar cognitive architecture. Frontiers in Artificial Intelligence and Applications, 171, 224.
- Lhommet, M., & Marsella, S. C. (2015). Expressing emotion through posture and gesture (pp. 273–285). Oxford: Oxford University Press.
- Lisetti, C., & Hudlicka, E. (2015). Why and how to build emotion-based agent architectures, in The Oxford Handbook of Affective Computing, Chap. 8 (Oxford Library of Psychology, 2015) p. 94.
- Marinier, R., & Laird, J. E. (2008). Emotion-driven reinforcement learning. In Cognitive science (pp. 115–120).
- Marsella, S., Gratch, J., & Petta, P. (2010). Computational models of emotion. In K. Scherer, T. Bänziger, & E. Roesch (Eds.), A blueprint for affective computing (pp. 21–45). Oxford: Oxford University Press.
- Michaud, F. (2002). EMIB–Computational architecture based on emotion and motivation for in-tentional selection and configuration of behaviour-producing modules. Cognitive Science Quarterly, 3–4, 340–361.
- Ochs, M., Niewiadomski, R., & Pelachaud, C. (2015). Facial expressions of emotions for virtual characters (pp. 261–272). Oxford: Oxford University Press.
- Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1964). The measurement of meaning. Champaign: University of Illinois Press.
- Paiva, A., Leite, I., & Ribeiro, T. (2015). Emotion modeling for social robots (pp. 296–308). Oxford: Oxford University Press.
- Rummery, G. A., & Niranjan, M. (1994). On-line Q-learning using connectionist systems. Technical Report, University of Cambridge, Department of Engineering.
- Russell, S., Norvig, P., & Intelligence, A. (1995). A modern approach. Artificial Intelligence, 25, 27.
- Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge: MIT Press.
- Velasquez, J. (1998). Modeling emotion-based decision-making. In Emotional and intelligent: The tangled knot of cognition (pp. 164–169).
- Watkins, C. J. C. H. (1989). Learning from delayed rewards. Ph.D. Thesis, University of Cambridge England.
Cited by
Multi-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957
A Survey of Robot Learning Strategies for Human-Robot Collaboration in Industrial Settings
Mukherjee, D., Gupta, K., Chang, L. H., & Najjaran, H. (2022) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2021.102231 · Cited by 344
A Taxonomy of Social Errors in Human-Robot Interaction
Tian, L., & Oviatt, S. (2021) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3439720 · Cited by 133
An Overview of Emotion in Artificial Intelligence
Assunção, G., Patrão, B., Castelo-Branco, M., & Menezes, P. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3159614 · Cited by 103
Metadata
- Authors
- Moerland, T. M., Broekens, J., & Jonker, C. M.
- Affiliations
- Delft University of Technology (NL)
- Corresponding author
- Moerland, Thomas M.
- Venue
- Machine Learning
- Published
- 2017-08-25
- Volume
- 107
- Issue
- 2
- Pages
- 443-480
- ISSN
- 0885-6125 (ISSN-L)
- 1573-0565
- OpenAlex
- W2614309179
- Cited by
- 205
- References
- 107
- Citations by year
- 2026: 17 · 2025: 14 · 2024: 21 · 2023: 14 · 2022: 40 · 2021: 32 · 2020: 23 · 2019: 29 · 2018: 13 · 2017: 2
- FWCI
- 10.78
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Machine Learning (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Research Repository (Delft University of Technology) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Delft University of Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Keywords
- computational models of emotion, reinforcement learning agents, robot emotions, affective computing, emotion theory, homeostasis, appraisal theory, intrinsic motivation, model-based reinforcement learning, learning efficiency, human-robot interaction, social signals, evaluation criteria
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Emotion in reinforcement learning agents and robots: a survey. https://app.cochat.ai/papers/10.1007/s10994-017-5666-0
- The paper
- Moerland, T. M., Broekens, J., & Jonker, C. M. (2018). Emotion in reinforcement learning agents and robots: a survey. Machine Learning. https://doi.org/10.1007/s10994-017-5666-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.