Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.

Emotion in reinforcement learning agents and robots: a survey

Open access

Abstract

This article provides the first survey of computational models of emotion in reinforcement learning (RL) agents. The survey focuses on agent/robot emotions, and mostly ignores human user emotions. Emotions are recognized as functional in decision-making by influencing motivation and action selection. Therefore, computational emotion models are usually grounded in the agent’s decision making architecture, of which RL is an important subclass. Studying emotions in RL-based agents is useful for three research fields. For machine learning (ML) researchers, emotion models may improve learning efficiency. For the interactive ML and human–robot interaction community, emotions can communicate state and enhance user investment. Lastly, it allows affective modelling researchers to investigate their emotion theories in a successful AI agent class. This survey provides background on emotion theory and RL. It systematically addresses (1) from what underlying dimensions (e.g. homeostasis, appraisal) emotions can be derived and how these can be modelled in RL-agents, (2) what types of emotions have been derived from these dimensions, and (3) how these emotions may either influence the learning efficiency of the agent or be useful as social signals. We also systematically compare evaluation criteria, and draw connections to important RL sub-domains like (intrinsic) motivation and model-based RL. In short, this survey provides both a practical overview for engineers wanting to implement emotions in their RL agents, and identifies challenges and directions for future emotion-RL research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10994-017-5666-0

Cites

107 of 107 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 107 references

Research works printed but not linked: 22

  1. Antos, D., & Pfeffer, A. (2011). Using emotions to enhance decision-making. In Proceedings of the international joint conference on artificial intelligence (IJCAI) (Vol. 22, p. 24).
  2. Belavkin, R. V. (2004). On relation between emotion and entropy. In Proceedings of the AISB’04 symposium on emotion, cognition and affective computing (pp. 1–8). AISB Press.
  3. Bozinovski, S. (1982). A self-learning system using secondary reinforcement. In E. Trappl (Ed.), Cybernetics and Systems Research (pp. 397–402). North-Holland, Amsterdam.
  4. Damasio, A. R. (1994). Descartes’ error: Emotion, reason and the human brain. New York: Grosset/Putnam.
  5. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Cambridge: MIT Press.
  6. Gratch, J., & Marsella, S. (2014). Appraisal models. Oxford: Oxford University Press.
  7. Joffily, M., & Coricelli, G. (2013). Emotional valence and the free-energy principle. PLOS Computational Biology, 9(6), e1003094. doi: 10.1371/journal.pcbi.10030946 . · doi:10.1371/journal.pcbi.10030946
  8. Lahnstein, M. (2005). The emotive episode is a composition of anticipatory and reactive evaluations. In Proceedings of the AISB’05 symposium on agents that want and like (pp. 62–69).
  9. Laird, J. E. (2008). Extending the Soar cognitive architecture. Frontiers in Artificial Intelligence and Applications, 171, 224.
  10. Lhommet, M., & Marsella, S. C. (2015). Expressing emotion through posture and gesture (pp. 273–285). Oxford: Oxford University Press.
  11. Lisetti, C., & Hudlicka, E. (2015). Why and how to build emotion-based agent architectures, in The Oxford Handbook of Affective Computing, Chap. 8 (Oxford Library of Psychology, 2015) p. 94.
  12. Marinier, R., & Laird, J. E. (2008). Emotion-driven reinforcement learning. In Cognitive science (pp. 115–120).
  13. Marsella, S., Gratch, J., & Petta, P. (2010). Computational models of emotion. In K. Scherer, T. Bänziger, & E. Roesch (Eds.), A blueprint for affective computing (pp. 21–45). Oxford: Oxford University Press.
  14. Michaud, F. (2002). EMIB–Computational architecture based on emotion and motivation for in-tentional selection and configuration of behaviour-producing modules. Cognitive Science Quarterly, 3–4, 340–361.
  15. Ochs, M., Niewiadomski, R., & Pelachaud, C. (2015). Facial expressions of emotions for virtual characters (pp. 261–272). Oxford: Oxford University Press.
  16. Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1964). The measurement of meaning. Champaign: University of Illinois Press.
  17. Paiva, A., Leite, I., & Ribeiro, T. (2015). Emotion modeling for social robots (pp. 296–308). Oxford: Oxford University Press.
  18. Rummery, G. A., & Niranjan, M. (1994). On-line Q-learning using connectionist systems. Technical Report, University of Cambridge, Department of Engineering.
  19. Russell, S., Norvig, P., & Intelligence, A. (1995). A modern approach. Artificial Intelligence, 25, 27.
  20. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge: MIT Press.
  21. Velasquez, J. (1998). Modeling emotion-based decision-making. In Emotional and intelligent: The tangled knot of cognition (pp. 164–169).
  22. Watkins, C. J. C. H. (1989). Learning from delayed rewards. Ph.D. Thesis, University of Cambridge England.

Cited by

See all 205

Metadata

Authors
Moerland, T. M., Broekens, J., & Jonker, C. M.
Affiliations
Delft University of Technology (NL)
Corresponding author
Moerland, Thomas M.
Venue
Machine Learning
Published
2017-08-25
Volume
107
Issue
2
Pages
443-480
ISSN
  • 0885-6125 (ISSN-L)
  • 1573-0565
OpenAlex
W2614309179
Cited by
205
References
107
Citations by year
2026: 17 · 2025: 14 · 2024: 21 · 2023: 14 · 2022: 40 · 2021: 32 · 2020: 23 · 2019: 29 · 2018: 13 · 2017: 2
FWCI
10.78
Type
journal article
Language
en
Open access
Locations
  • Machine Learning (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Research Repository (Delft University of Technology) (submittedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
Delft University of Technology
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
  • Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
Keywords
computational models of emotion, reinforcement learning agents, robot emotions, affective computing, emotion theory, homeostasis, appraisal theory, intrinsic motivation, model-based reinforcement learning, learning efficiency, human-robot interaction, social signals, evaluation criteria

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Emotion in reinforcement learning agents and robots: a survey. https://app.cochat.ai/papers/10.1007/s10994-017-5666-0
The paper
Moerland, T. M., Broekens, J., & Jonker, C. M. (2018). Emotion in reinforcement learning agents and robots: a survey. Machine Learning. https://doi.org/10.1007/s10994-017-5666-0

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.