Emotion in reinforcement learning agents and robots: a survey
Open accessThe DOI matches its Crossref record; 107 of the 107 references it cites were found in a registry or an index.
Abstract
This article provides the first survey of computational models of emotion in reinforcement learning (RL) agents. The survey focuses on agent/robot emotions, and mostly ignores human user emotions. Emotions are recognized as functional in decision-making by influencing motivation and action selection. Therefore, computational emotion models are usually grounded in the agent’s decision making architecture, of which RL is an important subclass. Studying emotions in RL-based agents is useful for three research fields. For machine learning (ML) researchers, emotion models may improve learning efficiency. For the interactive ML and human–robot interaction community, emotions can communicate state and enhance user investment. Lastly, it allows affective modelling researchers to investigate their emotion theories in a successful AI agent class. This survey provides background on emotion theory and RL. It systematically addresses (1) from what underlying dimensions (e.g. homeostasis, appraisal) emotions can be derived and how these can be modelled in RL-agents, (2) what types of emotions have been derived from these dimensions, and (3) how these emotions may either influence the learning efficiency of the agent or be useful as social signals. We also systematically compare evaluation criteria, and draw connections to important RL sub-domains like (intrinsic) motivation and model-based RL. In short, this survey provides both a practical overview for engineers wanting to implement emotions in their RL agents, and identifies challenges and directions for future emotion-RL research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · cited 5×
“Potentially, the advances in solving more complex AI scenarios with (deep) RL (Silver et al, 2016; Mnih et al, 2015) may provide more realistic test scenarios in the future as well.”
BackgroundInfluentialMastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
The Cognitive Structure of Emotions
Ortony, A., Clore, G. L., & Collins, A. (1988) · doi:10.1017/cbo9780511571299 · Cited by 6,208
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148 · cited 2×
“Well-known algorithms are Q-learning (Watkins, 1989), SARSA (Rummery and Niranjan, 1994) and TD( λ ) (Sutton, 1988).”
MethodA survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227
A survey of socially interactive robots
Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003) · Robotics and Autonomous Systems · doi:10.1016/s0921-8890(02)00372-x · Cited by 3,162
Russell, J. A., & Barrett, L. F. (1999) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.76.5.805 · Cited by 3,102
Appraisal Processes in Emotion: Theory, Methods, Research
(2001) · doi:10.1093/oso/9780195130072.001.0001 · Cited by 2,384
Relations among emotion, appraisal, and emotional action readiness
Frijda, N. H., Kuipers, P., & ter Schure, E. (1989) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.57.2.212 · Cited by 2,379
Cognition and motivation in emotion.
Lazarus, R. S. (1991) · American Psychologist · doi:10.1037/0003-066x.46.4.352 · Cited by 2,096
PILCO: A Model-Based and Data-Efficient Approach to Policy Search
Deisenroth, M. P., & Rasmussen, C. E. (2011) · Cambridge University Engineering Department Publications Database · Cited by 1,945
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893
The Emotional Brain, Fear, and the Amygdala
LeDoux, J. (2003) · Cellular and Molecular Neurobiology · doi:10.1023/a:1025048802629 · Cited by 1,818
Universals and cultural differences in the judgments of facial expressions of emotion.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.53.4.712 · Cited by 1,763
Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping
Ng, A. Y., Harada, D., & Russell, S. (1999) · Cited by 1,744
How Emotion Shapes Behavior: Feedback, Anticipation, and Reflection, Rather Than Direct Causation
Baumeister, R. F., Vohs, K. D., Nathan DeWall, C., & Zhang, L. (2007) · Personality and Social Psychology Review · doi:10.1177/1088868307301033 · Cited by 1,743
Principles of Behavior. An Introduction to Behavior Theory
N., E., & Hull, C. L. (1943) · The Journal of Philosophy · doi:10.2307/2019960 · Cited by 1,739
Emotion and sociable humanoid robots
Breazeal, C. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00018-1 · Cited by 1,279
Intrinsic Motivation Systems for Autonomous Mental Development
Oudeyer, P.-Y., Kaplan, F., & Hafner, V. V. (2007) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2006.890271 · Cited by 1,126
The somatic marker hypothesis: A critical evaluation
Dunn, B. D., Dalgleish, T., & Lawrence, A. D. (2006) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2005.07.001 · Cited by 894
The orbitofrontal cortex: Neuronal activity in the behaving monkey
Thorpe, S. J., Rolls, E. T., & Maddison, S. (1983) · Experimental Brain Research · doi:10.1007/bf00235545 · Cited by 836 · cited 1×
“Earlier work already identified ’error neurons’, which mainly respond when an expected reward is not received (Thorpe et al, 1983).”
BackgroundWhat is intrinsic motivation? A typology of computational approaches
Oudeyer, P.-Y. (2007) · Frontiers in Neurorobotics · doi:10.3389/neuro.12.006.2007 · Cited by 811
The orbitofrontal cortex and beyond: From affect to decision-making
Rolls, E. T., & Grabenhorst, F. (2008) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2008.09.001 · Cited by 807
Metalearning and neuromodulation
Doya, K. (2002) · Neural Networks · doi:10.1016/s0893-6080(02)00044-8 · Cited by 704
Reinforcement Learning: State-of-the-Art
(2012) · Adaptation, Learning, and Optimization · doi:10.1007/978-3-642-27645-3 · Cited by 691
Intrinsically Motivated Reinforcement Learning
Singh, S., Barto, A. G., & Chentanez, N. (2005) · doi:10.21236/ada440280 · Cited by 610
Scherer, K. R. (1999) · Handbook of Cognition and Emotion · doi:10.1002/0470013494.ch30 · Cited by 603
Gustatory, olfactory, and visual convergence within the primate orbitofrontal cortex
Rolls, E., & Baylis, L. (1994) · The Journal of Neuroscience · doi:10.1523/jneurosci.14-09-05437.1994 · Cited by 587
Evidence of convergent validity on the dimensions of affect
Russell, J. A. (1978) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.36.10.1152 · Cited by 495
Basic emotions, rationality, and folk theory
Johnson-laird, P. N., & Oatley, K. (1992) · Cognition and Emotion · doi:10.1080/02699939208411069 · Cited by 488
The Oxford Handbook of Affective Computing
(2015) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199942237.001.0001 · Cited by 482
VIME: Variational Information Maximizing Exploration
Houthooft, R., Chen, X., Duan, Y., Schulman, J., Turck, F. D., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.09674 · Cited by 472
EMA: A process model of appraisal dynamics
Marsella, S. C., & Gratch, J. (2009) · Cognitive Systems Research · doi:10.1016/j.cogsys.2008.03.005 · Cited by 452 · cited 1×
“Several well-known symbolic architectures have also incorporated emotions, either based on categorical emotions (Murphy et al, 2002), somatic marker hypothesis (Laird, 2008), or appraisal theories (EMIB (Michaud, 2002), EMA (Marsella and Gratch, 2009) and LIDA (Franklin et al, 2014)).”
BackgroundFLAME—Fuzzy Logic Adaptive Model of Emotions
El-Nasr, M. S., Yen, J., & Ioerger, T. R. (2000) · Autonomous Agents and Multi-Agent Systems · doi:10.1023/a:1010030809960 · Cited by 437
Action-Conditional Video Prediction using Deep Networks in Atari Games
Oh, J., Guo, X., Lee, H., Lewis, R. L., & Singh, S. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.08750 · Cited by 419
Intrinsically Motivated Reinforcement Learning: An Evolutionary Perspective
Singh, S., Lewis, R. L., Barto, A. G., & Sorg, J. (2010) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2010.2051031 · Cited by 417
Teachable robots: Understanding human teaching behavior to build more effective robot learners
Thomaz, A. L., & Breazeal, C. (2008) · Artificial Intelligence · doi:10.1016/j.artint.2007.09.009 · Cited by 405
Bridging the Gap between Social Animal and Unsocial Machine: A Survey of Social Signal Processing
Vinciarelli, A., Pantic, M., Heylen, D., Pelachaud, C., Poggi, I., D'Errico, F., & Schroeder, M. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.27 · Cited by 381
Modeling motivations and emotions as a basis for intelligent behavior
Cañamero, D. (1997) · Proceedings of the first international conference on Autonomous agents - AGENTS '97 · doi:10.1145/267658.267688 · Cited by 271
Meta-learning in Reinforcement Learning
Schweighofer, N., & Doya, K. (2003) · Neural Networks · doi:10.1016/s0893-6080(02)00228-9 · Cited by 261 · cited 4×
“(2007) Gridworld agent Maze Learning Emotional agent cannot improve results of (Broekens et al, 2007b; Schweighofer and Doya, 2003) Shi et al.”
BackgroundInfluentialLIDA: A Systems-level Architecture for Cognition, Emotion, and Learning
Franklin, S., Madl, T., D'Mello, S., & Snaider, J. (2014) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2013.2277589 · Cited by 261
A Computational Model of Emotional Learning in the Amygdala
Morén, J., & Balkenius, C. (2000) · From Animals to Animats 6 · doi:10.7551/mitpress/3120.003.0041 · Cited by 255
Reinforcement Learning in Robotics: A Survey
Kober, J., & Peters, J. (2012) · Adaptation, learning, and optimization · doi:10.1007/978-3-642-27645-3_18 · Cited by 228
Training a Robot via Human Feedback: A Case Study
Knox, W. B., Stone, P., & Breazeal, C. (2013) · Lecture notes in computer science · doi:10.1007/978-3-319-02675-6_46 · Cited by 130
Emotional Experience in the Computational Belief–Desire Theory of Emotion
Reisenzein, R. (2009) · Emotion Review · doi:10.1177/1754073909103589 · Cited by 109
Parisi, D., & Petrosino, G. (2010) · Adaptive Behavior · doi:10.1177/1059712310388528 · Cited by 105
Robot Learning Driven by Emotions
Gadanho, S. C., & Hallam, J. C. T. (2001) · Adaptive Behavior · doi:10.1177/105971230200900102 · Cited by 103
Emotion-based control of cooperating heterogeneous mobile robots
Murphy, R. R., Lisetti, C., Tardif, R., Irish, L., & Gage, A. (2002) · IEEE Transactions on Robotics and Automation · doi:10.1109/tra.2002.804503 · Cited by 101
How Humans Teach Agents: A New Experimental Perspective
Knox, W. B., Glass, B. D., Love, B. C., Maddox, W. T., & Stone, P. (2012) · International Journal of Social Robotics · doi:10.1007/s12369-012-0163-x · Cited by 99
Hedonic value: enhancing adaptation for motivated agents
Cos, I., Cañamero, L. D., Hayes, G. M., & Gillies, A. (2013) · Adaptive Behavior · doi:10.1177/1059712313486817 · Cited by 94
Artificial emotional creature for human-machine interaction
Shibata, T., Yoshida, M., & Yamato, J. (2002) · 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation · doi:10.1109/icsmc.1997.635205 · Cited by 92
Emotion and Reinforcement: Affective Facial Expressions Facilitate Robot Learning
Broekens, J. (2007) · Springer eBooks · doi:10.1007/978-3-540-72348-6_6 · Cited by 82
Self-citationLearning Behavior-Selection by Emotions and Cognition in a Multi-Goal Robot Task
Gadanho, S. C., & Dayan, P. (2003) · Cited by 78
Emotional Multiagent Reinforcement Learning in Spatial Social Dilemmas
Yu, C., Zhang, M., Ren, F., & Tan, G. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2403394 · Cited by 77
Promoting Interactions Between Humans and Robots Using Robotic Emotional Behavior
Ficocelli, M., Terao, J., & Nejat, G. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2015.2492999 · Cited by 77
An Adaptive Robot Motivational System
Konidaris, G., & Barto, A. G. (2006) · Lecture notes in computer science · doi:10.1007/11840541_29 · Cited by 75
A Computational Model of Emotional Conditioning in the Brain
Balkenius, C., & Morén, J. (1998) · Cited by 73
A New Approach to Modeling Emotions and Their Use on a Decision-Making System for Artificial Agents
Salichs, M. A., & Malfáz, M. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.32 · Cited by 72
Emotion-Based Intrinsic Motivation for Reinforcement Learning Agents
Sequeira, P., Melo, F. S., & Paiva, A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24600-5_36 · Cited by 66
Modeling appraisal in theory of mind reasoning
Si, M., Marsella, S., & Pynadath, D. V. (2009) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-009-9093-x · Cited by 65
Broekens, J., Kosters, W. A., & Verbeek, F. J. (2007) · Adaptive Behavior · doi:10.1177/1059712307084686 · Cited by 55
Self-citationDesigning Emotions for Activity Selection in Autonomous Agents
Cañamero, L. D. (2003) · The MIT Press eBooks · doi:10.7551/mitpress/2705.003.0005 · Cited by 55
Mobile Robot Navigation Modulated by Artificial Emotions
Lee-Johnson, C. P., & Carnegie, D. A. (2009) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2009.2026826 · Cited by 54
Emotions in autonomous agents: comparative analysis of mechanisms and functions
Rumbell, T. H., Barnden, J. A., Denham, S. L., & Wennekers, T. (2011) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-011-9166-5 · Cited by 47
Kim, H.-R., & Kwon, D. (2010) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-010-9418-7 · Cited by 47
Bayesian Affect Control Theory
Hoey, J., Schröder, T., & Alhothali, A. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.34 · Cited by 44
Challenges in Computational Modeling of Affective Processes
Broekens, J., Bosse, T., & Marsella, S. (2013) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2013.23 · Cited by 43
Self-citationHester, T., & Stone, P. (2012) · Adaptation, learning, and optimization · doi:10.1007/978-3-642-27645-3_4 · Cited by 42
Learning by appraising: an emotion-based approach to intrinsic reward design
Sequeira, P., Melo, F. S., & Paiva, A. (2014) · Adaptive Behavior · doi:10.1177/1059712314543837 · Cited by 40
A Reinforcement Learning Theory for Homeostatic Regulation
Keramati, M., & Gutkin, B. S. (2011) · City Research Online (City University London) · Cited by 39
Intrinsically motivated model learning for a developing curious agent
Hester, T., & Stone, P. (2012) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2012.6400802 · Cited by 39
Learning Multimodal Transition Dynamics for Model-Based Reinforcement LearningPreprint
Moerland, T. M., Broekens, J., & Jonker, C. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.00470 · Cited by 38
Self-citationEmotionally motivated reinforcement learning based controller
Ayesh, A. (2005) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2004.1398413 · Cited by 31
Hasson, C., Gaussier, P., & Boucenna, S. (2011) · Paladyn Journal of Behavioral Robotics · doi:10.2478/s13230-012-0005-4 · Cited by 30
An Autonomous Social Robot in Fear
Castro‐González, Á., Malfáz, M., & Salichs, M. A. (2013) · IEEE Transactions on Autonomous Mental Development · doi:10.1109/tamd.2012.2234120 · Cited by 29
Strategies for Affect-Controlled Action-Selection in Soar-RL
Hogewoning, E., Broekens, J., Eggermont, J., & Bovenkamp, E. G. P. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73055-2_52 · Cited by 27
Self-citationOn Affect and Self-adaptation: Potential Benefits of Valence-Controlled Action-Selection
Broekens, J., Kosters, W. A., & Verbeek, F. J. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-73053-8_36 · Cited by 26
Self-citationBayesian Affect Control Theory of Self
Hoey, J., & Schroeder, T. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9222 · Cited by 26
Associated Emotion and Its Expression in an Entertainment Robot QRIO
Tanaka, F., Noda, K., Sawada, T., et al. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-28643-1_64 · Cited by 23
Emergent Dynamics of Joy, Distress, Hope and Fear in Reinforcement Learning Agents
Jacobs, E., Broekens, J., & Jonker, C. M. (2014) · Cited by 22
Self-citationTowards a Model for Embodied Emotions
Coutinho, E., Miranda, E. R., & Cangelosi, A. (2005) · Portuguese Conference on Artificial Intelligence · doi:10.1109/epia.2005.341264 · Cited by 21
Toward socially responsible agents: integrating attachment and learning in emotional decision‐making
Moussa, M. B., & Magnenat‐Thalmann, N. (2013) · Computer Animation and Virtual Worlds · doi:10.1002/cav.1515 · Cited by 20
Emotion inspired adaptive robotic path planning
Williams, H. A., Lee-Johnson, C. P., Browne, W. N., & Carnegie, D. A. (2015) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec.2015.7257263 · Cited by 20
Strong mitigation: nesting search for good policies within search for good reward
Bratman, J., Singh, S. P., Sorg, J., & Lewis, R. L. (2012) · doi:10.5555/2343576.2343634 · Cited by 19
From Imprinting to Adaptation : Building a History of Affective Interaction
Blanchard, A., & Cañamero, L. D. (2005) · CogPrints (University of Southampton) · Cited by 17
Fear and hope emerge from anticipation in model-based reinforcement learning
Moerland, T. M., Broekens, J., & Jonker, C. M. (2016) · International Joint Conference on Artificial Intelligence · Cited by 15
Self-citationZhou, W., & Coggins, R. (2002) · Lecture notes in computer science · doi:10.1007/3-540-36187-1_37 · Cited by 14
Emotional Multiagent Reinforcement Learning in Social Dilemmas
Yu, C., Zhang, M., & Ren, F. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-44927-7_25 · Cited by 14
Emotion, Embodiment, and Consequence Driven Systems
Bozinovski, S., Stojanov, G., & Božinovska, L. (1996) · Cited by 13
Decision making based on reinforcement learning and emotion learning for social behavior
Matsuda, A., Misawa, H., & Horio, K. (2011) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzzy.2011.6007506 · Cited by 13
Emotion-based parameter modulation for a hierarchical mobile robot planning and control architecture
Lee-Johnson, C. P., & Carnegie, D. A. (2007) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2007.4399080 · Cited by 11
Teachable Characters: User Studies, Design Principles, and Learning Performance
Thomaz, A. L., & Breazeal, C. (2006) · Lecture notes in computer science · doi:10.1007/11821830_32 · Cited by 11
Behavior decision model of intelligent agent based on artificial emotion
Guojiang, W., Xiaoxiao, W., & Fu, K. (2010) · International Conference on Advanced Computer Control · doi:10.1109/icacc.2010.5487217 · Cited by 10
Emotions and Personality in Agent Design and Modeling
Gmytrasiewicz, P. J., & Lisetti, C. (2002) · Multiagent systems, artificial societies, and simulated organizations · doi:10.1007/978-1-4615-1107-6_5 · Cited by 10
EMOBOT: A Robot Control Architecture Based on Emotion-Like Internal Values
Goerke, N. (2006) · doi:10.5772/4715 · Cited by 9
An Improved Reinforcement Learning System Using Affective Factors
Kuremoto, T., Tsurusaki, T., Kobayashi, K., Mabu, S., & Obayashi, M. (2013) · Robotics · doi:10.3390/robotics2030149 · Cited by 9
An emotional model embedded reinforcement learning system
Obayashi, M., Takuno, T., Kuremoto, T., & Kobayashi, K. (2012) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2012.6377870 · Cited by 9
Design of autonomous navigation system based on affective cognitive learning and decision-making
Zhang, H., & Liu, S. (2009) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio.2009.5420477 · Cited by 8
Emotionally influenced coordination of behaviors for autonomous mobile robots
Tsankova, D. (2003) · doi:10.1109/is.2002.1044234 · Cited by 8
Modulating reinforcement-learning parameters using agent emotions
von Haugwitz, R., Kitamura, Y., & Takashima, K. (2012) · Joint International Conference on Soft Computing and Intelligent Systems and International Symposium on Advanced Intelligent Systems (SCIS&ISIS) · doi:10.1109/scis-isis.2012.6505340 · Cited by 8
Emotional Model Based on Computational Intelligence for Partner Robots
Kubota, N., & Wakisaka, S. (2010) · Smart innovation, systems and technologies · doi:10.1007/978-3-642-12604-8_5 · Cited by 6
Goal-oriented action planning in partially observable stochastic domains
Huang, X., Du, C., Peng, Y. P., et al. (2012) · IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS) · doi:10.1109/ccis.2012.6664612 · Cited by 6
Doshi, P., & Gmytrasiewicz, P. J. (2004) · National Conference on Artificial Intelligence · Cited by 5
Artificial Emotion Model Based on Neuromodulators and Q-learning
Shi, X., Wang, Z., & Zhang, Q. (2012) · Lecture notes in electrical engineering · doi:10.1007/978-3-642-31006-5_35 · Cited by 5
Emotion-triggered Learning for Autonomous Robots
Gadanho, S., Hallam, J., & Intelligence, E. U. (. K. D. O. A. (1998) · Cited by 3
Research works printed but not linked: 22
- Antos, D., & Pfeffer, A. (2011). Using emotions to enhance decision-making. In Proceedings of the international joint conference on artificial intelligence (IJCAI) (Vol. 22, p. 24).
- Belavkin, R. V. (2004). On relation between emotion and entropy. In Proceedings of the AISB’04 symposium on emotion, cognition and affective computing (pp. 1–8). AISB Press.
- Bozinovski, S. (1982). A self-learning system using secondary reinforcement. In E. Trappl (Ed.), Cybernetics and Systems Research (pp. 397–402). North-Holland, Amsterdam.
- Damasio, A. R. (1994). Descartes’ error: Emotion, reason and the human brain. New York: Grosset/Putnam.
- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Cambridge: MIT Press.
- Gratch, J., & Marsella, S. (2014). Appraisal models. Oxford: Oxford University Press.
- Joffily, M., & Coricelli, G. (2013). Emotional valence and the free-energy principle. PLOS Computational Biology, 9(6), e1003094. doi: 10.1371/journal.pcbi.10030946 . · doi:10.1371/journal.pcbi.10030946
- Lahnstein, M. (2005). The emotive episode is a composition of anticipatory and reactive evaluations. In Proceedings of the AISB’05 symposium on agents that want and like (pp. 62–69).
- Laird, J. E. (2008). Extending the Soar cognitive architecture. Frontiers in Artificial Intelligence and Applications, 171, 224.
- Lhommet, M., & Marsella, S. C. (2015). Expressing emotion through posture and gesture (pp. 273–285). Oxford: Oxford University Press.
- Lisetti, C., & Hudlicka, E. (2015). Why and how to build emotion-based agent architectures, in The Oxford Handbook of Affective Computing, Chap. 8 (Oxford Library of Psychology, 2015) p. 94.
- Marinier, R., & Laird, J. E. (2008). Emotion-driven reinforcement learning. In Cognitive science (pp. 115–120).
- Marsella, S., Gratch, J., & Petta, P. (2010). Computational models of emotion. In K. Scherer, T. Bänziger, & E. Roesch (Eds.), A blueprint for affective computing (pp. 21–45). Oxford: Oxford University Press.
- Michaud, F. (2002). EMIB–Computational architecture based on emotion and motivation for in-tentional selection and configuration of behaviour-producing modules. Cognitive Science Quarterly, 3–4, 340–361.
- Ochs, M., Niewiadomski, R., & Pelachaud, C. (2015). Facial expressions of emotions for virtual characters (pp. 261–272). Oxford: Oxford University Press.
- Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1964). The measurement of meaning. Champaign: University of Illinois Press.
- Paiva, A., Leite, I., & Ribeiro, T. (2015). Emotion modeling for social robots (pp. 296–308). Oxford: Oxford University Press.
- Rummery, G. A., & Niranjan, M. (1994). On-line Q-learning using connectionist systems. Technical Report, University of Cambridge, Department of Engineering.
- Russell, S., Norvig, P., & Intelligence, A. (1995). A modern approach. Artificial Intelligence, 25, 27.
- Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge: MIT Press.
- Velasquez, J. (1998). Modeling emotion-based decision-making. In Emotional and intelligent: The tangled knot of cognition (pp. 164–169).
- Watkins, C. J. C. H. (1989). Learning from delayed rewards. Ph.D. Thesis, University of Cambridge England.
Cited by
Multi-agent deep reinforcement learning: a survey
Gronauer, S., & Diepold, K. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-09996-w · Cited by 957
Q-Learning Algorithms: A Comprehensive Classification and Applications
Jang, B., Kim, M., Harerimana, G., & Kim, J. W. (2019) · IEEE Access · doi:10.1109/access.2019.2941229 · Cited by 635
A Survey of Robot Learning Strategies for Human-Robot Collaboration in Industrial Settings
Mukherjee, D., Gupta, K., Chang, L. H., & Najjaran, H. (2022) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2021.102231 · Cited by 344
A Taxonomy of Social Errors in Human-Robot Interaction
Tian, L., & Oviatt, S. (2021) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3439720 · Cited by 133
An Overview of Emotion in Artificial Intelligence
Assunção, G., Patrão, B., Castelo-Branco, M., & Menezes, P. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3159614 · Cited by 103
Metadata
- Authors
- Moerland, T. M., Broekens, J., & Jonker, C. M.
- Affiliations
- Delft University of Technology (NL)
- Corresponding author
- Moerland, Thomas M.
- Venue
- Machine Learning
- Published
- 2017-08-25
- Volume
- 107
- Issue
- 2
- Pages
- 443-480
- ISSN
- 0885-6125 (ISSN-L)
- 1573-0565
- OpenAlex
- W2614309179
- Cited by
- 205
- References
- 107
- Citations by year
- 2026: 17 · 2025: 14 · 2024: 21 · 2023: 14 · 2022: 40 · 2021: 32 · 2020: 23 · 2019: 29 · 2018: 13 · 2017: 2
- FWCI
- 10.78
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Machine Learning (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Research Repository (Delft University of Technology) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Delft University of Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Keywords
- computational models of emotion, reinforcement learning agents, robot emotions, affective computing, emotion theory, homeostasis, appraisal theory, intrinsic motivation, model-based reinforcement learning, learning efficiency, human-robot interaction, social signals, evaluation criteria
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.