Towards morally sensitive action selection for autonomous social robots
Abstract
Autonomous social robots embedded in human societies have to be sensitive to human social interactions and thus to moral norms and principles guiding these interactions. Actions that violate norms can lead to the violator being blamed. Robots thus need to be able to anticipate possible norm violations and attempt to prevent them while they execute actions. If norm violations cannot be prevented (e.g., in a moral dilemma situation in which every action leads to a norm violation), then the robot needs to be able to justify the action to address any potential blame. In this paper, we present a first attempt at an action execution system for social robots that can (a) detect (some) norm violations, (b) consult an ethical reasoner for guidance on what to do in moral dilemma situations, and (c) it can keep track of execution traces and any resulting states that might have violated norms in order to produce justifications.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Robot speech may lead users to attribute robots with intelligence, social agency, moral agency, and a uniquely potent combination of these factors.
“Because robot speech may lead users to attribute robots with intelligence [6, 14], social agency [28], moral agency [16, 52, 57], and the uniquely potent combination of these factors [27]”
From Robots for Social Justice (R4SJ): Toward a More Equitable Practice of Human-Robot Interaction (2024) · cites this paper as [52]
Cites
Clark, H. H. (1996) · doi:10.1017/cbo9780511620539 · Cited by 5,123 · see citing works
Using language: Communicative acts
Clark, H. H. (1996) · Cited by 2,709 · see citing works
Using language: Index of names
Clark, H. H. (1996) · Cited by 2,708 · see citing works
Breazeal, C. (2002) · The MIT Press eBooks · doi:10.7551/mitpress/2376.001.0001 · Cited by 1,595 · see citing works
Andrew, A. M. (2003) · Kybernetes · doi:10.1108/k.2003.06732iae.003 · Cited by 405 · see citing works
Sacrifice One For the Good of Many?: People Apply Different Moral Norms to Human and Robot Agents
Malle, B. F., Scheutz, M., Arnold, T., Voiklis, J., & Cusimano, C. (2015) · Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2696454.2696458 · Cited by 246 · see citing works
Self-citation“My Roomba Is Rambo”: Intimate Home Appliances
Sung, J.-Y., Guo, L., Grinter, R. E., & Christensen, H. I. (2007) · Springer eBooks · doi:10.1007/978-3-540-74853-3_9 · Cited by 219 · see citing works
Do people hold a humanoid robot morally accountable for the harm it causes?
Kahn, P. H., Kanda, T., Ishiguro, H., Gill, B. T., Ruckert, J. H., Shen, S., Gary, H. E., Reichert, A. L., Freier, N. G., & Severson, R. L. (2012) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/2157689.2157696 · Cited by 169 · see citing works
My roomba is rambo: Intimate home appliances
Sung, J.-Y., Guo, L., Grinter, R. E., & Christensen, H. I. (2007) · Cited by 163 · see citing works
The Inherent Dangers of Unidirectional Emotional Bonds between Humans and Social Robots
Lin, P. K. F., Abney, K., & Bekey, G. A. (2012) · Cited by 153 · see citing works
Monroe, A. E., Dillon, K. D., & Malle, B. F. (2014) · Consciousness and Cognition · doi:10.1016/j.concog.2014.04.011 · Cited by 125 · see citing works
Self-citationMoral competence in social robots
Malle, B. F., & Scheutz, M. J. (2014) · IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS) · doi:10.1109/ethics.2014.6893446 · Cited by 112 · see citing works
Self-citationThe utility of affect expression in natural language interactions in joint human-robot tasks
Scheutz, M. J., Schermerhorn, P., & Kramer, J. (2006) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/1121241.1121281 · Cited by 98 · see citing works
Self-citationReal robots that pass human tests of self-consciousness
Bringsjord, S., Licato, J., Govindarajulu, N. S., Ghosh, R., & Sen, A. (2015) · IEEE RO-MAN · doi:10.1109/roman.2015.7333698 · Cited by 88 · see citing works
Briggs, G., & Scheutz, M. J. (2014) · International Journal of Social Robotics · doi:10.1007/s12369-014-0235-1 · Cited by 86 · see citing works
Self-citationThe Need for Moral Competency in Autonomous Agent Architectures
Scheutz, M. J. (2016) · Synthese Library/Synthese library · doi:10.1007/978-3-319-26485-1_30 · Cited by 72 · see citing works
Self-citation“Think and do the right thing” — A Plea for morally competent autonomous robots
Scheutz, M. J., & Malle, B. F. (2014) · doi:10.1109/ethics.2014.6893457 · Cited by 67 · see citing works
Self-citationAdaptive eye gaze patterns in interactions with human and artificial agents
Yu, C., Schermerhorn, P., & Scheutz, M. J. (2012) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2070719.2070726 · Cited by 59 · see citing works
Self-citationInvestigating multimodal real-time patterns of joint attention in an HRI word learning task
Yu, C., Scheutz, M. J., & Schermerhorn, P. (2010) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2010.5453181 · Cited by 55 · see citing works
Self-citationWada, K., & Shibata, T. (2007) · IEEE RO-MAN · doi:10.1109/roman.2007.4415062 · Cited by 53 · see citing works
Behavioral synchronization in human conversational interaction
Rotondo, J. L., & Boker, S. M. (2002) · Advances in consciousness research · doi:10.1075/aicr.42.13rot · Cited by 33 · see citing works
Are Robots Ready for Administering Health Status Surveys'
Briggs, P., Scheutz, M. J., & Tickle‐Degnen, L. (2015) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/2696454.2696476 · Cited by 31 · see citing works
Self-citationSociable Robots and the Future of Social Relations: Proceedings of Robo-Philosophy 2014
Seibt, J., Hakli, R., & Nørskov, M. (2014) · Cited by 25 · see citing works
When will people regard robots as morally competent social partners?
Malle, B. F., & Scheutz, M. J. (2015) · IEEE RO-MAN · doi:10.1109/roman.2015.7333667 · Cited by 17 · see citing works
Self-citationModeling Blame to Avoid Positive Face Threats in Natural Language Generation
Briggs, G., & Scheutz, M. J. (2014) · doi:10.3115/v1/w14-5001 · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
2 of the 27 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Integrating robot ethics and machine morality: the study and design of moral competence in robots
Malle, B. F. (2015) · Ethics and Information Technology · doi:10.1007/s10676-015-9367-8 · Cited by 202 · see citing works
Self-citationA Survey of Mental Modeling Techniques in Human–Robot Teaming
Tabrez, A., Luebbers, M. B., & Hayes, B. (2020) · Current Robotics Reports · doi:10.1007/s43154-020-00019-0 · Cited by 98 · see citing works
Gómez, M. M., Alonso-Martín, F., Malfáz, M., Castro‐González, Á., Castillo, J. C., & Salichs, M. A. (2023) · International Journal of Social Robotics · doi:10.1007/s12369-023-00977-3 · Cited by 68 · see citing works
Language-Capable Robots may Inadvertently Weaken Human Moral Norms
Jackson, R. B., & Williams, T. (2019) · 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2019.8673123 · Cited by 64 · see citing works
Malle, B. F., & Scheutz, M. J. (2019) · Springer eBooks · doi:10.1007/978-3-658-17483-5_17 · Cited by 48 · see citing works
Self-citationKim, B., Wen, R., Zhu, Q., Williams, T., & Phillips, E. K. (2021) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/3434074.3446908 · Cited by 48 · see citing works
Ethics, Equity, & Justice in Human-Robot Interaction: A Review and Future Directions
Ostrowski, A. K., Walker, R., Das, M., Yang, M., Breazea, C., Park, H. W., & Verma, A. (2022) · 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) · doi:10.1109/ro-man53752.2022.9900805 · Cited by 47 · see citing works
Methods for Expressing Robot Intent for Human–Robot Collaboration in Shared Workspaces
LeMasurier, G., Bejerano, G., Albanese, V., Parrillo, J., Yanco, H. A., Amerson, N., Hetrick, R., & Phillips, E. K. (2021) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3472223 · Cited by 35 · see citing works
The Confucian Matador: Three Defenses Against the Mechanical Bull
Williams, T., Zhu, Q., Wen, R.-C., & de Visser, E. (2020) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/3371382.3380740 · Cited by 28 · see citing works
Robots for Social Justice (R4SJ): Toward a More Equitable Practice of Human-Robot Interaction
Zhu, Y., Wen, R., & Williams, T. (2024) · Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3610977.3634944 · Cited by 19 · see citing works
“Because robot speech may lead users to attribute robots with intelligence [6, 14], social agency [28], moral agency [16, 52, 57], and the uniquely potent combination of these factors [27], robots may be accepted as informed community members (although cp. [7]), granting them an outsized potential…”
BackgroundHaring, K. S., Novitzky, M. M., Robinette, P., de Visser, E. J., Wagner, A. R., & Williams, T. (2019) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri.2019.8673184 · Cited by 19 · see citing works
Wen, R., Zhao, H., & Williams, T. (2022) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1109/hri53351.2022.9889594 · Cited by 17 · see citing works
Towards Formal Definitions of Blameworthiness, Intention, and Moral Responsibility
Halpern, J. Y., & Max Kleiman-weiner, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11557 · Cited by 16 · see citing works
Malle, B. F., & Scheutz, M. J. (2018) · Springer Reference Geisteswissenschaften · doi:10.1007/978-3-658-17484-2_17-1 · Cited by 15 · see citing works
Self-citationEnabling Morally Sensitive Robotic Clarification Requests
Jackson, R. B., & Williams, T. (2022) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3503795 · Cited by 15 · see citing works
The power of justifications to repair human-robot trust, even under moral disagreement
Phillips, E. K., & Malle, B. F. (2025) · Scientific Reports · doi:10.1038/s41598-025-17983-2 · Cited by 10 · see citing works
Self-citationAn Integrated Approach to Context-Sensitive Moral Cognition in Robot Cognitive Architectures
Jackson, R. B., Li, S., Banisetty, S. B., Siva, S., Zhang, H., Dantam, N. T., & Williams, T. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636434 · Cited by 9 · see citing works
Safety in AI-HRI: Challenges Complementing User Experience Quality.
Freedman, R. G., & Shlomo Zilberstein, B. (2016) · National Conference on Artificial Intelligence · Cited by 7 · see citing works
Enabling Basic Normative HRI in a Cognitive Robotic ArchitecturePreprint
Sarathy, V., Wilson, J. R., Arnold, T., & Scheutz, M. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03814 · Cited by 6 · see citing works
Self-citationMoral Competence and Moral Orientation in Robots
Schmiljun, A. (2019) · ETHICS IN PROGRESS · doi:10.14746/eip.2019.2.9 · Cited by 3 · see citing works
4.1 Moralische Kompetenz in Robotern: Wie ist der Stand der Forschung?
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-020 · Cited by 1 · see citing works
Gittens, C. L., Jiang, Y., & Hung, P. C. K. (2022) · IEEE RO-MAN · doi:10.1109/ro-man53752.2022.9900609
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-032
3.1 Welche Ethik ist die Richtige?
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-015
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-028
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-004
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-017
3.4 Altruistische und empathische Maschinen
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-018
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-fm
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-033
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-027
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-007
4.5 Die Stufen der moralischen Orientierung
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-024
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-026
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-016
1.5 Können Roboter Personen sein?
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-008
2.3 »Kognitive Wende« in der Psychologie«
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-011
2.5 Linds Zwei-Aspekt-Theorie: Moral ist lehrbar
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-013
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-003
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-019
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-002
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-025
2.6 Warum brauchen Roboter moralische Kompetenz?
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-014
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-021
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-toc
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-009
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-001
4.4 Dual-Aspekt-Dual-Ebenen Modell
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-023
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-029
6.2 Mein Freund, der Roboter – eine Unterrichtseinheit für die Grundschule
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-030
4.3 Der funktionalistische Ansatz: Moralischen Kompetenz als Komponenten-Modell
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-022
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-010
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-005
1.3 Roboter versus KI: Was ist der Unterschied?
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-006
2.4 Die »Renaissance des Gefühls«
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-012
Schmiljun, A., & Schmiljun, I. M. (2024) · Edition Moderne, Postmoderne · doi:10.14361/9783839467206-031
- Authors
- Scheutz, M., Malle, B., & Briggs, G.
- Affiliations
- Tufts University (US)
- Brown University (US)
- Venue
- 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN)
- Publisher
- IEEE
- Published
- 2015-08
- Pages
- 492-497
- ISSN
- 1944-9437 (ISSN-L)
- OpenAlex
- W2132655439
- Cited by
- 59
- References
- 27
- Citations by year
- 2025: 1 · 2024: 37 · 2023: 1 · 2022: 4 · 2021: 3 · 2020: 2 · 2019: 5 · 2018: 3 · 2016: 2 · 2015: 1
- FWCI
- 3.52
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE RO-MAN (publishedVersion)
- Funders
- Multidisciplinary University Research Initiative
- Office of Naval Research
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Life Sciences › Neuroscience › Cognitive Neuroscience › Psychology of Moral and Emotional Judgment
- Keywords
- moral norms, norm violation detection, action selection, ethical dilemmas, moral reasoning, blame, execution traces, human-robot interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
