Risk of injury in moral dilemmas with autonomous vehicles
Open accessAbstract
As autonomous machines, such as automated vehicles (AVs) and robots, become pervasive in society, they will inevitably face moral dilemmas where they must make decisions that risk injuring humans. However, prior research has framed these dilemmas in starkly simple terms, i.e., framing decisions as life and death and neglecting the influence of risk of injury to the involved parties on the outcome. Here, we focus on this gap and present experimental work that systematically studies the effect of risk of injury on the decisions people make in these dilemmas. In four experiments, participants were asked to program their AVs to either save five pedestrians, which we refer to as the utilitarian choice, or save the driver, which we refer to as the nonutilitarian choice. The results indicate that most participants made the utilitarian choice but that this choice was moderated in important ways by perceived risk to the driver and risk to the pedestrians. As a second contribution, we demonstrate the value of formulating AV moral dilemmas in a game-theoretic framework that considers the possible influence of others’ behavior. In the fourth experiment, we show that participants were more (less) likely to make the utilitarian choice, the more utilitarian (nonutilitarian) other drivers behaved; furthermore, unlike the game-theoretic prediction that decision-makers inevitably converge to nonutilitarianism, we found significant evidence of utilitarianism. We discuss theoretical implications for our understanding of human decision-making in moral dilemmas and practical guidelines for the design of autonomous machines that solve these dilemmas while, at the same time, being likely to be adopted in practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Utilitarianism has garnered the highest support in previous surveys of trolley dilemma.
“Among these approaches, utilitarianism, advocating for the preservation of more lives at the expense of fewer, has garnered the highest support in previous surveys of trolley dilemma (Hauser et al., 2007; Rehman and Dzionek Kozłowska, 2018; de Melo et al., 2021).”
From Exploring moral algorithm preferences in autonomous vehicle dilemmas: an empirical study (2023) · cites this paper as de Melo et al., 2021
Experiment 4 (Method, §2.4; page unavailable) reports that participants often made utilitarian choices, and the General Discussion (§3; page unavailable) describes a preference for utilitarian choices under some risk conditions. The paper does not establish that utilitarianism received the highest support in previous trolley-dilemma surveys; its findings concern its own autonomous-vehicle scenarios. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 37 · Cited by 17
Research works printed but not linked: 28
- Bazerman M. D Greene J. (2010). In favor of clear thinking: incorporating moral rules into a wise cost-benefit analysis-commentary on bennis, medin, & bartels (2010). Perspect. Psychol. Sci.) 5, 209– 212. 10.1177/1745691610362362 26162128 · DOI
- Bennis W. Medin D. Bartels D. (2010). The costs and benefits of calculation and moral rules. Perspect. Psychol. Sci. 5, 187– 202. 10.1177/1745691610362354 26162125 · DOI
- Bentham J. (1879). The collected works of Jeremy Bentham: deontology, together with a table of the springs of action; and the article on utilitarianism. Oxford, UK: Oxford University Press.
- Bonnefon J. F. Shariff A. Rahwan I. (2016). The social dilemma of autonomous vehicles. Science. 352, 1573– 1576. 10.1126/science.aaf2654 27339987 · DOI · Paper page
- Bostyn D. Roets A. (2017). An asymmetric moral conformity effect: subjects conform to deontological but not consequentialist majorities. Social Psychological and Personality Science. 8, 323– 330. 10.1177/1948550616671999 · DOI
- Bostyn D. Sevenhant S. Roets A. (2018). Of mice, men, and trolleys: hypothetical judgment versus real-life behavior in trolley-style moral dilemmas. Psychol. Sci. 29, 1084– 1093. 10.1177/0956797617752640 29741993 · DOI
- Conitzer V. Sinnott-Armstrong W. Borg J. Deng Y. Kramer M. (2017). “Moral decision making frameworks for artificial intelligence.” Proceedings of AAAI’17, New York, NY, February 4–10, 2017.
- de Melo C. Marsella S. Gratch J. (2019). Human cooperation when acting through autonomous machines. Proceedings of the National Academy of Sciences U.S.A. 116, 3482– 3487. 10.1073/pnas.1817656116 PMC6397531 30808742 · DOI
- Deng B. (2015). Machine ethics: the robot's dilemma. Nature. 523, 24– 26. 10.1038/523024a 26135432 · DOI
- Faulhaber A. Dittmer A. Blind F. Wächter M. Timm S. Sütfeld L. R. (2019). Human decisions in moral dilemmas are largely described by utilitarianism: virtual car driving study provides guidelines for autonomous driving vehicles. Sci. Eng. Ethics. 25, 399– 418. 10.1007/s11948-018-0020-x 29357047 · DOI
- Francis K. Howard C. Howard I. Gummerum M. Ganis G. (2016). Virtual morality: transitioning from moral judgment to moral action? PloS One 12(1), e0170133. 10.1371/journal.pone.0164374 PMC5226716 28076403 · DOI
- Gogoll J. Müller J. (2017). Autonomous cars: in favor of a mandatory ethics setting. Sci. Eng. Ethics. 23, 681– 700. 10.1007/s11948-016-9806-x 27417644 · DOI · Paper page
- Greene J. Sommerville R. Nystrom L. Darley J. Cohen J. (2001). An fmri investigation of emotional engagement in moral judgment. Science. 293, 2105– 2108. 10.1126/science.1062872 11557895 · DOI · Paper page
- Hertwig R. Ortmann A. (2001). Experimental practices in economics: a methodological challenge for psychologists?. Behav. Brain Sci. 24, 383– 451. 10.1037/e683322011-032 11682798 · DOI
- Hornsey M. Majkut L. Terry D. McKimmie B. (2003). On being loud and proud: non-conformity and counter-conformity to group norms. Br. J. Soc. Psychol. 42, 319– 335. 10.1348/014466603322438189 14567840 · DOI
- Kant I. (1797). Groundwork for the metaphysics of morals. New Haven, CT: Yale University Press.
- Kundu P. Cummins D. (2013). Morality and conformity: the asch paradigm applied to moral decisions. Soc. Influ. 8, 268– 279. 10.1080/15534510.2012.727767 · DOI
- Lin P. (2015). in Why ethics matters for autonomous cars. Autonomes Fahren. Technische, rechtliche und gesellschaftliche Aspekte. Editors Maurer M. Gerdes J. C. Lenz B. Winner H. (Berlin, Germany: Springer), 69– 85.
- Mikhail J. (2007). Universal moral grammar: theory, evidence and the future. Trends Cognit. Sci. 11, 143– 152. 10.1016/j.tics.2006.12.007 17329147 · DOI
- Pan X. Slater M. (2011). Real-time human motion capture with multiple depth cameras,” in Proceedings of the 25th BCS Conference on human-computer interaction, The University of British Columbia, Vancouver, Canada,
- Rand D. Nowak M. (2013). Human cooperation. Trends Cognit. Sci. 17, 413– 425. 10.1016/j.tics.2013.06.003 23856025 · DOI · Paper page
- Rom S. Conway P. (2018). The strategic moral self: self-presentation shapes moral dilemma judgments. J. Exp. Soc. Psychol. 74, 24– 37. 10.1016/j.jesp.2017.08.003 · DOI
- Scanlon T. (1998). What we owe to each other., Belknap, MA: Harvard University Press.
- Swann W. Buhrmester A. Gomez López-Rodríguez L. Vázquez J. (2014). Contemplating the ultimate sacrifice: identity fusion channels pro-group affect, cognition, and moral decision making. J. Pers. Soc. Psychol. 106, 713– 727. 10.1037/a0035809 24749820 · DOI
- Tetlock P. (2003). Thinking the unthinkable: sacred values and taboo cognitions. Trends Cognit. Sci. 7, 320– 324. 12860191
- von Neumann J. Morgenstern O. (1944). Theory of Games and economic behavior. New Jersey, NJ: Princeton University Press.
- Waldrop M. (2015). Autonomous vehicles: No drivers required. Nature. 518, 20– 23. 10.1038/518020a 25652978 · DOI
- Wang X. Krumhuber E. Gratch J. (2018). The interpersonal effects of emotions in money versus candy games. J. Exp. Soc. Psychol. 79, 315– 327. 10.1016/j.jesp.2018.08.014 · DOI
- Authors
- de Melo, C. M., Marsella, S., & Gratch, J.
- Affiliations
- Northeastern University (US)
- University of Southern California (US)
- Venue
- Frontiers in Robotics and AI
- Publisher
- Frontiers Media SA
- Published
- 2021-01-20
- Volume
- 7
- Pages
- 572529-572529
- ISSN
- 2296-9144 (ISSN-L)
- PMC
- PMC8239464
- OpenAlex
- W3125839461
- Cited by
- 17
- References
- 37
- Citations by year
- 2025: 5 · 2024: 2 · 2023: 8 · 2022: 2
- FWCI
- 1.58
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Robotics and AI (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Air Force Office of Scientific Research (FA9550-18-1–0182)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Psychology of Moral and Emotional Judgment
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Social Sciences › Decision Sciences › Information Systems and Management › Ethics in Business and Education
- Keywords
- autonomous vehicles, ethical dilemmas, injury risk, utilitarianism, human decision-making, risk perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
