CoChat
Sign Up
Cited by 151Psychology of Moral and Emotional JudgmentSocial Robot Interaction and HRIEthics and Social Impacts of AI

The Moral Psychology of Artificial Intelligence

Open access
Bonnefon, Jean-François, Rahwan, Iyad, Shariff, Azim·2024·Annual Review of Psychology
DOI 10.1146/annurev-psych-030123-113559

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Moral psychology was shaped around three categories of agents and patients: humans, other animals, and supernatural beings. Rapid progress in artificial intelligence has introduced a fourth category for our moral psychology to deal with: intelligent machines. Machines can perform as moral agents, making decisions that affect the outcomes of human patients or solving moral dilemmas without human supervision. Machines can be perceived as moral patients, whose outcomes can be affected by human decisions, with important consequences for human–machine cooperation. Machines can be moral proxies that human agents and patients send as their delegates to moral interactions or use as a disguise in these interactions. Here we review the experimental literature on machines as moral agents, moral patients, and moral proxies, with a focus on recent findings and the open questions that they suggest.

AI snapshot

Written from the paper’s full text.

TL;DR

This review organizes moral psychology research on AI into machines as moral agents, moral patients, and moral proxies, focusing on experimental findings and open questions. It argues that people apply distinct moral expectations and reactions to machines, including a persistent “machine penalty” in cooperation and unresolved challenges for value alignment.

Key findings

  • As implicit moral agents, people demand high machine performance: a representative German sample believed human experts would have a 20–30% mistake rate for credit default or recidivism but required machines to have a mistake rate below 10%; in a US driver sample, the median respondent believed they were in the top 25% of drivers and estimated two-thirds of crashes would be avoided if everyone drove like them, then required very high safety from self-driving cars.
  • Blame patterns are asymmetric: people judge crashes caused by self-driving cars as more severe and place more blame on the car than on a human causing a comparable crash, but when human and machine jointly produce a crash, people typically blame the human most; algorithmic discrimination also causes less moral outrage than human discrimination.
  • Cooperation shows a machine penalty in incentivized games: in a one-shot trust game, 34% reciprocated a machine’s trust versus 75% for a human; in a one-shot prisoners’ dilemma, 36% cooperated with machines versus 49% with humans; in a dictator game, people allocated 16% to a machine versus 39% to a human; in a public good game, contributions were 40% with machines versus 55% with humans. Repeated games suggest the machine penalty carries over unchanged.
  • Value alignment and proxy use remain unsettled: the Moral Machine Experiment considered nine possible priorities for AVs, leading to millions of scenarios; in a survey of 160 experiments, 82% of instructions did not explain what happened to machine earnings; machine masquerade/AI-mediated communication is described as a vast but understudied field.

Methods

  • Design: narrative review of experimental literature on machines as moral agents, moral patients, and moral proxies; no new empirical data are reported.
  • Data/sample: synthesizes cited experiments including representative population samples, incentivized economic games, autonomous-vehicle dilemma studies, and studies of online bot interactions.
  • Analysis: organizes findings by moral role and discusses open questions; no formal meta-analysis or systematic search protocol is described.

Limitations

  • The paper states that any review will only be a snapshot given the pace of AI change, and that it has not addressed every issue at the intersection of AI and moral psychology, such as AI plagiarism, sexbots, or AI effects on trust between human groups.
  • The paper notes that evidence is often concentrated in automated driving and it is not yet clear whether patterns such as the blame reversal generalize to other domains; it also states that value-alignment normative issues fall beyond moral psychology’s purview and that the machine masquerade section is speculative.
  • The text makes evident that, as a narrative review, it does not report a systematic search or quantitative synthesis, and some cited findings come from single experiments or specific samples.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1146/annurev-psych-030123-113559

Cites

151 of 151 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 117 cited sources checked · 0 supported · 0 flagged

  • Premisecitation

    Providing objective evidence that a machine performs better than humans is not trivial.

    “Providing objective evidence that a machine performs better than humans is not trivial to begin with, though (Kalra & Paddock 2016, Kleinberg et al. 2018, Noy et al. 2018), and psychological biases may complicate things even further.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Kalra & Paddock 2016

      Not checked in the free run.

    • cites Kleinberg et al. 2018

      Not checked in the free run.

    • cites Noy et al. 2018

      Not checked in the free run.

  • Premisecitation

    Waiting for skin cancer detection algorithms to be perfectly accurate would sacrifice thousands of lives that could have been saved by earlier detection.

    “If we wait for skin cancer detection algorithms to be perfectly accurate, we sacrifice the thousands of lives that could have been saved by an earlier detection (Esteva et al. 2017).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Esteva et al. 2017

      Not checked in the free run.

  • Premisecitation

    Requiring self-driving cars to be perfectly safe before allowing them on the road would sacrifice thousands of lives that could have been saved by allowing them on the road sooner.

    “If we require self-driving cars to be perfectly safe before we allow them on the road, we sacrifice the thousands of lives they could have saved by being allowed on the road just a little sooner (Kalra & Groves 2017).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Kalra & Groves 2017

      Not checked in the free run.

  • Premisecitation

    There is ample discussion in the media about the danger that machines will learn, amplify and legitimize the biases embedded in the human decisions they are trained from.

    “First, there is ample discussion in the media about the danger that machines will learn, amplify and legitimize the biases embedded in the human decisions they are trained from (O'neil 2017).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites O'neil 2017

      Not checked in the free run.

  • Premisecitation

    People seem to prefer simple demographic parity when considering university admission algorithms, requiring similar admission rates for all demographic groups of applicants.

    “For example, people seem to prefer simple demographic parity when considering university admission algorithms, that is, to require similar admission rates for all demographic groups of applicants (Srivastava et al. 2019);”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Srivastava et al. 2019

      Not checked in the free run.

  • May combine two claims. 30
  • Classified as probably not a substantial claim. 14

  • May combine two claims.

    Interpretationcitation

    People may have extreme performance requirements for implicit moral machines because they expect a substantial increase over baseline human performance while overestimating that baseline.

    “Indeed, it appears that people may have extreme performance requirements for implicit moral machines, because they expect a substantial increase over baseline human performance, while overestimating this baseline human performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    People may expect machines to not only inherit biases from humans, but also the difficulty of fixing them, perhaps underestimating our ability to reprogram machines given our relative inability to reprogram humans.

    “Third, people may expect machines to not only inherit biases from humans, but also the difficulty of fixing them—perhaps underestimating our ability to reprogram machines, given our relative inability to reprogram humans (Mullainathan 2019).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mullainathan 2019

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Much of the literature on algorithmic fairness is grounded in computer science and impossibility theorems, dealing with the problem that there are many possible mathematical definitions of fairness whose requirements are sometimes impossible to achieve simultaneously.

    “Much of this literature on algorithmic fairness is grounded in computer science and impossibility theorems, dealing with the problem that there are many possible mathematical definitions of fairness, whose requirements are sometimes impossible to achieve simultaneously (for entry points, see Chouldechova 2017, Kleinberg et al. 2017, Pleiss et al. 2017).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Chouldechova 2017

      Not checked in the free run.

    • cites Kleinberg et al. 2017

      Not checked in the free run.

    • cites Pleiss et al. 2017

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In a representative sample of US drivers, the median respondent believed to be in the top 25 percent of drivers, and estimated that two-thirds of car crashes would be avoided if everyone drove like them.

    “In a representative sample of US drivers, the median respondent believed to be in the top 25 percent of drivers, and estimated that two-thirds of car crashes would be avoided if everyone drove like them.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    People do not feel especially outraged when machines discriminate, or at least not as outraged as they would feel if humans discriminated.

    “Experimental results, however, suggest that people do not feel especially outraged when machines discriminate, or at least not as outraged as they would feel if humans discriminated (Bigman et al. 2022, Hidalgo et al. 2021).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Bigman et al. 2022

      Not checked in the free run.

    • cites Hidalgo et al. 2021

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The COMPAS algorithm made the same number of mistakes for black defendants and for white defendants, but it did not make the same mistakes: favorable mistakes were more likely when the defendant was white, and unfavorable mistakes were more likely when the defendant was black.

    “The key result of the analysis was that while the algorithm made the same number of mistakes for black defendants and for white defendants, it did not make the same mistakes—mistakes which were favorable to the defendant were more likely when the defendant was white, and mistakes which were unfavorable to the defendant were more likely when the defendant was black.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Angwin et al. 2016

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The very groups that feel at risk of biased human decisions may be the least averse to letting machines make decisions, seemingly because they are worried enough about the current decisions of humans to be willing to take a chance with machines.

    “There is also a growing body of evidence suggesting that the very groups that feel at risk of biased human decisions may be the least averse to letting machines make decisions—seemingly because they are worried enough about the current decisions of humans to be willing to take a chance with machines (Bigman et al. 2021, Fumagalli et al. 2022, Jago & Laurin 2022, Pammer et al. 2021, 2023).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Bigman et al. 2021

      Not checked in the free run.

    • cites Fumagalli et al. 2022

      Not checked in the free run.

    • cites Jago & Laurin 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    One of the main concerns that Americans have about implicit moral machines is that they do not understand nuance and complexity as well as humans do.

    “For example, one of the main concerns that Americans have about implicit moral machines is that they do not understand nuance and complexity as well as humans do (Smith 2019).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Smith 2019

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Experimental work on the forms of fairness people prefer is mostly disconnected from moral psychology, and its results seem to be highly dependent on the application domain considered in each article.

    “This experimental work is mostly disconnected from moral psychology (for a review, see Starke et al. 2022), and its results seem to be highly dependent on the application domain considered in each article.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Starke et al. 2022

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Agency and patiency are two continuous dimensions of mind perception, and people can perceive machines as occupying various positions in that two-dimensional space.

    “Agency and patiency are two continuous dimensions of mind perception (Wegner & Gray 2016), and people can perceive machines as occupying various positions in that two-dimensional space.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wegner & Gray 2016

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several policy reports about autonomous driving suggest that the minimal requirement before deploying self-driving cars is that they are provably safer than the average human driver.

    “This is the approach taken in several policy reports about autonomous driving, which suggest that the minimal requirement before deploying self-driving cars on our roads is that they are provably safer than the average human driver (Bonnefon et al. 2020b, Luetge 2017, Santoni de Sio 2021).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Bonnefon et al. 2020b

      Not checked in the free run.

    • cites Luetge 2017

      Not checked in the free run.

    • cites Santoni de Sio 2021

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In the domain of autonomous driving, people require their self-driving car to be significantly safer than they themselves are, while substantially overestimating the safety of their own driving.

    “An even stronger bias exists in the domain of autonomous driving (Liu et al. 2019b, Shariff et al. 2021), where people require their self-driving car to be significantly safer than they themselves are, while substantially overestimating the safety of their own driving.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Liu et al. 2019b

      Not checked in the free run.

    • cites Shariff et al. 2021

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    People express concerns about the transparency or intelligibility of medical AI recommendations compared to human doctors, without realizing that they overestimate their ability to understand human doctors.

    “In like vein, people express concerns about the transparency or intelligibility of medical AI recommendations, compared to that of human doctors, without realizing that they overestimate their ability to understand human doctors in the first place (Cadario et al. 2021).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Cadario et al. 2021

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    A representative sample of the German population believed that human experts would have a 20-30 percent mistake rate when predicting credit default or recidivism, and required that machines should have a mistake rate lower than 10 percent.

    “For example, a representative sample of the German population believed that human experts would have a 20-30 percent mistake rate when predicting credit default or recidivism, which is probably an underestimation—and working from this baseline, the same sample required that machines should have a mistake rate lower than 10 percent (Rebitschek et al. 2021).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Rebitschek et al. 2021

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In classic dilemmas where an agent must decide whether to save several lives by sacrificing one, humans are blamed more when they choose to sacrifice one, but this pattern is eliminated or even reversed when a machine makes the decision.

    “For example, in classic dilemmas where an agent must decide whether to save several lives by sacrificing one, humans are blamed more when they choose to sacrifice one, but this pattern is eliminated or even reversed when a machine makes the decision (Malle et al. 2015, Komatsu et al. 2021).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Malle et al. 2015

      Not checked in the free run.

    • cites Komatsu et al. 2021

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    It is very common in AI ethics to provide lists of moral values or ethical principles that AI should simultaneously pursue, but it is far less common to provide guidance on what machines should do when these values are in conflict.

    “It is very common in AI ethics to provide lists of moral values or ethical principles that AI should simultaneously pursue (e.g., beneficence, privacy, dignity, transparency) but it is far less common to provide guidance on what machines should do when these values are in conflict (Mittelstadt 2019, Morley et al. 2020).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Mittelstadt 2019

      Not checked in the free run.

    • cites Morley et al. 2020

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Delegating difficult moral decisions to machines may not merely remove a psychological burden from humans, but also change social expectations about which decision should be made.

    “This implies that delegating difficult moral decisions to machines may not merely remove a psychological burden from humans, but also change social expectations about which decision should be made (Gill 2020).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Gill 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Moral Machine Experiment purposefully used stylized scenarios when a collision is unavoidable.

    “For example, the Moral Machine Experiment purposefully used stylized scenarios when a collision is unavoidable (Awad et al. 2020a)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Awad et al. 2020a

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    As implicit moral agents, AVs are expected to lead to an absolute increase in road safety; but AVs are also explicit moral agents in the sense that every action they take can redistribute relative levels of safety between the road users that surround them.

    “As implicit moral agents, AVs are expected to lead to an absolute increase in road safety; but AVs are also explicit moral agents in the sense that every action they take can redistribute relative levels of safety between the road users that surround them (Goodall 2016, Bonnefon et al. 2019).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Goodall 2016

      Not checked in the free run.

    • cites Bonnefon et al. 2019

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Ethicists are not immune to biases, and they do not always come to an agreement.

    “Ethicists, however, are not immune to biases (Schwitzgebel & Cushman 2015), and they do not always come to an agreementfor example, a German national ethics committee could not reach a consensus on what an autonomous vehicle should do when deciding whether to save its passengers or to save other road users (Luetge 2017).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Schwitzgebel & Cushman 2015

      Not checked in the free run.

    • cites Luetge 2017

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The main promise of AVs is that they can reduce the number of road casualties by being safer than human drivers; but these lives will not be saved if consumers opt out of the technology because they are unsatisfied or outraged with the way AVs solve moral dilemmas.

    “The main promise of AVs is that they can reduce the number of road casualties, by being safer than human drivers; but these lives will not be saved if consumers opt out of the technology because they are unsatisfied or even outraged with the way AVs solve moral dilemmas (Bonnefon et al. 2020a, De Freitas & Cikara 2021).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Bonnefon et al. 2020a

      Not checked in the free run.

    • cites De Freitas & Cikara 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    More realistic scenarios would have manipulated the probability of the collisions.

    “but more realistic scenarios would have manipulated the probability of the collisions (Krügel & Uhl 2022).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Krügel & Uhl 2022

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    People hesitate when asked to strike a robot, or to topple a block tower that a robot built and pretends to care about.

    “they hesitate when asked to strike a robot (Darling et al. 2015), or to topple a block tower that a robot built and pretends to care about (Briggs & Scheutz 2014)”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Darling et al. 2015

      Not checked in the free run.

    • cites Briggs & Scheutz 2014

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Experts from the AV industry may feel a duty to protect their customers, which could explain why they have a stronger preference to save passengers, compared to the general population.

    “Experts from the AV industry may also feel a duty to protect their customers, which could explain why they have a stronger preference to save passengers, compared to the general population (Zhu et al. 2022).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Zhu et al. 2022

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The Moral Machine Experiment considered nine possible priorities for AVs to decide which group of road users to save or to sacrifice, which led to millions of possible scenarios.

    “The Moral Machine Experiment (Awad et al. 2018) considered nine possible priorities for AVs to decide which group of road users to save or to sacrifice, which led to millions of possible scenarios.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Awad et al. 2018

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisereasoning

    Humans will inevitably be blamed for the way they solved a moral dilemma, since by definition a moral dilemma has no universally accepted solution.

    “In like vein, we know that humans will inevitably be blamed for the way they solved a moral dilemma, since by definition a moral dilemma has no universally accepted solution—hence, we may want to relieve human decision-makers from unavoidable blame, by delegating the decision to a machine (see Textbox Blaming machines for solving dilemmas for further discussion).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    The autonomous vehicle industry is very reluctant to engage in the debate about what AVs should do in moral dilemmas, since any position they take may alienate either their consumer base or the general population.

    “One problem is that the industry is very reluctant to engage in this debate (Martinho et al. 2021), since any position they take may alienate either their consumer base or the general population.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Martinho et al. 2021

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    People sometimes assume that machines have preferences, which can turn machines into moral patients.

    “In other terms, and as we will consider in more detail in the rest of this section, people sometimes assume that machines have preferences-which can turn machines into moral patients, who experience preferred or dis-preferred outcomes as a result of the actions taken by other agents (Pauketat & Anthis 2022).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Pauketat & Anthis 2022

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    People do seem to experience toward machines the same manifold of negative reactions they experience toward humans, perhaps because the machines are perceived as autonomous enough to warrant these reactions.

    “But from a psychological perspective, people do seem to experience toward machines the same manifold of negative reactions they experience toward humans, perhaps because the machines are perceived as autonomous enough to warrant these reactions (Bigman et al. 2019, Epstein et al. 2020, Franklin et al. 2022).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Bigman et al. 2019

      Not checked in the free run.

    • cites Epstein et al. 2020

      Not checked in the free run.

    • cites Franklin et al. 2022

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    People show physiological signs of discomfort when watching a baby dinosaur robot being punched and choked, or a robot hand being cut by a knife.

    “For example, research participants show physiological signs of discomfort when watching a baby dinosaur robot being punched and choked (Rosenthal-von der Pütten et al. 2013), or a robot hand being cut by a knife (Suzuki et al. 2015)”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Rosenthal-von der Pütten et al. 2013

      Not checked in the free run.

    • cites Suzuki et al. 2015

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisecitation

    Making moral decisions inflicts an emotional cost on the decision-maker, both in the short and the long term.

    “If we agree that making such decisions inflict an emotional cost on the decision-maker, both in the short and the long term, we may agree that it is good to delegate this burden to machines, who do not experience psychological suffering (Danaher 2022).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Danaher 2022

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    All other things being equal, people judge crashes as more severe and less acceptable when they are caused by self-driving cars, and place more blame and responsibility on a self-driving car causing a crash, than on a human causing a comparable crash.

    “All other things being equal, people judge crashes as more severe and less acceptable when they are caused by self-driving cars, and place more blame and responsibility on a self-driving car causing a crash, than on a human causing a comparable crash.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Franklin et al. 2021

      Not checked in the free run.

    • cites Hidalgo et al. 2021

      Not checked in the free run.

    • cites Hong et al. 2020

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Humanization strategies usually fail to reduce the machine penalty, and the ones that succeed fall short of current ethical standards.

    “In sum, humanization strategies usually fail to reduce the machine penalty, and the ones that succeed (partially or totally) fall short of current ethical standards.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Climbing up the humanization gradient does not improve cooperation much, and can even make things worse due to the uncanny valley effect.

    “Climbing up the humanization gradient does not improve cooperation much, and can even make things worse, due to the uncanny valley effect-that is, the feeling of strangeness and discomfort elicited by a machine that is largely but not quite human-like.”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Mathur & Reichling 2016

      Not checked in the free run.

    • cites Zlotowski et al. 2016

      Not checked in the free run.

    • cites Cominelli et al. 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Written text, profile pictures, as well as voice and facial dynamics in live online interactions can already be altered to achieve various presentation goals.

    “Written text, profile pictures, as well as voice and facial dynamics in live online interactions can already be altered to achieve various presentation goals.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Incentivized games provide a controlled, stylized environment to measure other-regarding preferences and prosocial behavior, that allows for easy comparison of studies and experimental treatments.

    “Incentivized games provide a controlled, stylized environment to measure other-regarding preferences and prosocial behavior, that allows for easy comparison of studies and experimental treatments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    ChatGPT attracted more than a hundred million users within weeks of its public launch.

    “Within weeks of its public launch, ChatGPT attracted more than a hundred million of users, affording them seemingly endless possibilities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    AI-powered deepfakes can create fake identities, which allows phishing attacks to become more personalized and effective.

    “AI-powered deepfakes can create fake identities, which allows phishing attacks to become more personalized and effective.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Seymour & Tully 2016

      Not checked in the free run.

    • cites Jagatic et al. 2007

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    These alterations can raise new ethical issues for moral psychology to investigate.

    “but they can also raise new ethical issues for moral psychology to investigate”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A panel of experts has nominated deepfakes as the most dangerous tool for AI-enabled crime.

    “Reflecting on this emerging worry, a panel of experts has nominated deepfakes as the most dangerous tool for AI-enabled crime (Caldwell et al. 2020).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Caldwell et al. 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    AI-mediated communication has received little attention in moral psychology.

    “Moral psychology has given little attention so far to AI-mediated communication, but this is likely to change given the incoming availability of machine masquerade tools, the way they will transform moral interactions, and the ethical challenges they raise.”
    Quote in the paperNot a substantial claim
41 of 41

  • The paper relies on this without support.

    There will almost always be a human in the same loop as the machine, so mistakes will most often be the result of a joint failure of human and machine.

    “Since there will almost always be a human in the same loop as the machine, mistakes will most often be the result of a joint failure of human and machine—so, how do people allocate responsibility and blame between humans and machines, when both contributed to a harmful mistake?”

  • The authors state this as an assumption.

    It is possible to capture the manifold of human cooperation by using lab-based games with financial incentives.

    “experimental economics has popularized the assumption that it is possible to capture the manifold of human cooperation by using lab-based games with financial incentives, such as dictator games, prisoners' dilemmas, ultimatum games, or public good games.”

  • The authors state this as an assumption.

    Money can be used as a proxy for the many currencies of human-machine cooperation.

    “These studies carry over the assumption that just as money can be used as a proxy for the many currencies of human-human cooperation, it can be used as a proxy for the many currencies of human-machine cooperation.”

  • The paper relies on this without support.

    Machines have no affects, needs, or desires for anything.

    “This may sound strange, since machines have no affects, nor needs or desires for anything.”

  • The paper relies on this without support.

    AI enables machines to make autonomous decisions on behalf of human stakeholders.

    “By design, AI enables machines to make autonomous decisions on behalf of human stakeholders.”

  • The paper relies on this without support.

    AI can act independently.

    “because AI can act independently and has the potential to cause harm with unparalleled efficiency and at scale.”

  • The paper relies on this without support.

    People can already experiment with generative AI to create their profile pictures.

    “People can already experiment with generative AI to create their profile pictures”

  • The paper relies on this without support.

    The very recent and rapid emergence of AI-driven technology is colliding with moral intuitions forged by culture and evolution over the span of millennia.

    “the very recent and rapid emergence of AI-driven technology is colliding with moral intuitions forged by culture and evolution over the span of millennia.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Simulations are used to model ethical dilemmas, especially in areas like autonomous vehicles and healthcare, allowing people to evaluate AI decisions in realistic and interactive scenarios.

    “Simulations [162] are used to model ethical dilemmas, especially in areas like autonomous vehicles and healthcare, allowing people to evaluate AI decisions in realistic and interactive scenarios.”

    From The Rise of Agentic AI: A Review of Definitions, Frameworks, Architectures, Applications, Evaluation Metrics, and Challenges (2025) · cites this paper as [162]

    In “Blaming Machines for Solving Dilemmas” (page unavailable), the review describes autonomous-vehicle dilemma experiments, including virtual environments in which participants make decisions. It does not describe comparable healthcare simulations, and it says the Moral Machine Experiment used stylized scenarios rather than realistic ones. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Systems currently struggle to properly interpret and align with the subtlety of human intention, leading to misalignment of action, including on even simple tasks and moral misalignment on work meant to help others.

    “Systems currently struggle to properly interpret and align with the subtlety of human intention, leading to misalignment of action, including on even simple tasks and moral misalignment on work meant to help others [162].”

    From The Rise of Agentic AI: A Review of Definitions, Frameworks, Architectures, Applications, Evaluation Metrics, and Challenges (2025) · cites this paper as [162]

    Section 4.1, “Delegation to Machines” (page unavailable), says AI delegation can cause moral violations without bad intent and gives examples of unintended collusion and deception. Section 2.1.1, “Performance” (page unavailable), describes concern that machines miss nuance, but frames it as a perception rather than establishing that systems generally fail to interpret human intentions. The paper does not substantiate the claim about misaligned action even on simple tasks. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Lack of relevant oversight processes as a safeguard may make it difficult for humans to detect or intervene in harmful behavior when it occurs.

    “Lack of relevant oversight processes as a safeguard may also make it difficult for humans to detect or intervene in harmful behavior when it occurs [162].”

    From The Rise of Agentic AI: A Review of Definitions, Frameworks, Architectures, Applications, Evaluation Metrics, and Challenges (2025) · cites this paper as [162]

    Section 4.1, “Delegation to Machines” (page not provided), says AI delegation can make unethical behavior undetectable and gives examples of people being unaware of harmful AI conduct. It does not say that a lack of oversight processes makes human intervention difficult when harm occurs. (LLM, conf=0.79)

    Experimental — accuracy not yet measured

Cites

Flagged first, then by citations

Research works printed but not linked: 12

  1. Angwin J, 2016, ProPublica
  2. Buolamwini J, 2018, PMLR, 81, 77
  3. Hamilton M., 2019, Am. Crim. Law Rev., 56, 1553
  4. Hao K, 2019, MIT Technology Review
  5. Kleinberg J, 2018, Q. J. Econ., 133, 237
  6. McAllister A., 2016, Minn. Law Rev., 101, 2527
  7. McCallum S, 2022, BBC News
  8. Mullainathan S., 2019, New York Times
  9. Pleiss G, 2017, Adv. Neural Inform. Process. Syst., 30, 5684
  10. Samuel S, 2020, Transp. Res. Interdiscip. Perspect., 5, 100147
  11. Smith A., 2019, Public attitudes toward computer algorithms
  12. Złotowski J, 2016, Paladyn J. Behav. Robot., 7, 55

Cited by

Sorted by citations

Related

Authors
Bonnefon, J.-F., Rahwan, I., & Shariff, A.
Affiliations
  • Centre National de la Recherche Scientifique (FR)
  • Max Planck Institute for Human Development (DE)
  • University of British Columbia (CA)
ORCID
Venue
Annual Review of Psychology
Publisher
Annual Reviews
Published
2024-01-18
Volume
75
Issue
1
Pages
653-675
ISSN
  • 0066-4308 (ISSN-L)
  • 1545-2085
DOI
10.1146/annurev-psych-030123-113559
OpenAlex
W4386820363
Semantic Scholar
97217d457944ee4267d9e439646fa75e5d5cd156
Cited by
151
References
148
Citations by year
2026: 61 · 2025: 58 · 2024: 26 · 2023: 6
FWCI
5.67
Type
journal article
Language
en
Open access
Licence
Locations
  • Annual Review of Psychology (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • HAL (submittedVersion, other-oa)
  • HAL (submittedVersion)
  • Toulouse 1 Capitole Publications (Université Toulouse I Capitole) (submittedVersion)
  • MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
Funders
  • Agence Nationale de la Recherche (ANR-17-EURE-0010, ANR-19-P3IA-0004, ANR-17-EURE-, ANR‐19, PI3A-0004, 19-PI3A-0004, 17-EURE-00-10, ANR-19-PI3A-0004, ANR-17)
  • Social Sciences and Humanities Research Council of Canada
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Psychology of Moral and Emotional Judgment
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
moral psychology, artificial intelligence, moral agents, moral patients, human-machine collaboration, ethical dilemmas, anthropomorphism
MeSH terms
Animals, Artificial Intelligence, Humans, Intelligence, Morals

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review