CoChat

Capable but Amoral? Comparing AI and Human Expert Collaboration in Ethical Decision Making

Open access
Tolmeijer, Suzanne, Christen, Markus, Kandul, Serhiy, Kneer, Markus, Bernstein, Abraham · 2022 · CHI Conference on Human Factors in Computing Systems

The DOI matches its Crossref record; 115 of the 116 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

While artificial intelligence (AI) is increasingly applied for decision-making processes, ethical decisions pose challenges for AI applications. Given that humans cannot always agree on the right thing to do, how would ethical decision-making by AI systems be perceived and how would responsibility be ascribed in human-AI collaboration? In this study, we investigate how the expert type (human vs. AI) and level of expert autonomy (adviser vs. decider) influence trust, perceived responsibility, and reliance. We find that participants consider humans to be more morally trustworthy but less capable than their AI equivalent. This shows in participants’ reliance on AI: AI recommendations and decisions are accepted more often than the human expert’s. However, AI team experts are perceived to be less responsible than humans, while programmers and sellers of AI systems are deemed partially responsible instead.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3491102.3517732

Cites

115 of 116 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Screenshot of a decision during the simulation for a human-on-the-loop setting and defense scenario. For the decisions, the left part of the screen showed the possible crash sights, while the upper right corner showed the expert's opinion. The blue icons is the drone being operated, the red icon is an income bomb.
Figure 1 Screenshot of a decision during the simulation for a human-on-the-loop setting and defense scenario. For the decisions, the left part of the screen showed the possible crash sights, while the upper right corner showed the expert's opinion. The blue icons is the drone being operated, the red icon is an income bomb.CC-BY · From the PDF
Figure 2 Overview of the experimental setup. Blue boxes indicate the independent variables: decision type (human-in-theloop vs. human-on-the-loop) and expert type (human vs. AI expert). Green boxes and terms are control variables. Red italic terms are the dependent variables: the trust participants report, the responsibility they assign, and the reliance they show in the decisions they make.
Figure 2 Overview of the experimental setup. Blue boxes indicate the independent variables: decision type (human-in-theloop vs. human-on-the-loop) and expert type (human vs. AI expert). Green boxes and terms are control variables. Red italic terms are the dependent variables: the trust participants report, the responsibility they assign, and the reliance they show in the decisions they make.CC-BY · From the PDF
Figure 3 Boxplot of the assigned responsibility scores. The 'notch' around the median shows a 95% confidence interval of the median.
Figure 3 Boxplot of the assigned responsibility scores. The 'notch' around the median shows a 95% confidence interval of the median.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 33

  1. Nadia Adnan, Shahrina Md Nordin, Mohamad Ariff bin Bahruddin, and Murad Ali. 2018. How trust can drive forward the user acceptance to the technology? In-vehicle technology for autonomous vehicle. Transportation research part A: policy and practice 118 (2018), 819ś836.
  2. Michael Anderson, Susan Leigh Anderson, and Chris Armen. 2004. Towards machine ethics. In AAAI-04 workshop on agent publishers: theory and practice, San Jose, CA. AAAI Press, Phoenix, Arizona, USA, 2ś7.
  3. Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan. 2018. The moral machine experiment. Nature 563, 7729 (2018), 59ś64.
  4. Annette Baier. 1986. Trust and antitrust. ethics 96, 2 (1986), 231ś260.
  5. Judith Baker. 1987. Trust and rationality. Pacific philosophical quarterly 68, 1 (1987), 1ś13.
  6. Gagan Bansal, Besmira Nushi, Ece Kamar, Walter S Lasecki, Daniel S Weld, and Eric Horvitz. 2019. Beyond accuracy: The role of mental models in human-AI team performance. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 7. The AAAI Press, Palo Alto, California USA, 2ś11.
  7. Danton S Char, Nigam H Shah, and David Magnus. 2018. Implementing machine learning in health careÐaddressing ethical challenges. The New England journal of medicine 378, 11 (2018), 981.
  8. Mary L Cummings. 2006. Integrating ethics in design through the value-sensitive design approach. Science and engineering ethics 12, 4 (2006), 701ś715.
  9. Berkeley J Dietvorst, Joseph P Simmons, and Cade Massey. 2015. Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General 144, 1 (2015), 114.
  10. Ziv Epstein, Sydney Levine, David G Rand, and Iyad Rahwan. 2020. Who gets credit for AI-generated art? Iscience 23, 9 (2020), 101515.
  11. Mike Farjam. 2019. On whom would I want to depend; humans or computers? Journal of Economic Psychology 72 (2019), 219ś228.
  12. Ethan Fast and Eric Horvitz. 2017. Long-term trends in the public perception of artificial intelligence. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31. The AAAI Press, Palo Alto, California USA, 963ś969.
  13. Russell Hardin. 1993. The street-level epistemology of trust. Politics & society 21, 4 (1993), 505ś529.
  14. Maria Hedlund and Erik Persson. 2021. Expert responsibility in AI development. Proceedings of the International Conference of Public Policy (ICPP5) 5 (2021), 1ś24.
  15. Richard Holton. 1994. Deciding to trust, coming to believe. Australasian journal of philosophy 72, 1 (1994), 63ś76.
  16. Ekaterina Jussupow, Izak Benbasat, and Armin Heinzl. 2020. Why are we Averse towards Algorithms? A Comprehensive Literature Review on Algorithmic Aver- sion. In Proceedings of the 28th European Conference on Information Systems (ECIS). ECIS, Marrakech, Morocco, 1ś16.
  17. Guy Kahane, Jim AC Everett, Brian D Earp, Lucius Caviola, Nadira S Faber, Molly J Crockett, and Julian Savulescu. 2018. Beyond sacrificial harm: A two- dimensional model of utilitarian psychology. Psychological Review 125, 2 (2018), 131.
  18. Serhiy Kandul and Oliver Kirchkamp. 2018. Do I care if others lie? Current and future effects when lies can be delegated. Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics) 74, C (2018), 70ś78.
  19. Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017. Inherent Trade-Offs in the Fair Determination of Risk Scores. In 8th Innovations in Theoret- ical Computer Science Conference (ITCS 2017) (Leibniz International Proceedings in Informatics (LIPIcs), Vol. 67, Christos H. Papadimitriou (Ed.). Schloss Dagstuhlś Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 43:1ś43:23.
  20. Markus Kneer and Michael T Stuart. 2021. Playing the Blame Game with Robots. In Companion of the 2021 ACM/IEEE International Conference on Human-Robot Interaction. Association for Computing Machinery, New York, NY, United States, 407ś411.
  21. John D Lee and Katrina A See. 2004. Trust in automation: Designing for appro- priate reliance. Human factors 46, 1 (2004), 50ś80.
  22. Stefanie M. Faas, Johannes Kraus, Alexander Schoenhals, and Martin Baumann. 2021. Calibrating Pedestrians' Trust in Automated Vehicles: Does an Intent Display in an External HMI Support Trust Calibration and Safe Crossing Behav- ior?. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1ś17.
  23. Andreas Matthias. 2004. The responsibility gap: Ascribing responsibility for the actions of learning automata. Ethics and information technology 6, 3 (2004), 175ś183.
  24. Stephanie M Merritt, Heather Heimbaugh, Jennifer LaChapell, and Deborah Lee. 2013. I trust it, but I don't know why: Effects of implicit attitudes toward automation on trust in an automated system. Human factors 55, 3 (2013), 520ś 534.
  25. John Stuart Mill. 1861. 1998. Utilitarianism, edited with an introduction by Roger Crisp. Oxford University Press, New York, NY, USA.
  26. Mahsan Nourani, Chiradeep Roy, Jeremy E Block, Donald R Honeycutt, Tahrima Rahman, Eric Ragan, and Vibhav Gogate. 2021. Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems. In 26th Inter- national Conference on Intelligent User Interfaces. Association for Computing Machinery, New York, NY, USA, 340ś350.
  27. International Review of the Red Cross. 2019. Artificial intelligence and machine learning in armed conflict: A human-centred approach. Technical Report 102. ICRC.
  28. Marianne Promberger and Jonathan Baron. 2006. Do patients trust computers? Journal of Behavioral Decision Making 19, 5 (2006), 455ś468.
  29. Anuschka Schmitt, Thiemo Wambsganss, Matthias Söllner, and Andreas Janson. 2021. Towards a Trust Reliance Paradox? Exploring the Gap Between Perceived Trust in and Reliance on Algorithmic Advice. In International Conference on Information Systems (ICIS), Vol. 1. ICIS, Austin, Texas, 1ś17.
  30. S Shyam Sundar. 2008. The MAIN model: A heuristic approach to understanding technology effects on credibility. MacArthur Foundation Digital Media and Learning Initiative, Chicago, IL, USA.
  31. the Committee of Ministers of the Council of Europe. 2020. Recommendation CM/Rec(2020)1 of the Committee of Ministers to member States on the human rights impacts of algorithmic systems. Technical Report. Council of Europe.
  32. Peter M Todd and Gerd Gigerenzer. 2000. Précis of simple heuristics that make us smart. Behavioral and brain sciences 23, 5 (2000), 727ś741.
  33. Dakuo Wang, Elizabeth Churchill, Pattie Maes, Xiangmin Fan, Ben Shneiderman, Yuanchun Shi, and Qianying Wang. 2020. From human-human collaboration to Human-AI collaboration: Designing AI systems that can work together with people. In Extended abstracts of the 2020 CHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 1ś6.

Cited by

See all 126

Metadata

Authors
Tolmeijer, S., Christen, M., Kandul, S., Kneer, M., & Bernstein, A.
Affiliations
University of Zurich (CH)
Venue
CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2022-04-27
Pages
1-17
OpenAlex
W4224990314
Cited by
70
References
107
Citations by year
2026: 12 · 2025: 25 · 2024: 23 · 2023: 7 · 2022: 3
FWCI
14.41
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by)
  • PhilPapers (PhilPapers Foundation) (submittedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
  • Zurich Open Repository and Archive (University of Zurich)
Funders
  • Swiss National Science Foundation (07740_187494)
  • Armasuisse Science and Technology (S+T)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › Ethics in Business and Education
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Psychology of Moral and Emotional Judgment
Keywords
artificial intelligence, human-AI collaboration, ethical decision-making, trust in AI, perceived responsibility, AI reliance, moral trust, AI decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.