Human Perceptions on Moral Responsibility of AI: A Case Study in AI-Assisted Bail Decision-Making
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
How to attribute responsibility for autonomous artificial intelligence (AI) systems’ actions has been widely debated across the humanities and social science disciplines. This work presents two experiments (N=200 each) that measure people’s perceptions of eight different notions of moral responsibility concerning AI and human agents in the context of bail decision-making. Using real-life adapted vignettes, our experiments show that AI agents are held causally responsible and blamed similarly to human agents for an identical task. However, there was a meaningful difference in how people perceived these agents’ moral responsibility; human agents were ascribed to a higher degree of present-looking and forward-looking notions of responsibility than AI agents. We also found that people expect both AI and human decision-makers and advisors to justify their decisions regardless of their nature. We discuss policy and HCI implications of these findings, such as the need for explainable AI in high-stakes scenarios.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“Algorithms are used to choose which candidate is most ft for a job position [111], decide which defendants are granted bail [33], guide health-related decision [73], and assess credit risk [49].”
MethodLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“We highlight that those who are in charge of developing interpretable systems should not try to “nudge” recipients so they can be manipulated [63], e.”
BackgroundFloridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., Luetge, C., Madelin, R., Pagallo, U., Rossi, F., et al. (2018) · Minds and Machines · doi:10.1007/s11023-018-9482-5 · Cited by 4,774 · cited 1×
“They argue that understanding how these systems come to their conclusions is necessary for both safe deployment and wide adoption [36].”
BackgroundProlific.ac—A subject pool for online experiments
Palan, S., & Schitter, C. (2018) · Journal of Behavioral and Experimental Finance · doi:10.1016/j.jbef.2017.12.004 · Cited by 3,450
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“Explainable AI (XAI) [46] is a feld of computer science that has been given much attention in the community [38], and our results suggest that people agree with its importance.”
BackgroundPrinciples alone cannot guarantee ethical AI
Mittelstadt, B. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0114-4 · Cited by 1,886 · cited 2×
“Explanations involve both an explainer and explainee, meaning that conficts might arise concerning how they are evaluated [69].”
BackgroundAwad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.-F., & Rahwan, I. (2018) · Nature · doi:10.1038/s41586-018-0637-6 · Cited by 1,859 · cited 1×
“Literature exists on the public perception of moral and legal issues concerning AI [6, 7, 62].”
BackgroundThe social dilemma of autonomous vehicles
Bonnefon, J.-F., Shariff, A., & Rahwan, I. (2016) · Science · doi:10.1126/science.aaf2654 · Cited by 1,527
Lee, M. K. (2018) · Big Data & Society · doi:10.1177/2053951718756684 · Cited by 1,338 · cited 2×
“, [43, 56, 81]), particularly the prospect of responsibility ascription to the AI system per se.”
BackgroundThe responsibility gap: Ascribing responsibility for the actions of learning automata
Matthias, A. (2004) · Ethics and Information Technology · doi:10.1007/s10676-004-3422-1 · Cited by 1,101 · cited 3×
“Who should be held responsible for the harm caused by artifcial intelligence (AI)? This question has been debated for over a decade since Matthias’ landmark essay on the responsibility gap of autonomous machines [68].”
BackgroundInfluentialThe accuracy, fairness, and limits of predicting recidivism
Dressel, J., & Farid, H. (2018) · Science Advances · doi:10.1126/sciadv.aao5580 · Cited by 1,071 · cited 2×
“Firstly, we adapted the vignette presentation, which was initially adapted from previous work [33].”
MethodCrime and punishment: Distinguishing the roles of causal and intentional analyses in moral judgment
Cushman, F. (2008) · Cognition · doi:10.1016/j.cognition.2008.03.006 · Cited by 990
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988
People are averse to machines making moral decisions.
Bigman, Y. E., & Gray, K. (2018) · Cognition · doi:10.1016/j.cognition.2018.08.003 · Cited by 738 · cited 1×
“There exists empirical evidence that people might be averse to machines making moral decisions [10].”
BackgroundMalle, B. F., Guglielmo, S., & Monroe, A. E. (2014) · Psychological Inquiry · doi:10.1080/1047840x.2014.877340 · Cited by 632
SPARROW, R. (2007) · Journal of Applied Philosophy · doi:10.1111/j.1468-5930.2007.00346.x · Cited by 593
Society-in-the-loop: programming the algorithmic social contract
Rahwan, I. (2018) · Ethics and Information Technology · doi:10.1007/s10676-017-9430-8 · Cited by 553
Coeckelbergh, M. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-019-00146-8 · Cited by 475
The ethics of algorithms: key problems and solutions
Tsamados, A., Aggarwal, N., Cowls, J., Morley, J., Roberts, H., Taddeo, M., & Floridi, L. (2022) · AI & SOCIETY · doi:10.1007/s00146-021-01154-8 · Cited by 384
Designing fair AI for managing employees in organizations: a review, critique, and design agenda
Robert, L. P., Pierce, C., Marquis, L., Kim, S., & Alahmad, R. (2020) · Human–Computer Interaction · doi:10.1080/07370024.2020.1735391 · Cited by 268 · cited 1×
“, [43, 56, 81]), particularly the prospect of responsibility ascription to the AI system per se.”
BackgroundSacrifice One For the Good of Many?: People Apply Different Moral Norms to Human and Robot Agents
Malle, B. F., Scheutz, M., Arnold, T., Voiklis, J., & Cusimano, C. (2015) · Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/2696454.2696458 · Cited by 246 · cited 2×
“Likewise, HCI research has found that people attribute blame to robotic agents upon harm, particularly if they are described to be autonomous and serve the main cause of harm [37, 52, 67].”
BackgroundNyholm, S. (2018) · Science and Engineering Ethics · doi:10.1007/s11948-017-9943-x · Cited by 242 · cited 3×
“their automated systems’ actions and consequences [20, 72].”
BackgroundWho should i blame? effects of autonomy and transparency on attributions in human-robot interaction
Kim, T., & Hinds, P. (2006) · ROMAN 2006 - The 15th IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2006.314398 · Cited by 217
Wang, R., Harper, F. M., & Zhu, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376813 · Cited by 211
Grgic-Hlaca, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186138 · Cited by 207
Robots, law and the retribution gap
Danaher, J. (2016) · Ethics and Information Technology · doi:10.1007/s10676-016-9403-3 · Cited by 192 · cited 7×
“The same argument could also be applied to the practice of blame [26].”
BackgroundInfluentialSrivastava, M., Heidari, H., & Krause, A. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330664 · Cited by 179 · cited 1×
“Other scholars explored people’s perceptions of procedural [41] and distributive [84, 90] aspects of algorithmic fairness and studied how they relate to individual diferences [42, 76, 108].”
BackgroundMind the gap: responsible robotics and the problem of responsibility
Gunkel, D. J. (2020) · Ethics and Information Technology · doi:10.1007/s10676-017-9428-2 · Cited by 174 · cited 2×
“These scholars often view these human-AI collaborations as extended agencies where all entities should be held jointly responsible [45, 48].”
BackgroundMoral Responsibility and the Problem of Many Hands
van de Poel, I., Royakkers, L., & Zwart, S. D. (2015) · doi:10.4324/9781315734217 · Cited by 152
Human Decision Making with Machine Assistance: An Experiment on Bailing and Jailing
Grgić-Hlača, N., Engel, C., & Gummadi, K. P. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359280 · Cited by 113 · cited 2×
“, [43, 56, 81]), particularly the prospect of responsibility ascription to the AI system per se.”
BackgroundSelf-citationAgency Laundering and Information Technologies
Rubel, A., Castro, C., & Pham, A. (2019) · Ethical Theory and Moral Practice · doi:10.1007/s10677-019-10030-w · Cited by 90
The Explanatory Component of Moral Responsibility
Björnsson, G., & Persson, K. (2012) · Noûs · doi:10.1111/j.1468-0068.2010.00813.x · Cited by 81 · cited 1×
“According to some scholars, holding AI and robots responsible per se could fulfll specifc social goals [23] and promote critical social functions [11, 91].”
BackgroundGrgić-Hlača, N., Redmiles, E. M., Gummadi, K. P., & Weller, A. (2018) · arXiv · doi:10.48550/arxiv.1802.09548 · Cited by 75
Self-citationAI Systems Under Criminal Law: a Legal Analysis and a Regulatory Perspective
Lagioia, F., & Sartor, G. (2020) · Philosophy & Technology · doi:10.1007/s13347-019-00362-x · Cited by 75
Moral coherence processes: constructing culpability and consequences
Clark, C. J., Chen, E. E., & Ditto, P. H. (2015) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2015.07.016 · Cited by 50 · cited 1×
“The general public might choose to hold a wrongdoer responsible for restoring moral coherence [22] or reafrming a communal moral values [109].”
BackgroundA Summary of Survey Methodology Best Practices for Security and Privacy Researchers
Redmiles, E. M., Acar, Y., Fahl, S., & Mazurek, M. L. (2017) · Digital Repository at the University of Maryland · doi:10.13016/m22k2w · Cited by 50 · cited 1×
“All vignettes were presented in random order to eliminate any order efect [44, 79].”
Cited by
The Moral Psychology of Artificial Intelligence
Bonnefon, J.-F., Rahwan, I., & Shariff, A. (2024) · Annual Review of Psychology · doi:10.1146/annurev-psych-030123-113559 · Cited by 151
Fietta, V., Zecchinato, F., Stasi, B. D., Polato, M., & Monaro, M. (2022) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2021.3125280 · Cited by 81
Capable but Amoral? Comparing AI and Human Expert Collaboration in Ethical Decision Making
Tolmeijer, S., Christen, M., Kandul, S., Kneer, M., & Bernstein, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517732 · Cited by 70
Salimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“…wide range of studies, researchers have consistently found that reliance on AI systems is influenced by various factors including human-related aspects [37, 51, 81, 101, 120], attributes of the AI systems [43, 79, 106, 107], and characteristics of the decision-making tasks [9, 12, 16, 46, 126].”
BackgroundAnthis, J. R., Pauketat, J. V. T., Ladak, A., & Manoli, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713329 · Cited by 24
“In terms of being threatened by AI (i.e., the attribution of negative moral agency), this has been most salient with AI that assists in consequential tasks, such as medical imaging [101] and bail decisions [61, 105].”
Metadata
- Authors
- Lima, G., Grgić-Hlača, N., & Cha, M.
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-17
- OpenAlex
- W3126538636
- Cited by
- 70
- References
- 36
- FWCI
- 9.31
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.