CoChat
Sign Up
Cited by 36Ethics and Social Impacts of AIEthics in Business and EducationPhilosophical Ethics and Theory

Ethical and methodological challenges in building morally informed AI systems

Open access
Hagendorff, Thilo, Danks, David·2023·AI and Ethics
DOI 10.1007/s43681-022-00188-y
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent progress in large language models has led to applications that can (at least) simulate possession of full moral agency due to their capacity to report context-sensitive moral assessments in open-domain conversations. However, automating moral decision-making faces several methodological as well as ethical challenges. They arise in the fields of bias mitigation, missing ground truth for moral “correctness”, effects of bounded ethicality in machines, changes in moral norms over time, risks of using morally informed AI systems as actual advice, as well as societal implications an increasing importance of algorithmic moral decision-making would have. This paper comments on all these challenges and provides critical considerations for future research on full artificial moral agency. Importantly, some of the adduced challenges can be met by more careful technology design, but others necessarily require engagement with core problems of meta-ethics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s43681-022-00188-y

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI models trained on data imbued with moral and scientific values could produce outputs that are not widely recognized as morally correct because the data reflects the moral biases of its developers.

    “The model could produce outputs that are not widely recognized as morally correct because it has been trained on data that reflects the moral biases of its developers [112].”

    From Autonomous artificial intelligence, scientific research, and human values (2026) · cites this paper as [112]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 16

  1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin, I.: Attention is all you need. arXiv 1-15 (2017)
  2. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. arXiv 1-75 (2020)
  3. Fedus, W., Zoph, B., Shazeer, N.: Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. arXiv 1-31 (2021)
  4. Liu, Y., Ott, M., Goyal, N., Jingfei, DU., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv 1-13 (2019)
  5. Devlin, J., Chang, M-W., Lee, K., Toutanova, K.: BERT: Pre- training of deep bidirectional transformers for language under- standing. arXiv (2019)
  6. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le Q.V.: XLNet: Generalized autoregressive pretraining for language understanding. arXiv 1-18 (2020)
  7. Botzer, N., Gu, S., Weninger, T.: Analysis of moral judgement on reddit. arXiv 1-9 (2021)
  8. Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., Steinhardt, J.: Aligning AI with shared human values. arXiv 1-29 (2021)
  9. Schramowski, P., Turan, C., Andersen, N., Rothkopf, C., Kersting, K.: Language models have a moral dimension. arXiv 1-19 (2021)
  10. Jiang, L., Hwang, J.D., Bhagavatula, C., Le Bras, R., Forbes, M., Borchardt, J., Liang, J., Etzioni, O., Sap, M., Choi, Y.: Delphi: towards machine ethics and norms. arXiv 1-42 (2021)
  11. Lehman, J., Clune, J., Misevic, D., Adami, C., Altenberg, L., Beaulieu, J., Bentley, P.J., Bernard, S., Beslon, G., Bryson, D.M., Chrabaszcz, P., Cheney, N., Cully, A., Doncieux, S., Dyer, F.C., Ellefsen, K.O., Feldt, R., Fischer, S., Forrest, S., Frénoy, A., Gagné, C., Le Goff, L., Grabowski, L.M., Hodjat, B., Hutter, F., Keller, L., Knibbe, C., Krcah, P., Lenski, R.E., Lipson, H., Mac- Curdy, R., Maestre, C., Miikkulainen, R., Mitri, S., Moriarty, D.E., Mouret, J.-B., Nguyen, A., Ofria, C., Parizeau, M., Parsons, D., Pennock, R.T., Punch, W.F., Ray, T.S., Schoenauer, M., Shulte, E., Sims, K., Stanley, K.O., Taddei, F., Tarapore, D., Thibault, S., Weimer, W., Watson, R., Yosinski, J.: The surprising creativity of digital evolution: a collection of anecdotes from the evolutionary computation and artificial life research communities. arXiv 1-32 (2018)
  12. Elgammal, A., Liu, B., Elhoseiny, M., Mazzone, M.: CAN: crea- tive adversarial networks, generating "Art" by learning about styles and deviating from style norms. arXiv 1-22 (2017)
  13. Bazerman, M.H., Tenbrunsel, A.E.: Blind spots: why we fail to do what's right and what to do about it. Princeton University Press, Princeton (2011)
  14. Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., Arx, S.V., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chat- terji, N., Chen, A., Creel, K., Davis, J.Q., Demszky, D., Dona- hue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel, K., Goodman, N., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D.E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keel- ing, G., Khani, F., Khattab, O., Koh, P.W., Krass, M., Krishna, R., Kuditipudi, R., Kumar, A., Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li, X.L., Li, X., Ma, T., Malik, A., Man- ning, C.D., Mirchandani, S., Mitchell, E., Munyikwa, Z., Nair, S., Narayan, A., Narayanan, D., Newman, B., Nie, A., Niebles, J.C., Nilforoshan, H., Nyarko, J., Ogut, G., Orr, L., Papadimitriou, I., Park, J.S., Piech, C., Portelance, E., Potts, C., Raghunathan, A., Reich, R., Ren, H., Rong, F., Roohani, Y., Ruiz, C., Ryan, J., Ré, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K., Tamkin, A., Taori, R., Thomas, A.W., Tramèr, F., Wang, R.E., Wang, W., Wu, B., Wu, J., Wu, Y., Xie, S.M., Yasunaga, M., You, J., Zaharia, M., Zhang, M., Zhang, T., Zhang, X., Zhang, Y., Zheng, L., Zhou, K., Liang, P.: On the opportunities and risks of foundation models. arXiv 1-212 (2021)
  15. Talat, Z., Blix, H., Valvoda, J., Ganesh, M.I., Cotterell, R., Wil- liams, A.: A word on machine ethics: a response to Jiang et al. (2021). arXiv 1-11 (2021)
  16. Asch, S.: Effects of group pressure upon the modification and distortion of judgment. In: Guetzkow, H.S. (Ed.) Groups, leader- ship and men: research in human relations, pp. 177-190.

Cited by

Sorted by citations

Related

Authors
Hagendorff, T., & Danks, D.
Affiliations
  • University of Tübingen (DE)
  • University of California San Diego (US)
Corresponding author
Hagendorff, Thilo
ORCID
Venue
AI and Ethics
Publisher
Springer Science and Business Media LLC
Published
2022-06-22
Volume
3
Issue
2
Pages
553-566
ISSN
  • 2730-5953 (ISSN-L)
  • 2730-5961
DOI
10.1007/s43681-022-00188-y
OpenAlex
W4283385132
Cited by
36
References
42
Citations by year
2026: 8 · 2025: 10 · 2024: 16 · 2023: 1 · 2022: 1
FWCI
8.12
Type
journal article
Language
en
Open access
Licence
Locations
AI and Ethics (publishedVersion, cc-by)
Funders
  • Deutsche Forschungsgemeinschaft (Reference Number EXC 2064/1–Project ID 390727645)
  • Eberhard Karls Universität Tübingen
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Decision Sciences › Information Systems and Management › Ethics in Business and Education
  • Social Sciences › Arts and Humanities › Philosophy › Philosophical Ethics and Theory
Keywords
large language models, moral reasoning, bias mitigation, moral correctness, metaethics, AI advice, societal impact of AI, technology design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review