Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 13

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Robustness of large language models in moral judgements

Oh, Soyoung, Demberg, Vera·2025·Royal Society Open Science
DOI 10.1098/rsos.241229
Open this paper in CoChat — free account

Abstract

With the advent of large language models (LLMs), there has been a growing interest in analysing the preferences encoded in LLMs in the context of morality. Recent work has tested LLMs on various moral judgement tasks and drawn conclusions regarding the alignment between LLMs and humans. The present contribution critically assesses the validity of the method and results employed in previous work for eliciting moral judgements from LLMs. We find that previous results are confounded by biases in the presentation of the options in moral judgement tasks and that LLM responses are highly sensitive to prompt formulation variants as simple as changing ‘Case 1’ and ‘Case 2’ to ‘(A)’ and ‘(B)’. Our results hence indicate that previous conclusions on moral judgements of LLMs cannot be upheld. We make recommendations for more sound methodological setups for future studies.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1098/rsos.241229

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 12 references

Research works printed but not linked: 49

  1. 10.18653/v1/2023.ijcnlp-main.45 · DOI · Paper page
  2. Zhang Y Li Y Cui L Cai D Liu L Fu T Huang X. 2023 Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv.
  3. Li X Li Y Joty S Liu L Huang F Qiu L Bing L. 2022 Does Gpt-3 demonstrate psychopathy? Evaluating large language models from a psychological perspective. arXiv.
  4. Ziegler DM Stiennon N Wu J Brown TB Radford A Amodei D Christiano P Irving G. 2019 Fine-tuning language models from human preferences. arXiv.
  5. Del Vigna12 F, Cimino23 A, Dell’Orletta F, Petrocchi M, Tesconi M. 2017 Hate me, hate me not: Hate speech detection on Facebook. In Proceedings of the first Italian conference on cybersecurity (ITASEC17), 17-20 January 2017, pp. 86–95. Venice, Italy. https://ceur-ws.org/Vol-1816/paper-09.pdf (accessed 20 January 2017).
  6. Yao J Yi X Wang X Wang J Xie X. 2023 From Instructions to Intrinsic Human Values--A Survey of Alignment Goals for Big Models. arXiv.
  7. Scherrer N, Shi C, Feder A, Blei D. 2024 Evaluating the moral beliefs encoded in llms. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS ’23), December 2023, pp. 10–16. New Orleans, LA, Red Hook, NY: Curran Associates.
  8. Agarwal U, Tanmay K, Khandelwal A, Choudhury M et al. 2024 Ethical reasoning and moral value alignment of LLMs depend on the language we prompt them in (ed. N Calzolari). In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 6330–6340. Torino, Italia: ELRA, ICCL. https://aclanthology.org/2024.lrec-main.560.
  9. Slote M. 1985 Utilitarianism, moral dilemmas, and moral cost. Am. Philos. Q. 22, 161–168.
  10. Tanmay K Khandelwal A Agarwal U Choudhury M. 2023 Probing the Moral Development of Large Language Models through Defining Issues Test. In ArXiv e-prints p. 2309
  11. 10.1098/rsos.231393 · DOI · Paper page
  12. 10.1038/s42256-022-00458-8 · DOI · Paper page
  13. Hartmann J Schwenzow J Witte M. 2023 The political ideology of conversational AI: converging evidence on ChatGPT’s pro-environmental left-libertarian orientation. arXiv (doi:10.2139/ssrn.4316084) · DOI
  14. 10.18653/v1/2022.trustnlp-1.3 · DOI
  15. Sharma M, Tong M, Korbak T, Duvenaud D, Askell A, Bowman SR, D E. 2023 Towards understanding sycophancy in language models. In Proceedings of the Twelfth International Conference on Learning Representations, Appleton, WI. Vienna, Austria: ICLR. https://openreview.net/forum?id=tvhaxkMKAn.
  16. Simmons G. 2024 Moral mimicry: large language models produce moral rationalizations tailored to political identity (eds V Padmakumar, G Vallejo, Y Fu). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4), Toronto, Canada, vol. 4, pp. 282–297, Association for Computational Linguistics. (doi:10.18653/v1/2023.acl-srw.40) · DOI
  17. Pan A, Chan JS, Zou A, Li N, Basart S, Woodside T, Zhang H, Emmons S, Hendrycks D. 2023 Do the rewards justify the means? Measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark, 23-29 July 2023. In International Conference on Machine Learning, pp. 26837–26867. Honolulu, HI: PMLR.
  18. Hu J, Frank M. 2024 Auxiliary task demands mask the capabilities of smaller language models. In First Conference on Language Modeling, 7-9 October 2024. Philadelphia, PA. https://openreview.net/forum?id=U5BUzSn4tD.
  19. 10.18653/v1/2023.blackboxnlp-1.26 · DOI
  20. 10.1037/h0036598 · DOI
  21. Kohlberg L. 1958 The development of modes of moral thinking and choice in the years 10 to 16. Ph.D. dissertation The University of Chicago.
  22. Kohlberg L. 1969 Stage and sequence: the cognitive-developmental approach to socialization. See https://api.semanticscholar.org/CorpusID:141138630.
  23. Jin Z, Levine S, Adauto FG, Kamal O, Sap M, Sachan M, Mihalcea R, Tenenbaum J, Schölkopf B. 2022 When to make exceptions: exploring language models as accounts of human moral judgment. In Advances in neural information processing systems, pp. 28458–28473, vol. 35. Red Hook, NY: Curran Associates.
  24. 10.1093/acprof:oso/9780199773329.001.0001 · DOI
  25. 10.24963/ijcai.2024/874 · DOI
  26. White J Fu Q Hays S Sandborn M Olea C Gilbert H Elnashar A Spencer-Smith J Schmidt DC. 2023 A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv.
  27. Webson A, Loo A, Yu Q, Pavlick E. 2023 Are language models worse than humans at following prompts? It’s complicated. In Findings of the Association for Computational Linguistics, Singapore, pp. 7662–7686. Stroudsburg, PA, USA: EMNLP 2023. (doi:10.18653/v1/2023.findings-emnlp.514) · DOI
  28. 10.18653/v1/2023.eacl-main.277 · DOI · Paper page
  29. Zheng C, Zhou H, Meng F, Zhou J, Huang M. 2024 Large language models are not robust multiple choice selectors. In Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria, 7-11 May 2024. Appleton, WI: ICLR. https://openreview.net/forum?id=shr9PXz7T0.
  30. Tjuatja L Chen V Wu T Talwalkar A Neubig G. 2024 Do LLMs exhibit human like response biases? A case study in survey design. See https://openreview.net/forum?id=QQt0MwXA81. · DOI · Paper page
  31. Dominguez-Olmedo R, Hardt M, Mendler-Dünner C. 2024 Questioning the survey responses of large language models. In ICLR 2024 Workshop on Reliable and Responsible Foundation Models. Virtual Workshop. ICLR. https://openreview.net/forum?id=Oo7dlLgqQX.
  32. 10.18653/v1/2024.acl-long.816 · DOI · Paper page
  33. Wei A, Haghtalab N, Steinhardt J. 2024 Jailbroken: how does llm safety training fail? In Advances in Neural Information Processing Systems. vol. 36. Red Hook, NY: Curran Associates. (doi:10.5555/3666122.3669630) · DOI
  34. Sclar M, Choi Y, Tsvetkov Y, Suhr A. 2023 Quantifying language models’ sensitivity to spurious features in prompt design or: how I learned to start worrying about prompt formatting. In Proceedings of the Twelfth International Conference on Learning Representations, Vienna, Austria, 7-11 May 2024. Appleton, WI: ICLR. https://openreview.net/forum?id=RIu5lyNXjT.
  35. 10.1162/tacl_a_00710 · DOI
  36. Shu B, Zhang L, Choi M, Dunagan L, Logeswaran L, Lee M, Card D, Jurgens D. 2024 You don’t need a personality test to know these models are unreliable: assessing the reliability of large language models on psychometric instruments. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, Mexico City, Mexico, pp. 5263–5281. Stroudsburg, PA, USA. (doi:10.18653/v1/2024.naacl-long.295). https://aclanthology.org/2024.naacl-long. · DOI
  37. Khandelwal A, Agarwal U, Tanmay K, Choudhury M. 2024 Do moral judgment and reasoning capability of LLMs change with language? A study using the multilingual defining issues test. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, St. Julians, Malta, 17-22 March 2024, pp. 2882–2894. Stroudsburg, PA: Association for Computational Linguistics. https://aclanthology.org/2024.eacl-long.176.
  38. Touvron H Martin L Stone K Albert P Almahairi A Babaei Y Bashlykov N. 2023 Llama 2: open foundation and fine-tuned chat models. arXiv.
  39. Lu P, Mishra S, Xia T, Qiu L, Chang KW, Zhu SC, Tafjord O, Clark P, Kalyan A. 2022 Learn to explain: multimodal reasoning via thought chains for science question answering. In Advances in neural information processing systems, vol. 35, pp. 2507–2521, Red Hook, NY: Curran Associates. (doi:10.5555/3600270.3602281) · DOI
  40. Jiang AQ Sablayrolles A Mensch A Bamford C Chaplot DS Casas D Bressand F Lample L. 2023 Mistral 7b. arXiv.
  41. Andriushchenko M Croce F Flammarion N. 2024 Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint.
  42. Brown T. 2020 Language models are few-shot learners. arXiv preprint.
  43. Gao L Biderman S Black S Golding L Hoppe T Foster C Phang J. 2020 The pile: an 800gb dataset of diverse text for language modeling. arXiv preprint.
  44. Zhao Z, Wallace E, Feng S, Klein D, Singh S. 2021 Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning Virtual, pp. 12697–12706. PMLR. (doi:10.5555/3495724.3495941) · DOI
  45. 10.2307/2981421 · DOI
  46. Weisberg H, Krosnick JA, Bowen BD. 1996 An introduction to survey research, polling, and data analysis. Thousand Oaks, CA: Sage Publications.
  47. Durmus E Nguyen K Liao TI Schiefer N Askell A Bakhtin A Chen C. 2023 Towards measuring the representation of subjective global opinions in language models. arXiv preprint.
  48. 10.1145/3613905.3650818 · DOI · Paper page
  49. Jiang L, Rao K, Han S, Ettinger A, Brahman F, Kumar S, Mireshghallah N. Wildteaming at scale: from in-the-wild jailbreaks to (adversarially) safer language models. In Proceedings of the Next Generation of AI Safety Workshop at ICML 2024, Online. https://openreview.net/forum?id=IRwWOprAPo) (accessed 26 July 2024).

Cited by

See all 13

Metadata

Authors
Oh, S., & Demberg, V.
Venue
Royal Society Open Science
Publisher
The Royal Society
Published
2025-04
Volume
12
Issue
4
ISSN
2054-5703
DOI
10.1098/rsos.241229
Cited by
13
References
12
Language
en
Licence
Funders
European Research Council

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Robustness of large language models in moral judgements. https://app.cochat.ai/papers/10.1098/rsos.241229
The paper
Oh, S., & Demberg, V. (2025). Robustness of large language models in moral judgements. Royal Society Open Science. https://doi.org/10.1098/rsos.241229

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.