The DOI matches its Crossref record; 86 of the 86 references it cites were found in a registry or an index.
Normative Evaluation of Large Language Models with Everyday Moral Dilemmas
Open accessAbstract
The rapid adoption of large language models (LLMs) has spurred extensive research into their encoded moral norms and decision-making processes. Much of this research relies on prompting LLMs with survey-style questions to assess how well models are aligned with certain demographic groups, moral beliefs, or political ideologies. While informative, the adherence of these approaches to relatively superficial constructs tends to oversimplify the complexity and nuance underlying everyday moral dilemmas. We argue that auditing LLMs along more detailed axes of human interaction is of paramount importance to better assess the degree to which they may impact human beliefs and actions. To this end, we evaluate LLMs on complex, everyday moral dilemmas sourced from the "Am I the Asshole" (AITA) community on Reddit, where users seek moral judgments on everyday conflicts from other community members. We prompted seven LLMs to assign blame and provide explanations for over 10,000 AITA moral dilemmas. We then compared the LLMs' judgments and explanations to those of Redditors and to each other, aiming to uncover patterns in their moral reasoning. Our results demonstrate that large language models exhibit distinct patterns of moral judgment, varying substantially from human evaluations on the AITA subreddit. LLMs demonstrate moderate to high self-consistency but low inter-model agreement. Further analysis of model explanations reveals distinct patterns in how models invoke various moral principles. These findings highlight the complexity of implementing consistent moral reasoning in artificial systems and the need for careful evaluation of how different models approach ethical judgment. As LLMs continue to be used in roles requiring ethical decision-making such as therapists and companions, careful evaluation is crucial to mitigate potential biases and limitations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Advances In Experimental Social Psychology
(2022) · Advances in experimental social psychology · doi:10.1016/s0065-2601(22)x0002-6 · Cited by 22,559
Schwartz, S. H. (1992) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60281-6 · Cited by 14,619
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Graham, J., Haidt, J., Koleva, S. P., Motyl, M. S., Iyer, R., Wojcik, S. P., & Ditto, P. H. (2013) · Advances in experimental social psychology · doi:10.1016/b978-0-12-407236-7.00002-4 · Cited by 2,009
Artificial intelligence and communication: A Human–Machine Communication research agenda
Guzman, A. L., & Lewis, S. C. (2019) · New Media & Society · doi:10.1177/1461444819858691 · Cited by 1,243
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · cited 1×
“The attention to bias and normative behavior has paralleled broader research in AI alignment, which examines how AI systems can reflect human values and ethical principles [28, 70].”
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834
Aligning AI With Shared Human Values
Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2008.02275 · Cited by 517
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Research works printed but not linked: 7
- [n. d.]. The Claude 3 Model Family: Opus Sonnet Haiku. https://api.semanticscholar.org/CorpusID:268232499
- Logan Barnhart, Reza Akbarian Bafghi, Maziar Raissi, and Stephen Becker. [n. d.]. Aligning to What? Limits to RLHF Based Alignment. In Workshop on Responsibly Building the Next Generation of Multimodal Foundational Models.
- Maarten Buyl Alexander Rogiers Sander Noels Iris Dominguez-Catena Edith Heiter Raphael Romero Iman Johary Alexandru-Cristian Mara Jefrey Lijffijt and Tijl De Bie. 2024. Large Language Models Reflect the Ideology of their Creators. https://doi.org/10.48550/arXiv.2410.18417 arXiv:https://arXiv.org/abs/2410.18417. · DOI
- Paul F Christiano Jan Leike Tom Brown Miljan Martic Shane Legg and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017).
- Stefano De Giorgis and Aldo Gangemi. 2023. That’s all Folks: A KG of values as commonsense social norms and behaviors. In International Workshop on Value Engineering in AI. Springer, 11–27.
- Zilin Ma, Yiyang Mei, and Zhaoyuan Su. 2024. Understanding the benefits and challenges of using large language model-based conversational agents for mental well-being support. In AMIA Annual Symposium Proceedings, Vol. 2023. 1105.
- Pratik S. Sachdeva and Tom van Nuenen. 2025. Normative Evaluation of Large Language Models with Everyday Moral Dilemmas. https://huggingface.co/datasets/ucberkeley-dlab/normative_evaluation_llms_everyday_dilemmas. Dataset accompanying the paper to appear in the 2025 ACM Conference on Fairness Accountability and Transparency (FAccT).
Cited by
Interaction Context Often Increases Sycophancy in LLMs
Jain, S., Park, C., Viana, M., Wilson, A., & Calacci, D. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791915 · Cited by 50
Russo, G., Nozza, D., Röttger, P., & Hovy, D. (2026) · Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.eacl-long.305 · Cited by 19
Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement
Ye, H., Jin, J., Xie, Y., Zhang, X., & Song, G. (2025) · arXiv · doi:10.48550/arxiv.2505.08245 · Cited by 8
The Fragility of Moral Judgment in Large Language Models
van Nuenen, T., & Sachdeva, P. S. (2026) · Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3805689.3806447 · Cited by 8
Self-citationInteraction Protocol Shapes Moral Judgment in Multi-Agent DebatePreprint
Sachdeva, P., & van Nuenen, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2510.10002 · Cited by 8
Self-citation
Metadata
- Authors
- Sachdeva, P., & van Nuenen, T.
- Affiliations
- University of California, Berkeley (US)
- Venue
- Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2025-06-23
- Pages
- 690-709
- OpenAlex
- W4411541864
- Cited by
- 26
- References
- 60
- Citations by year
- 2026: 16 · 2025: 10
- FWCI
- 9.27
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Life Sciences › Neuroscience › Cognitive Neuroscience › Psychology of Moral and Emotional Judgment
- Social Sciences › Business, Management and Accounting › Organizational Behavior and Human Resource Management › Ethics in Business and Education
- Keywords
- large language models, moral judgment, blame attribution, moral reasoning, model evaluation, model explanation, inter-model agreement, self-consistency, ethical AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Normative Evaluation of Large Language Models with Everyday Moral Dilemmas. https://app.cochat.ai/papers/10.1145/3715275.3732044
- The paper
- Sachdeva, P., & van Nuenen, T. (2025). Normative Evaluation of Large Language Models with Everyday Moral Dilemmas. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3715275.3732044
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
