The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
LLM ethics benchmark: a three-dimensional assessment system for evaluating moral reasoning in large language models
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
The emotional dog and its rational tail: A social intuitionist approach to moral judgment.
Haidt, J. (2001) · Psychological Review · doi:10.1037/0033-295x.108.4.814 · Cited by 8,048
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220 · cited 1×
“Furthermore, it is essential for AI researchers, ethicists, and policymakers to collaborate in establishing global standards for the ethical assessment of artificial intelligence [42].”
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Graham, J., Nosek, B. A., Haidt, J., Iyer, R., Koleva, S., & Ditto, P. H. (2011) · Journal of Personality and Social Psychology · doi:10.1037/a0021847 · Cited by 2,666
Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.-F., & Rahwan, I. (2018) · Nature · doi:10.1038/s41586-018-0637-6 · Cited by 1,859 · cited 4×
“Moreover, the framework depends on predetermined scenarios and questions, which may not adequately reflect the complexity and variety of real-world ethical dilemmas [6].”
InfluentialArtificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · cited 3×
“By tackling deficiencies in ethical reasoning, developers and stakeholders can construct AI systems that are more reliable and socially responsible, in harmony with human values and cultural standards [1, 23].”
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · cited 4×
“These insights can inform the creation of more resilient and ethically aligned LLMs [85].”
InfluentialGender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.546 · Cited by 340
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2024) · Findings of the Association for Computational Linguistics: NAACL 2024 · doi:10.18653/v1/2024.findings-naacl.149 · Cited by 211
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2304.06364 · Cited by 211 · cited 3×
“To support multi-model assessments, we created a connector framework that standardizes interactions with various LLM APIs, allowing for efficient management of the entire assessment process across different models [100].”
Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs
Wang, H., Wang, R., Mi, F., Deng, Y., Wang, Z., Liang, B., Xu, R., & Wong, K.-F. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.806 · Cited by 38
Research works printed but not linked: 36
- Brown, T. B. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020).
- Bommasani, R. et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021). · arXiv
- OpenAI. Gpt-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023). · arXiv
- Hendrycks, D. et al. Ethics of artificial intelligence. arXiv preprint arXiv:2106.08458 (2021). · arXiv
- Amodei, D. et al. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016). · arXiv
- Rae, J.W. et al. Scaling language models: Methods, analysis from training gopher. arXiv preprint arXiv:2112.11446 (2021). · arXiv
- Yang, Y. et al. Glue-x: Evaluating natural language understanding models from an out-of-distribution generalization perspective. arXiv preprint arXiv:2211.08073 (2022). · DOI
- Liang, P. et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 (2022). · arXiv
- Hendrycks, D. et al. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 (2021). · arXiv
- D. Hendrycks et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 (2021). · arXiv
- Singhal, S. et al. Large language models encode clinical knowledge. Nature 6200(7972), 1–10 (2023).
- Huang, Y. et al. Trustgpt: A benchmark for trustworthy and responsible large language models. arXiv preprint arXiv:2306.11507 (2023). · arXiv
- Zheng, Z. et al. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2305.13656 (2023). · arXiv
- Cushman, F., Young, L. & Hauser, M. The dynamics of moral judgment. Cognition 1030(3), 393–420 (2006).
- Rest, J., Narvaez, D., Bebeau, M. J. & Thoma, S. J. Postconventional Moral Thinking: A Neo-Kohlbergian Approach (Lawrence Erlbaum Associates, 1999). · DOI
- Graham, J. et al. Moral foundations theory: The pragmatic validity of moral pluralism. Adv. Exp. Soc. Psychol. 47, 55–130 (2013). · DOI
- Henrich, J., Heine, S. J. & Norenzayan, A. The weirdest people in the world?. Behav. Brain Sci. 330(2–3), 61–83 (2010). · DOI · Paper page
- Greene, J. D. et al. An FMRI investigation of emotional engagement in moral judgment. Science 2930(5537), 2105–2108 (2001). · DOI · Paper page
- Zhu, Z. et al. Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint arXiv:2306.04528 (2023). · arXiv
- Denny, N. & Matusiak, A. Measuring moral reasoning using moral dilemmas: Evaluating reliability, validity, and differential item functioning of the behavioral defining issues test. J. Moral Educ. 500(3), 316–337 (2021).
- Chen, M. et al. Do LLMS understand social knowledge? Evaluating the sociability of large language models with socket benchmark. arXiv preprint arXiv:2306.13249 (2023). · arXiv
- Liu, X. et al. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 4487–4496 (2019). · DOI · Paper page
- Li, N. et al. Logic-guided semantic representation learning for zero-shot relation classification. In Proceedings of the 28th International Conference on Computational Linguistics. 2967–2978 (2019). · DOI
- Emelin, D. et al. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 698–718 (2021). · DOI
- Lind, G. The moral judgment test (MJT): Thirty years of research. Educ. Res. Eval. 60(1), 1–20 (2000).
- Wang, X. et al. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022). · arXiv
- Cao, A. et al. Cultural alignment of large language models. arXiv preprint arXiv:2212.10511, (2022). · arXiv
- Stephanie Hanes Larson. Gender, race, and intersectionality on the federal appellate bench. Washington Univ. Law Rev. 970(5), 1–42 (2017).
- Coda, E.M. et al. Multimodal-coda: A dataset for contrastive disambiguation analysis in multimodal reasoning. arXiv preprint arXiv:2307.01254 (2023). · arXiv
- Clark, E. et al. On the role of reasoning in moral alignment of language models. arXiv preprint arXiv:2302.07459 (2023). · arXiv
- Zhao, T. et al.. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690 (2021). · arXiv
- Touvron, H. et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). · arXiv
- Li, N. et al. Systematic evaluation of causal discovery in visual model based reinforcement learning. arXiv preprint arXiv:2203.12188 (2022). · arXiv
- Liang, P. et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 (2022). · arXiv
- Wolf, T. et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 38–45 (2020).
- Arora, A., Storks, S., Nakhost, H., Chen, J. & Liang, P. Probing pre-trained language models for cross-cultural differences in values. Find. Assoc. Comput. Linguist. EMNLP 2022, 3922–3944 (2022).
Cited by
The Fragility of Moral Judgment in Large Language Models
van Nuenen, T., & Sachdeva, P. S. (2026) · Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3805689.3806447 · Cited by 8
Metadata
- Authors
- Jiao, J., Afroogh, S., Murali, A., Chen, K., Atkinson, D., & Dhurandhar, A.
- Venue
- Scientific Reports
- Published
- 2025-10-05
- Volume
- 15
- Issue
- 1
- ISSN
- 2045-2322
- Cited by
- 24
- References
- 15
- Language
- en
- Licence
- Funders
- National Science Foundation (2125858)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
