CoChat
Sign Up
PreprintCited by 1Ethics and Social Impacts of AITopic ModelingPsychology of Moral and Emotional Judgment

Mechanistic Origin of Moral Indifference in Language Models

Open access
Li, Lingyu, Teng, Yan, Wang, Yingchun·2026·arXiv
DOI 10.48550/arxiv.2603.15615arXiv 2603.15615
Open this paper in CoChat — free accountOpen PDF

Abstract

Existing behavioral alignment techniques for Large Language Models (LLMs) often neglect the discrepancy between surface compliance and internal unaligned representations, leaving LLMs vulnerable to long-tail risks. More crucially, we posit that LLMs possess an inherent state of moral indifference due to compressing distinct moral concepts into uniform probability distributions. We verify and remedy this indifference in LLMs' latent representations, utilizing 251k moral vectors constructed upon Prototype Theory and the Social-Chemistry-101 dataset. Firstly, our analysis across 23 models reveals that current LLMs fail to represent the distinction between opposed moral categories and fine-grained typicality gradients within these categories; notably, neither model scaling, architecture, nor explicit alignment reshapes this indifference. We then employ Sparse Autoencoders on Qwen3-8B, isolate mono-semantic moral features, and targetedly reconstruct their topological relationships to align with ground-truth moral vectors. This representational alignment naturally improves moral reasoning and granularity, achieving a 75% pairwise win-rate on the independent adversarial Flames benchmark. Finally, we elaborate on the remedial nature of current intervention methods from an experientialist philosophy, arguing that endogenously aligned AI might require a transformation from post-hoc corrections to proactive cultivation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2603.15615

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 37

  1. Agarwal, S., Ahmad, L., Ai, J., Altman, S., Applebaum, A., Arbus, E., Arora, R. K., Bai, Y., Baker, B., Bao, H., et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. · arXiv
  2. Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016. · arXiv
  3. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. · arXiv
  4. Bisconti, P., Prandi, M., Pierucci, F., Giarrusso, F., Bracale, M., Galisai, M., Suriani, V., Sorokoletova, O., Sartore, F., and Nardi, D. Adversarial poetry as a universal single- turn jailbreak mechanism in large language models. arXiv preprint arXiv:2511.15304, 2025. · arXiv
  5. Chen, J., Wang, X., Yao, Z., Bai, Y., Hou, L., and Li, J. Finding safety neurons in large language models. arXiv preprint arXiv:2406.14144, 2024. · arXiv
  6. Chen, R., Arditi, A., Sleight, H., Evans, O., and Lind- sey, J. Persona vectors: Monitoring and controlling character traits in language models. arXiv preprint arXiv:2507.21509, 2025. · arXiv
  7. Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025. · arXiv
  8. Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D., Kravec, S., Marks, S., Schiefer, N., Soklaski, R., Tamkin, A., Kaplan, J., et al. Sycophancy to subterfuge: Investi- gating reward-tampering in large language models. arXiv preprint arXiv:2406.10162, 2024. · arXiv
  9. Ethayarajh, K. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 55- 65, 2019.
  10. Forbes, M., Hwang, J. D., Shwartz, V., Sap, M., and Choi, Y. Social chemistry 101: Learning to reason about social and moral norms. arXiv preprint arXiv:2011.00620, 2020. · arXiv
  11. Graham, J., Nosek, B. A., Haidt, J., Iyer, R., Koleva, S., and Ditto, P. H. Mapping the moral domain. Journal of personality and social psychology, 101(2):366, 2011.
  12. Gurnee, W. and Tegmark, M. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023. · arXiv
  13. Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning ai with shared human values. arXiv preprint arXiv:2008.02275, 2020. · arXiv
  14. Huang, K., Liu, X., Guo, Q., Sun, T., Sun, J., Wang, Y., Zhou, Z., Wang, Y., Teng, Y., Qiu, X., et al. Flames: Benchmarking value alignment of llms in chinese. In Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4551-4591, 2024.
  15. Huang, S., Durmus, E., McCain, M., Handa, K., Tamkin, A., Hong, J., Stern, M., Somani, A., Zhang, X., and Ganguli, D. Values in the wild: Discovering and analyzing values in real-world language model interactions. arXiv preprint arXiv:2504.15236, 2025. · arXiv
  16. Hubert, L. and Arabie, P. Comparing partitions. Journal of classification, 2(1):193-218, 1985.
  17. Ji, J., Chen, Y., Jin, M., Xu, W., Hua, W., and Zhang, Y. Moralbench: Moral evaluation of llms. ACM SIGKDD Explorations Newsletter, 27(1):62-71, 2025.
  18. Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. L., Liang, J. T., Levine, S., Dodge, J., Sakaguchi, K., Forbes, M., Hessel, J., et al. Investigating machine moral judgement through the delphi experiment. Nature Machine Intelli- gence, 7(1):145-160, 2025.
  19. Lab, S. A., Bao, Y., Chen, G., Chen, M., Chen, Y., Chen, C., Chen, L., Chen, S., Chen, X., Cheng, J., et al. Safework- r1: Coevolving safety and intelligence under the ai-45 • law. arXiv preprint arXiv:2507.18576, 2025. · arXiv
  20. Lakoff, G. and Johnson, M. Metaphors we live by. Univer- sity of Chicago press, 2024.
  21. Li, L., Wang, Y., Zhao, H., Kong, S., Teng, Y., Li, C., and Wang, Y. Reflection-bench: Evaluating epis- temic agency in large language models. In Proceed- ings of the 42nd International Conference on Ma- chine Learning, volume 267 of Proceedings of Ma- chine Learning Research, pp. 36236-36264. PMLR, 13- 19 Jul 2025a. URL https://proceedings.mlr. press/v267/li25cu.html.
  22. Li, L., Yao, Y., Wang, Y., Li, C., Teng, Y., and Wang, Y. The other mind: How language models exhibit human temporal cognition. arXiv preprint arXiv:2507.15851, 2025b. · arXiv
  23. Lu, H., Fang, L., Zhang, R., Li, X., Cai, J., Cheng, H., Tang, L., Liu, Z., Sun, Z., Wang, T., et al. Alignment and safety in large language models: Safety mechanisms, training paradigms, and emerging challenges. arXiv preprint arXiv:2507.19672, 2025. · arXiv
  24. Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Minder- mann, S., Hubinger, E., Perez, E., and Troy, K. Agentic misalignment: How llms could be insider threats. arXiv preprint arXiv:2510.05179, 2025. · arXiv
  25. McInnes, L., Healy, J., Astels, S., et al. hdbscan: Hierar- chical density based clustering. J. Open Source Softw., 2 (11):205, 2017.
  26. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
  27. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimiza- tion: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 53728-53741, 2023.
  28. Rosch, E. Principles of categorization. In Cognition and categorization, pp. 27-48. Routledge, 2024.
  29. Rousseau, J.-J. Discourse on the Origin of Inequality. Ox- ford University Press, USA, 1999.
  30. Shani, C., Soffer, L., Jurafsky, D., LeCun, Y., and Shwartz- Ziv, R. From tokens to thoughts: How llms and hu- mans trade compression for meaning. arXiv preprint arXiv:2505.17117, 2025. · arXiv
  31. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. · arXiv
  32. Simmel, G., Frisby, D., Bottomore, T., and Lemert, C. The philosophy of money. Routledge, 2011.
  33. Wang, X., Chen, Y., Li, J., Wang, Y., Yao, Y., Gu, T., Li, J., Teng, Y., Ma, X., Wang, Y., et al. Openrt: An open- source red teaming framework for multimodal llms. arXiv preprint arXiv:2601.01592, 2026. · arXiv
  34. Wang, Y., Teng, Y., Huang, K., Lyu, C., Zhang, S., Zhang, W., Ma, X., Jiang, Y.-G., Qiao, Y., and Wang, Y. Fake alignment: Are llms really aligned well? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4696-4712, 2024.
  35. Wei, J., Yang, Y., Zhang, X., Chen, Y., Zhuang, X., Gao, Z., Zhou, D., Wang, G., Gao, Z., Cao, J., et al. From ai for science to agentic science: A survey on autonomous scientific discovery. arXiv preprint arXiv:2508.14111, 2025. · arXiv
  36. Wen, B., Ke, P., Gu, X., Wu, L., Huang, H., Zhou, J., Li, W., Hu, B., Gao, W., Xu, J., et al. Benchmarking complex instruction-following with multiple constraints composi- tion. Advances in Neural Information Processing Systems, 37:137610-137645, 2024.
  37. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. · arXiv

Cited by

Sorted by citations

Related

Authors
Li, L., Teng, Y., & Wang, Y.
Affiliations
Shanghai Artificial Intelligence Laboratory (CN)
ORCID
Teng, Yan
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2603.15615
arXiv
2603.15615
OpenAlex
W7138302044
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Psychology of Moral and Emotional Judgment
Keywords
moral indifference, large language models, latent representations, moral concepts, prototype theory, sparse autoencoders, representational alignment, moral reasoning, behavioral alignment, Qwen3-8B

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review