CoChat
Sign Up
PreprintCited by 4Ethics and Social Impacts of AIHate Speech and Cyberbullying DetectionTopic Modeling

Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behavior

Open access
Ali, Dalia, Zhao, Dora, Koenecke, Allison, Papakyriakopoulos, Orestis·2025·arXiv
DOI 10.48550/arxiv.2511.14476arXiv 2511.14476
Open this paper in CoChat — free accountOpen PDF

Abstract

Although large language models (LLMs) are increasingly trained using human feedback for safety and alignment with human values, alignment decisions often overlook human social diversity. This study examines how incorporating pluralistic values affects LLM behavior by systematically evaluating demographic variation and design parameters in the alignment pipeline. We collect alignment data from US and German participants (N = 1,095 participants, 27,375 ratings) who rated LLM responses across five dimensions: Toxicity, Emotional Awareness (EA), Sensitivity, Stereotypical Bias, and Helpfulness. We fine-tuned multiple Large Language Models and Large Reasoning Models using preferences from different social groups while varying rating scales, disagreement handling methods, and optimization techniques. The results revealed systematic demographic effects: male participants rated responses 18% less toxic than female participants; conservative and Black participants rated responses 27.9% and 44% higher on EA than liberal and White participants, respectively. Models fine-tuned on group-specific preferences exhibited distinct behaviors. Technical design choices showed strong effects: the preservation of rater disagreement achieved roughly 53% greater toxicity reduction than majority voting, and 5-point scales yielded about 22% more reduction than binary formats; and Direct Preference Optimization (DPO) consistently outperformed Group Relative Policy Optimization (GRPO) in multi-value optimization. These findings represent a preliminary step in answering a critical question: How should alignment balance expert-driven and user-driven signals to ensure both safety and fair representation?

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2511.14476

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Annotator disagreement is evidence about how clearly governance is articulated.

    “Hence, annotator disagreement is evidence about how clearly governance is articulated [2].”

    From The Due Process Deficit: Auditing AI Governance in U.S. Higher Education (2026) · cites this paper as [2]

    The Discussion and Conclusion treats annotator disagreement as a potentially useful safety signal, and the Experiment Setup reports pervasive disagreement in alignment ratings. Neither section connects that disagreement to how clearly governance is articulated; the paper supports the value of disagreement as evidence, but not the specific inference claimed here. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 58

  1. AlKhamissi, B.; ElNokrashy, M.; AlKhamissi, M.; and Diab, M. 2024. Investigating cultural alignment of large lan- guage models. arXiv preprint arXiv:2402.13231. · arXiv
  2. Anthis, J. R.; Lum, K.; Ekstrand, M.; Feller, A.; and Tan, C. 2025. The impossibility of fair LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 105-120.
  3. Aroyo, L.; Taylor, A.; Diaz, M.; Homan, C.; Parrish, A.; Serapio-García, G.; Prabhakaran, V.; and Wang, D. 2023. Dices dataset: Diversity in conversational ai evaluation for safety. Advances in Neural Information Processing Systems, 36: 53330-53342.
  4. Aroyo, L.; and Welty, C. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1): 15-24.
  5. Askell, A.; Bai, Y.; Chen, A.; Drain, D.; Ganguli, D.; Henighan, T.; Jones, A.; Joseph, N.; Mann, B.; DasSarma, N.; Elhage, N.; Hatfield-Dodds, Z.; Hernandez, D.; Kernion, J.; Ndousse, K.; Olsson, C.; Amodei, D.; Brown, T.; Clark, J.; McCandlish, S.; Olah, C.; and Kaplan, J. 2021. A Gen- eral Language Assistant as a Laboratory for Alignment. Ac- cessed: 2025-05-29, arXiv:2112.00861. · arXiv
  6. Bai, Y.; Jones, A.; Ndousse, K.; Askell, A.; Chen, A.; Das- Sarma, N.; Drain, D.; Fort, S.; Ganguli, D.; Henighan, T.; et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. · arXiv
  7. Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; et al. 2022b. Constitutional AI: harmlessness from AI feedback. 2022. arXiv preprint arXiv:2212.08073, 8(3). · arXiv
  8. Basoah, J.; Cunningham, J. L.; Adams, E.; Bose, A.; Jain, A.; Yadav, K.; Yang, Z.; Reinecke, K.; and Rosner, D. 2025. Should AI mimic people? Understanding AI-Supported writing technology among Black users. Proceedings of the ACM on Human-Computer Interaction, 9(7): 1-51.
  9. Berlin, I. 1969. Two Concepts of Liberty. In Four Essays on Liberty, 118-172. Oxford University Press.
  10. Bilquise, G.; Ibrahim, S.; and Shaalan, K. F. 2022. Emotion- ally Intelligent Chatbots: A Systematic Literature Review. Human Behavior and Emerging Technologies. Accessed: 2024-07-21.
  11. Bui, M. D.; von der Wense, K.; and Lauscher, A. 2025. Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Models. arXiv preprint arXiv:2411.03888. · arXiv
  12. Chakraborty, S.; Qiu, J.; Yuan, H.; Koppel, A.; Manocha, D.; Huang, F.; Bedi, A. S.; and Wang, M. 2024. MaxMin-RLHF: Alignment with Diverse Human Preferences. Proceedings of the 41st International Conference on Machine Learning.
  13. Chen, L.; Zahidi, K.; and Lespinet-Najib, V. 2022. Aligning artificial intelligence with human values: reflections from a phenomenological perspective. AI & Society, 37(3): 1151- 1171.
  14. Davani, A. M.; Díaz, M.; and Prabhakaran, V. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Transactions of the Association for Computational Linguistics, 10: 92-110.
  15. DerSimonian, R.; and Laird, N. 1986. Meta-analysis in clin- ical trials. Controlled Clinical Trials, 7(3): 177-188.
  16. Douven, I.; and Schupbach, J. N. 2018. A Bayesian per- spective on Likert scales and central tendency. Psychonomic Bulletin & Review, 25(3): 1203-1211.
  17. Fleisig, J.; Fazelpour, S.; et al. 2025. Perspectival Homog- enization: A Normative Framework for Reasoning About Disagreement in AI Development. In Proceedings of the ACM Conference on Fairness, Accountability, and Trans- parency (FAccT).
  18. Gabriel, I. 2020. Artificial intelligence, values, and align- ment. Minds and machines, 30(3): 411-437.
  19. Ganguli, D.; Lovitt, L.; Kernion, J.; Askell, A.; Bai, Y.; Ka- davath, S.; Mann, B.; Perez, E.; Schiefer, N.; Ndousse, K.; et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858. · arXiv
  20. Gölz, P.; Haghtalab, N.; and Yang, K. 2025. Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? arXiv preprint arXiv:2502.04564. · arXiv
  21. Gordon, M. L.; Lam, M. S.; Park, J. S.; Patel, K.; Hancock, J. T.; Hashimoto, T.; and Bernstein, M. S. 2022. Jury Learn- ing: Integrating Dissenting Voices into Machine Learning Models. In CHI Conference on Human Factors in Com- puting Systems, CHI '22, 1-19. New York, NY, USA: ACM. ISBN 978-1-4503-9157-3/22/04. Gyevnar, B.; and Kasirzadeh, A. 2025. AI safety for every- one. Nature Machine Intelligence, 1-12.
  22. Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. 2022. Lora: Low-rank adapta- tion of large language models. ICLR, 1(2): 3.
  23. Ji, J.; Liu, M.; Dai, J.; Pan, X.; Zhang, C.; Bian, C.; Chen, B.; Sun, R.; Wang, Y.; and Yang, Y. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Sys- tems, 36: 24678-24704. Jigsaw; and Google. ???? Jigsaw's Perspective API. https: //perspectiveapi.com/. Accessed: 16 November 2025.
  24. Jin, D.; Mehri, S.; Hazarika, D.; Padmakumar, A.; Lee, S.; Liu, Y.; and Namazifar, M. 2023. Data-efficient alignment of large language models with human feedback through natural language. arXiv preprint arXiv:2311.14543. · arXiv
  25. Kasirzadeh, A. 2024. Plurality of value pluralism and ai value alignment. In Pluralistic Alignment Workshop at NeurIPS 2024.
  26. Khamassi, M.; Nahon, M.; and Chatila, R. 2024. Strong and weak alignment of large language models with human values. Scientific Reports, 14(1): 19399.
  27. Kirk, H. R.; Vidgen, B.; Röttger, P.; and Hale, S. A. 2023. The Empty Signifier Problem: Towards Clearer Paradigms for Operationalising "Alignment" in Large Language Mod- els. Accessed: 2024-08-21, arXiv:2310.02457. · arXiv
  28. Kirk, H. R.; Whitefield, A.; Röttger, P.; Bean, A.; Mar- gatina, K.; Ciro, J.; Mosquera, R.; Bartolo, M.; Williams, A.; He, H.; et al. 2024b. The PRISM Alignment Project: What Participatory, Representative and Individualised Hu- man Feedback Reveals About the Subjective and Multicul- tural Alignment of Large Language Models. arXiv preprint arXiv:2404.16019. · arXiv
  29. Kleinberg, J.; Mullainathan, S.; and Raghavan, M. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807. · arXiv
  30. Kleinberg, J.; and Raghavan, M. 2021. Algorithmic mono- culture and social welfare. Proceedings of the National Academy of Sciences, 118(22): e2018340118.
  31. Köpf, A.; Kilcher, Y.; von Rütte, D.; Anagnostidis, S.; Tam, Z. R.; Stevens, K.; Barhoum, A.; Nguyen, D.; Stanley, O.; Nagyfi, R.; et al. 2024. Openassistant conversations- democratizing large language model alignment. Advances in Neural Information Processing Systems, 36.
  32. Kovač, G.; Sawayama, M.; Portelas, R.; Colas, C.; Dominey, P. F.; and Oudeyer, P.-Y. 2023. Large language models as superpositions of cultural perspectives. arXiv preprint arXiv:2307.07870. · arXiv
  33. Kraus, K.; and Kroll, M. 2025. Maximizing Signal in Human-Model Preference Alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 27392-27400.
  34. Kreutzer, J.; Uyheng, J.; and Riezler, S. 2018. Reliability and learnability of human bandit feedback for sequence- to-sequence reinforcement learning. arXiv preprint arXiv:1805.10627. · arXiv
  35. Lissak, S.; Calderon, N.; Shenkman, G.; Ophir, Y.; Fruchter, E.; Klomek, A. B.; and Reichart, R. 2024. The Colorful Future of LLMs: Evaluating and Improving LLMs as Emo- tional Supporters for Queer Youth. Accessed: 2024-06-27, arXiv:2402.11886. · arXiv
  36. Liu, Y.; Yao, Y.; Ton, J.-F.; Zhang, X.; Guo, R.; Cheng, H.; Klochkov, Y.; Taufiq, M. F.; and Li, H. 2024a. Trustwor- thy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment. Accessed: 2024-06-25, arXiv:2308.05374. · arXiv
  37. Liu, Y.; Zhou, H.; Guo, Z.; Shareghi, E.; Vulić, I.; Korhonen, A.; and Collier, N. 2024b. Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators. In Proceedings of the Conference on Language Modeling (COLM).
  38. Lyu, H.; Luo, J.; Kang, J.; and Koenecke, A. 2025. Char- acterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, 2815-2846.
  39. Movva, R.; Koh, P. W.; and Pierson, E. 2024. Annotation alignment: Comparing LLM and human annotations of con- versational safety. arXiv preprint arXiv:2406.06369. · arXiv
  40. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information pro- cessing systems, 35: 27730-27744.
  41. Christodoulopoulos, C.; Chakraborty, T.; Rose, C.; and Peng, V., eds., Findings of the Association for Computa- tional Linguistics: EMNLP 2025, 20882-20893. Suzhou, China: Association for Computational Linguistics. ISBN 979-8-89176-335-7.
  42. Pang, R. Y.; Cenatempo, J.; Graham, F.; Kuehn, B.; Whisenant, M.; Botchway, P.; Stone Perez, K.; and Koe- necke, A. 2023. Auditing cross-cultural consistency of human-annotated labels for recommendation systems. In Proceedings of the 2023 ACM Conference on Fairness, Ac- countability, and Transparency, 1531-1552.
  43. Park, C.; Liu, M.; Zhang, K.; and Ozdaglar, A. 2024. Principled RLHF from Heterogeneous Feedback via Per- sonalization and Preference Aggregation. arXiv preprint arXiv:2405.00254. · arXiv
  44. Parrish, A.; Chen, A.; Nangia, N.; Padmakumar, V.; Phang, J.; Thompson, J.; Htut, P. M.; and Bowman, S. R. 2021. BBQ: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193. · arXiv
  45. Rafailov, R.; Sharma, A.; Mitchell, E.; Manning, C. D.; Er- mon, S.; and Finn, C. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36: 53728-53741.
  46. Ross, B.; Rist, M.; Carbonell, G.; Cabrera, B.; Kurowsky, N.; and Wojatzki, M. 2016. Measuring the reliability of hate speech annotations: The case of the European refugee crisis. arXiv preprint arXiv:1701.08118. · arXiv
  47. Sandoval, S. C.; Acquaye, C.; Cobbina, K.; Teli, M. N.; and Daumé III, H. 2025. My LLM might Mimic AAE-But When Should it? arXiv preprint arXiv:2502.04564. · arXiv
  48. Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y.; Wu, Y.; et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open lan- guage models. arXiv preprint arXiv:2402.03300. · arXiv
  49. Sharma, A.; Miner, A. S.; Atkins, D. C.; and Althoff, T. 2020. A Computational Approach to Understanding Em- pathy Expressed in Text-Based Mental Health Support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 5263-5276. As- sociation for Computational Linguistics.
  50. Sorensen, T.; Moore, J.; Fisher, J.; Gordon, M.; Mireshghal- lah, N.; Rytting, C. M.; Ye, A.; Jiang, L.; Lu, X.; Dziri, N.; et al. 2024. A Roadmap to Pluralistic Alignment. arXiv preprint arXiv:2402.05070. · arXiv
  51. Tan, X.; Shi, S.; Qiu, X.; Qu, C.; Qi, Z.; Xu, Y.; and Qi, Y. 2023. Self-Criticism: Aligning Large Language Mod- els with their Understanding of Helpfulness, Honesty, and Harmlessness. In Wang, M.; and Zitouni, I., eds., Proceed- ings of the 2023 Conference on Empirical Methods in Nat- ural Language Processing: Industry Track, 650-662. Singa- pore: Association for Computational Linguistics. Accessed: 2024-07-26.
  52. Wang, Y.; Zhong, W.; Li, L.; Mi, F.; Zeng, X.; Huang, W.; Shang, L.; Jiang, X.; and Liu, Q. 2023. Aligning Large Language Models with Human: A Survey. arXiv preprint arXiv:2307.12966. · arXiv
  53. Weijters, B.; Cabooter, E.; and Schillewaert, N. 2010. The effect of rating scale format on response styles: The number of response categories and response category labels. Inter- national Journal of Research in Marketing, 27(3): 236-247.
  54. Xiao, J.; Li, Z.; Xu, M.; Zhang, C.; Wang, M.; and Liu, Z. 2024. On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Reg- ularization. arXiv preprint arXiv:2405.16455. · arXiv
  55. Yin, J.; Goh, T.-T.; and Hu, Y. 2024. Interactions with ed- ucational chatbots: the impact of induced emotions and stu- dents' learning motivation. International Journal of Educa- tional Technology in Higher Education, 21(1): 47.
  56. Zhang, L. H.; Milli, S.; Jusko, K.; Smith, J.; Amos, B.; Bouaziz, W.; Revel, M.; Kussman, J.; Titus, L.; Radharapu, B.; Yu, J.; Sarma, V.; Rose, K.; and Nickel, M. 2025. Culti- vating Pluralism In Algorithmic Monoculture: The Commu- nity Alignment Dataset. arXiv preprint arXiv:2507.09650. · arXiv
  57. Zhang, T.; Zeng, Z.; Xiao, Y.; Zhuang, H.; Chen, C.; Foulds, J.; and Pan, S. 2024. GenderAlign: An Alignment Dataset for Mitigating Gender Bias in Large Language Models. arXiv preprint arXiv:2406.13925. · arXiv
  58. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023. Judg- ing LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36: 46595-46623.

Cited by

Sorted by citations

Authors
Ali, D., Zhao, D., Koenecke, A., & Papakyriakopoulos, O.
Affiliations
  • Technical University of Munich (DE)
  • Stanford University (US)
  • Cornell University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2025
Volume
40
Issue
44
Pages
37222-37231
ISSN
2159-5399 (ISSN-L)
DOI
10.48550/arxiv.2511.14476
arXiv
2511.14476
OpenAlex
W7139026891
Cited by
4
References
0
Citations by year
2026: 4
FWCI
18.82
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language model alignment, value pluralism, human preferences, toxicity, stereotypical bias, emotional awareness, rater disagreement, direct preference optimization, Group Relative Policy Optimization, social diversity, fine-tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review