Are Large Language Models Consistent over Value-laden Questions?
Open accessAbstract
Large language models (LLMs) appear to bias their survey answers toward certain values. Nonetheless, some argue that LLMs are too inconsistent to simulate particular values. Are they? To answer, we first define value consistency as the similarity of answers across (1) paraphrases of one question, (2) related questions under one topic, (3) multiple-choice and open-ended use-cases of one question, and (4) multilingual translations of a question to English, Chinese, German, and Japanese. We apply these measures to small and large, open LLMs including llama-3, as well as gpt-4o, using 8,000 questions spanning more than 300 topics. Unlike prior work, we find that models are relatively consistent across paraphrases, use-cases, translations, and within a topic. Still, some inconsistencies remain. Models are more consistent on uncontroversial topics (e.g., in the U.S., "Thanksgiving") than on controversial ones ("euthanasia"). Base models are both more consistent compared to fine-tuned models and are uniform in their consistency across topics, while fine-tuned models are more inconsistent about some topics ("euthanasia") than others ("women's rights") like our human subjects (n=165).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The difference between multiple choice and open-ended answering is not the case for larger models.
“although this is not the case for larger models [98]”
From Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. (2025) · cites this paper as [98]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 12
- Muhammad Farid Adilazuarda, Sagnik Mukherjee, Prad- hyumna Lavania, Siddhant Singh, Ashutosh Dwivedi, Al- ham Fikri Aji, Jacki O'Neill, Ashutosh Modi, and Mono- jit Choudhury. 2024. Towards Measuring and Mod- eling "Culture" in LLMs: A Survey. arXiv preprint. ArXiv:2403.15412 [cs]. · arXiv
- Lewis R. Goldberg, John A. Johnson, Herbert W. Eber, Robert Hogan, Michael C. Ashton, C. Robert Cloninger, and Har- rison G. Gough. 2006. The international personality item pool and the future of public-domain personality measures. Journal of Research in personality, 40(1):84-96. ISBN: 0092-6566 Publisher: Elsevier.
- Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Bi Pu- ranen. 2022a. World Values Survey Wave 7 (2017-2022) · DOI
- Cross-National Data-Set. · DOI
- Christian Haerpfer, Ronald Inglehart, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Jaime Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, Bi Puranen, et al. 2022b. World values survey: Round seven-country- pooled datafile version 5.0. Madrid, Spain & Vienna, Aus- tria: JD Systems Institute & WVSA Secretariat, 12(10):8.
- Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny T. Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. 2021. Delphi: To- wards Machine Ethics and Norms. ArXiv.
- Daniel Jurafsky and James H. Martin. 2024. Speech and Language Processing, 3rd ed. draft edition.
- John Rawls. 1971. A Theory of Justice. Belknap Press of Harvard University Press.
- Taylor Sorensen, Liwei Jiang, Jena Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, Maarten Sap, John Tasioulas, and Yejin Choi. 2023. Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties. arXiv preprint. ArXiv:2309.00779 [cs. · DOI
- Yan Tao, Olga Viberg, Ryan S. Baker, and Rene F. Kizilcec. 2023. Auditing and Mitigating Cultural Bias in LLMs. arXiv preprint. ArXiv:2311.14096 [cs]. · DOI
- Angelina Wang, Jamie Morgenstern, and John P. Dickerson. 2024a. Large language models cannot replace human participants because they cannot portray identity groups. arXiv preprint. ArXiv:2402.01908 [cs]. · arXiv
- Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, B. Schölkopf, and Kun Zhang. 2022. Adversarial Robustness through the Lens of Causal- ity. ArXiv.
Cited by
Sorted by citations
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732039 · Cited by 204 · see citing works
Self-citationWhy AI Is WEIRD and Shouldn't Be This Way: Towards AI for Everyone, with Everyone, by Everyone
Mihalcea, R. F., Ignat, O., Bai, L., Borah, A., Chiruzzo, L., Jin, Z., Kwizera, C., Nwatu, J. C., Poria, S., & Solorio, T. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i27.35092 · Cited by 42 · see citing works
Large language models reflect the ideology of their creators
Buyl, M., Rogiers, A., Noels, S., Bied, G., Dominguez-Catena, I., Heiter, E., Johary, I., Mara, A. C., Romero, R., Lijffijt, J., et al. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-025-00048-0 · Cited by 28 · see citing works
Balancing Large Language Model Alignment and Algorithmic Fidelity in Social Science Research
Lyman, A., Hepner, B., Argyle, L. P., Busby, E. C., Gubler, J. R., & Wingate, D. (2025) · Sociological Methods & Research · doi:10.1177/00491241251342008 · Cited by 19 · see citing works
Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs
Khan, A., Casper, S., & Hadfield-Menell, D. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732147 · Cited by 16 · see citing works
LLMs and Cultural Values: The Impact of Prompt Language and Explicit Cultural Framing
Bulté, B., & Terryn, A. R. (2025) · Computational Linguistics · doi:10.1162/coli.a.583 · Cited by 10 · see citing works
SocialGaze: Improving the Integration of Human Social Norms in Large Language Models
Vijjini, A. R., Menon, R. R., Fu, J., Srivastava, S., & Chaturvedi, S. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.962 · Cited by 3 · see citing works
Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday Tasks
Arunasalam, A., Pickering, M., Celik, Z. B., & Ur, B. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.847 · Cited by 1 · see citing works
Mind the Gap: How Elicitation Protocols Shape the Stated-Revealed Preference Gap in Language Models
Mahajan, P., Kendiukhov, I., Hussain, S., & Nottingham, L. (2026) · Workshop on Evaluating Evaluations (EvalEval) · doi:10.18653/v1/2026.evaleval-1.9 · Cited by 1 · see citing works
Assessing Large Language Model Alignment Towards Radiological Myths and MisconceptionsPreprint
West, C., & Wang, Y. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.05.16.652427
Smith-Vaniz, N., Lyon, H., Steigner, L., Armstrong, B., & Mattei, N. (2025) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v8i3.36727
Reward Model Perspectives: Whose Opinions Do Reward Models Reward?
Elle (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.754
Can Multimodal Large Language Models Understand Human Values in Videos?
Wang, Y., Qiu, H., Chang, H., Xiao, J., Tao, Z., & Jin, L. (2025) · ACM International Conference on Multimedia in Asia (MMAsia) · doi:10.1145/3743093.3770993
West, C., & Wang, Y. (2026) · International Journal of Radiation Biology · doi:10.1080/09553002.2026.2654427
Wang, F., Guo, L. P., & Zhang, Y. R. (2026) · Education and Information Technologies · doi:10.1007/s10639-026-14060-7
MMVUF: Improving video-based human value understanding in multimodal large language models
Wang, Y., Chang, H., Qiu, H., Jin, L., & Tao, Z. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133457
Position: Evaluations of AI Moral Reasoning Still Miss Half of the Picture
Kierans, A., Dutt, R., Rittichier, K. J., Dori-Hacohen, S., & Ghosh, A. (2026) · Workshop on Evaluating Evaluations (EvalEval) · doi:10.18653/v1/2026.evaleval-1.38
Dong, Q., & Qi, J. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-3734-0_13
Stochasticity-Induced Unreliability in LLM-Based Hiring Systems
Giri, A., Bhat, K., Nehal, G., & H, M. K. (2026) · IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) · doi:10.1109/conecct68978.2026.11700588
Limitations of Computational Morality: Can We Morally Enhance Ourselves via AI?
Richter, F. (2026) · Science and Engineering Ethics · doi:10.1007/s11948-026-00634-8
Cultural Alignment of Open-Weight LLMs on the Inglehart–Welzel Map
Vimalendiran, S. (2026) · doi:10.18653/2026.oracle-main.14
- Authors
- Moore, J., Deshpande, T., & Yang, D.
- Affiliations
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 15185-15221
- arXiv
- 2407.02996
- OpenAlex
- W4404780769
- Cited by
- 12
- References
- 0
- Citations by year
- 2026: 7 · 2025: 5
- FWCI
- 5.47
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2247357)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, value consistency, survey responses, multilingual translation, fine-tuned models, LLM bias, controversial issues
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
