CoChat
Sign Up
Cited by 1Natural Language Processing TechniquesTopic ModelingEthics and Social Impacts of AI

From Distributional to Overton Pluralism: Investigating Large Language Model Alignment

Open access
Lake, Thom, Choi, Eunsol, Durrett, Greg·2025·Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
DOI 10.18653/v1/2025.naacl-long.346
Open this paper in CoChat — free accountOpen PDF

Abstract

Thom Lake, Eunsol Choi, Greg Durrett. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2025.naacl-long.346

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Lake et al. (2025) proxy Overton pluralism via the proportion of model responses including both perspectives on simple yes-no questions, but the binary nature of the questions is unrealistic and unsuitable for benchmarking.

    “Lake et al. (2025) proxy Overton pluralism via the proportion of model responses including both perspectives on simple yes-no questions. However, the binary nature of the questions is unrealistic and unsuitable for benchmarking.”

    From Benchmarking Overton Pluralism in LLMs (2025) · cites this paper as Lake et al., 2025

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. Abeer AlDayel and Walid Magdy. 2021. Stance detec- tion on social media: State of the art and trends. In- formation Processing & Management, 58(4):102597.
  2. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. · arXiv
  3. Maksym Andriushchenko, Francesco Croce, and Nico- las Flammarion. 2024. Jailbreaking Leading Safety- Aligned LLMs with Simple Adaptive Attacks. arXiv preprint arXiv:2404.02151. · arXiv
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv preprint arXiv:2204.05862. · arXiv
  5. Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Ko- rbak, and Owain Evans. 2023. The Reversal Curse: LLMs Trained on "A is B" Fail to Learn "B is A". In International Conference on Learning Representa- tions (ICLR).
  6. Steven Bird, Ewan Klein, and Edward Loper. 2009. Nat- ural language processing with Python: analyzing text with the natural language toolkit. " O'Reilly Media, Inc.".
  7. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththa- ranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. 2023. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. Transactions on Machine Learning Research (TMLR).
  8. Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du. 2023. Everyone Deserves a Reward: Learn- ing Customized Human Preferences. arXiv preprint arXiv:2309.03126. · arXiv
  9. Le Bronnec Florian, Verine Alexandre, Negrevergne Benjamin, Chevaleyre Yann, and Allauzen Alexan- dre. 2024. Exploring Precision and Recall to Assess the Quality and Diversity of LLMs. arXiv preprint arXiv:2402.10693. · arXiv
  10. Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymetman. 2023. Aligning Language Models with Preferences through f-divergence Minimization. In International Conference on Machine Learning (ICML).
  11. Xiaochuang Han. 2023. In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuning. arXiv preprint arXiv:2308.04275. · arXiv
  12. Shirley Anugrah Hayati, Minhwa Lee, Dheeraj Ra- jagopal, and Dongyeop Kang. 2023. How Far Can We Extract Diverse Perspectives from Large Lan- guage Models? Criteria-Based Diversity Prompting! arXiv preprint arXiv:2311.09799. · arXiv
  13. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. · arXiv
  14. Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023. Personalized Soups: Personalized Large Lan- guage Model Alignment via Post-Hoc Parameter Merging. arXiv preprint arXiv:2310.11564. · arXiv
  15. AQ Jiang, A Sablayrolles, A Mensch, C Bamford, DS Chaplot, D de las Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825. · arXiv
  16. Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wen- juan Han, Chi Zhang, and Yixin Zhu. 2024. Evaluat- ing and Inducing Personality in Pre-trained Language Models. In Advances in Neural Information Process- ing Systems (NeurIPS).
  17. Muhammad Khalifa, Hady Elsahar, and Marc Dymet- man. 2020. A Distributional Approach to Controlled Text Generation. In International Conference on Learning Representations (ICLR).
  18. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. Understanding the Ef- fects of RLHF on LLM Generalisation and Diversity. In International Conference on Learning Representa- tions (ICLR).
  19. Preethi Lahoti, Nicholas Blumm, Xiao Ma, Raghaven- dra Kotikalapudi, Sahitya Potluri, Qijun Tan, Hansa Srinivasan, Ben Packer, Ahmad Beirami, Alex Beu- tel, et al. 2023. Improving Diversity of Demo- graphic Representation in Large Language Models via Collective-Critiques and Self-Voting. In Em- pirical Methods in Natural Language Processing (EMNLP).
  20. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A Diversity-Promoting Objective Function for Neural Conversation Models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies.
  21. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christo- pher D Manning, Christopher Re, Diana Acosta- Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan An- drew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Ko- reeda. 2023. Holistic Evaluation of Language Mod- els. Transactions on Machine Learning Research (TMLR).
  22. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chan- dra Bhagavatula, and Yejin Choi. 2024. The Unlock- ing Spell on Base LLMs: Rethinking Alignment via In-Context Learning. In International Conference on Learning Representations (ICLR).
  23. Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. 2024. Tun- ing Language Models by Proxy. arXiv preprint arXiv:2401.08565. · arXiv
  24. Siyang Liu, Sahand Sabour, Yinhe Zheng, Pei Ke, Xi- aoyan Zhu, and Minlie Huang. 2022. Rethinking and Refining the Distinct Metric. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
  25. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training Language Models to Follow Instruc- tions with Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS).
  26. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct Preference Optimization: Your Lan- guage Model is Secretly a Reward Model. In Ad- vances in Neural Information Processing Systems (NeurIPS).
  27. Michael J Ryan, William Held, and Diyi Yang. 2024. Unintended Impacts of LLM Alignment on Global Representation. arXiv preprint arXiv:2402.15018. · arXiv
  28. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task Prompted Training Enables Zero-Shot Task Gen- eralization. In International Conference on Learning Representations (ICLR).
  29. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose Opinions Do Language Models Reflect? In International Conference on Machine Learning (ICML).
  30. Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa F Siu, Byron C Wallace, and Ani Nenkova. 2024. Standard- izing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores. arXiv preprint arXiv:2403.00553. · arXiv
  31. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023. A Long Way to Go: Investigat- ing Length Correlations in RLHF. arXiv preprint arXiv:2310.03716. · arXiv
  32. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Li- wei Jiang, Ximing Lu, Nouha Dziri, et al. 2024. A Roadmap to Pluralistic Alignment. arXiv preprint arXiv:2402.05070. · arXiv
  33. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to Summarize with Human Feedback. In Ad- vances in Neural Information Processing Systems (NeurIPS).
  34. Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Large Language Models are In- consistent and Biased Vvaluators. arXiv preprint arXiv:2405.01724. · arXiv
  35. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open Founda- tion and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288. · arXiv
  36. Alexander Wan, Eric Wallace, and Dan Klein. 2024. What evidence do language models find convincing? arXiv preprint arXiv:2402.11782. · arXiv
  37. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368. · arXiv
  38. Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024. Long is More for Alignment: A Simple but Tough-to-Beat Base- line for Instruction Fine-Tuning. arXiv preprint arXiv:2402.04833. · arXiv
  39. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. LIMA: Less is More for Alignment. In Advances in Neural Information Pro- cessing Systems (NeurIPS).
  40. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A Benchmarking Platform for Text Generation Models. In The 41st international ACM SIGIR conference on research & development in information retrieval.

Cited by

Sorted by citations

Authors
Lake, T., Choi, E., & Durrett, G.
Affiliations
New York University (US)
ORCID
Venue
Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Publisher
Association for Computational Linguistics
Published
2025
Pages
6794-6814
DOI
10.18653/v1/2025.naacl-long.346
OpenAlex
W4411119313
Cited by
1
References
1
Citations by year
2025: 1
FWCI
0.95
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
Funders
National Science Foundation (2145280)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
Keywords
large language model alignment, Overton window, AI alignment, value alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review