One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity
Open accessPublished version of a preprint
Abstract
Sonia Krishna Murthy, Tomer Ullman, Jennifer Hu. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RLHF significantly reduces output diversity compared to a supervised baseline, and similar effects have been documented for other alignment fine-tuning methods.
“RLHF has been shown to significantly reduce output diversity compared to a supervised baseline [Kirk et al., 2024], and similar effects have been documented for other alignment fine-tuning methods [O'Mahony et al., 2024, Murthy et al., 2024].”
From Antislop: A Comprehensive Framework for Identifying and Eliminating Repetitive Patterns in Language Models (2025) · cites this paper as Murthy et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 18
- Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. 2021. Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 109-132, Online. Association for Computational Lin- guistics.
- Gati Aher, Rosa I. Arriaga, and Adam Tauman Kalai. 2023. Using large language models to simulate mul- tiple humans and replicate human subject studies. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org. Place: Honolulu, Hawaii, USA.
- Marcel Binz and Eric Schulz. 2024. Turning large lan- guage models into cognitive models. In Proceedings of the International Conference on Learning Repre- sentations.
- Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences of the United States of America, 120.
- Jennifer Hu and Michael Frank. 2024. Auxiliary task demands mask the capabilities of smaller language models. In First Conference on Language Modeling.
- Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv preprint.
- Louis Martí, Shengyi Wu, Steven T. Piantadosi, and Ce- leste Kidd. 2023. Latent diversity in human concepts. Open Mind : Discoveries in Cognitive Science, 7:79 -92. · DOI
- Laura O'Mahony, Leo Grinsztajn, Hailey Schoelkopf, and Stella Biderman. 2024. Attributing Mode Col- lapse in the fine-tuning of Large Language Models. 11251 In ICLR 2024 Workshop on Mathematical and Em- pirical Understanding of Foundation Models.
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744.
- Vishakh Padmakumar and He He. 2024. Does writing with language models reduce content diversity? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
- Roma Patel and Ellie Pavlick. 2022. Mapping Language Models to Grounded Conceptual Spaces. In Interna- tional Conference on Learning Representations.
- Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. 2024. Is Temperature the Creativity Parameter of Large Language Models? In Proceed- ings of the 15th International Conference on Compu- tational Creativity (ICCC'24).
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct Preference Optimization: Your Lan- guage Model is Secretly a Reward Model. In Thirty- seventh Conference on Neural Information Process- ing Systems.
- Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, and Zeynep Akata. 2024. In-context im- personation reveals large language models' strengths and biases. In Proceedings of the 37th International Conference on Neural Information Processing Sys- tems, NIPS '23, Red Hook, NY, USA.
- Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org. Place: Honolulu, Hawaii, USA.
- Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clé- mentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2024. Zephyr: Direct Distillation of LM Align- ment. In First Conference on Language Modeling.
- Pengda Wang, Huiqi Zou, Zihan Yan, Feng Guo, Tian- jun Sun, Ziang Xiao, and Bo Zhang. 2024c. Not Yet: Large Language Models Cannot Replace Human Re- spondents for Psychometric Research. · DOI
- Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023. Large Language Model as Attributed Training Data Generator: A Tale of Di- versity and Bias. In Advances in Neural Information Processing Systems, volume 36, pages 55734-55784.
Cited by
Sorted by citations
Six Fallacies in Substituting Large Language Models for Human Participants
Lin, Z. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251357566 · Cited by 20 · see citing works
Chen, C., Yao, B., Zou, R., Hua, W., Lyu, W., Li, T. J.-J., & Wang, D. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.938 · Cited by 11 · see citing works
Operationalizing Geographic Diversity for the Evaluation of AI ‐Generated Content
Liu, Z., Janowicz, K., Majić, I., Shi, M., Fortacz, A., Karimi, M., Mai, G., & Currier, K. (2025) · Transactions in GIS · doi:10.1111/tgis.70057 · Cited by 7 · see citing works
Whose Truth? Pluralistic Geo-Alignment for (Agentic) AI
Janowicz, K., Liu, Z., Mai, G., Wang, Z., Majić, I., Fortacz, A., McKenzie, G., & Gao, S. (2025) · ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (SIGSPATIAL GIS) · doi:10.1145/3748636.3760465 · Cited by 5 · see citing works
Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles
Truong, K. L., Fogliato, R., Heidari, H., & Wu, Z. S. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1155 · Cited by 3 · see citing works
Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Jiang, L., Chai, Y., Li, M., Liu, M., Fok, R., Dziri, N., Tsvetkov, Y., Sap, M., Albalak, A., & Choi, Y. (2025) · arXiv · doi:10.48550/arxiv.2510.22954 · Cited by 1 · see citing works
Lin, Z. (2026) · Annual Review of Psychology · doi:10.1146/annurev-psych-100925-034807 · Cited by 4 · see citing works
McK: Ontology for mapping semantic conflict in plot summary
Chiappucci, N. (2026) · Computational Humanities Research · doi:10.1017/chr.2026.10028 · Cited by 1 · see citing works
Stochastic parrot or not, it is less of a turkey than the colleague two desks over
Falegnami, A., Tomassi, A., Levorato, M., & Costantino, F. (2026) · AI & Society · doi:10.1007/s00146-026-03095-6 · Cited by 1 · see citing works
Paech, S., Roush, A., Goldfeder, J., & Shwartz-Ziv, R. (2025) · arXiv · doi:10.48550/arxiv.2510.15061
“RLHF has been shown to significantly reduce output diversity compared to a supervised baseline [Kirk et al., 2024], and similar effects have been documented for other alignment fine-tuning meth-ods [O’Mahony et al., 2024, Murthy et al., 2024].”
What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity
Audran-Reiss, A., Armengol-Estapé, J., Hambardzumyan, K., Budhiraja, A., Josifoski, M., Toledo, E., Hazra, R., Magka, D., Shvartsman, M., Pathak, P., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.15593
Leveraging Human Production-Interpretation Asymmetries to Test LLM Cognitive Plausibility
Lam, S.-Y., Zeng, Q., Wu, J., & Voigt, R. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-short.14
Xenoreproduction: Exploration and Recovery of Collapsible Modes as Core AI Safety ObjectivePreprint
Rios-Sialer, I. (2025) · doi:10.31224/5890
Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety
Janiak, D., Moska, J., Motyka, D., Seweryn, K., Walkowiak, P., Żuk, B., & Janz, A. (2026) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2026.eacl-srw.7
Heierli, J., Ehrensperger, I., Rüegg, M., de Spindler, A., & Gavagnin, E. (2026) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds70563.2026.00038
A Persona-Augmented Multi-Agent System for Varied Narrative Generation
Sillano, A., Arbore, G., & Russis, L. D. (2026) · ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) · doi:10.1145/3807968.3810931
A Neurosymbolic Architecture for Semiotic Screenplay Interpretation
Chiappucci, N., & Lechelt, S. (2026) · Creativity and Cognition · doi:10.1145/3803784.3816849
A Network Analytic Comparison of Word Associations Generated by Humans and Large Language Models
Siew, C. S. Q. (2026) · Quarterly Journal of Experimental Psychology · doi:10.1177/17470218261487165
- Authors
- Murthy, S. K., Ullman, T., & Hu, J.
- Affiliations
- Harvard University (US)
- Venue
- Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
- Published
- 2025
- Pages
- 11241-11258
- OpenAlex
- W4411119303
- Cited by
- 8
- References
- 0
- Citations by year
- 2026: 6 · 2025: 2
- FWCI
- 7.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (2140743)
- Harvard University
- Jacobs Foundation
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
- Keywords
- language model alignment, conceptual diversity, language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
