The benefits, risks and bounds of personalizing the alignment of large language models to individuals
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Character.AI's permissive customization features and weak content moderation mechanisms make it alarmingly easy for minors to encounter developmentally inappropriate material, including sexually explicit, fetishistic, or emotionally manipulative content.
“Character.AI's permissive customization features and weak content moderation mechanisms make it alarmingly easy for minors to encounter developmentally inappropriate material, including sexually explicit, fetishistic, or emotionally manipulative content (Kirk et al. 2024).”
From Toward an Ethic of Synthetic Relationality: Identity, Intimacy, and Risk in AI-Mediated Roleplay Environments (2025) · cites this paper as Kirk et al. 2024
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Social Media and Fake News in the 2016 Election
Allcott, H., & Gentzkow, M. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.211 · Cited by 6,767 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
The echo chamber effect on social media
Cinelli, M., De Francisci Morales, G., Galeazzi, A., Quattrociocchi, W., & Starnini, M. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2023301118 · Cited by 2,092 · cited 1× · see citing works
“Personalized LLMs could entrench individual biases, risking polarization and breakdown of shared social cohesion 124–126 .”
FILTER BUBBLE: Wie wir im Internet entmündigt werden
Pariser, E. (2012) · doi:10.3139/9783446431164 · Cited by 1,849 · see citing works
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243 · see citing works
Inkster, B., Sarda, S., & Subramanian, V. (2018) · doi:10.2196/preprints.12106 · Cited by 1,063 · see citing works
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · cited 1× · see citing works
“Personalized LLMs may help to avoid the ‘value monism’ of current alignment techniques 109 , whereby technology providers and/or crowdworkers decide which values are prioritized or what constitutes a ‘good’ output 7,12 , thus entrenching one set of political, cultural or religious standpoints 37,42…”
Taxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · cited 3× · see citing works
“Some risks are inherited from LLMs 15–17 and artificial intelligence (AI) systems 18 more generally.”
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · see citing works
Fine-Tuning Language Models from Human Preferences
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019) · arXiv · doi:10.48550/arxiv.1909.08593 · Cited by 413 · see citing works
Immanuel Kant: Groundwork of the Metaphysics of Morals: A German–English edition
Kant, I. (2011) · doi:10.1017/cbo9780511973741 · Cited by 356 · see citing works
Debunking in a world of tribes
Zollo, F., Bessi, A., Del Vicario, M., Scala, A., Caldarelli, G., Shekhtman, L., Havlin, S., & Quattrociocchi, W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0181821 · Cited by 345 · see citing works
Online manipulation: hidden influences in a digital world
Susser, D., Roessler, B., & Nissenbaum, H. F. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3306006 · Cited by 341 · see citing works
Wachter, S. (2018) · Computer Law & Security Review · doi:10.1016/j.clsr.2018.02.002 · Cited by 307 · cited 1× · see citing works
“This is similar to any technology that relies on personal information to deliver tailored benefits 99 , such as with the Internet of Things 100 or targeted advertising 101,102 .”
WebGPT: Browser-assisted question-answering with human feedback
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09332 · Cited by 272 · see citing works
Barua, P. D., Vicnesh, J., Gururajan, R., Oh, S. L., Palmer, E., Azizan, M. M., Kadri, N. A., & Acharya, U. R. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19031192 · Cited by 270 · see citing works
Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good
Wang, X., Shi, W., Kim, R., Oh, Y., Yang, S., Zhang, J., & Yu, Z. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1566 · Cited by 222 · see citing works
Abbass, H. A. (2019) · Cognitive Computation · doi:10.1007/s12559-018-9619-0 · Cited by 200 · see citing works
Motivations in personalisation behaviour
Oulasvirta, A., & Blom, J. (2008) · Interacting with Computers · doi:10.1016/j.intcom.2007.06.002 · Cited by 117 · see citing works
Automated Personalized Feedback Improves Learning Gains in An Intelligent Tutoring System
Kochmar, E., Vu, D. D., Belfer, R., Gupta, V., Serban, I. V., & Pineau, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-52240-7_26 · Cited by 103 · see citing works
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
Strict liability for AI and other emerging technologies
Wendehorst, C. (2020) · Journal of European Tort Law · doi:10.1515/jetl-2020-0140 · Cited by 70 · see citing works
Discovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.09251 · Cited by 53 · cited 2× · see citing works
“…LLMs may help to avoid the ‘value monism’ of current alignment techniques 109 , whereby technology providers and/or crowdworkers decide which values are prioritized or what constitutes a ‘good’ output 7,12 , thus entrenching one set of political, cultural or religious standpoints 37,42 .”
Refocusing on Relevance: Personalization in NLG
Dudy, S., Bedrick, S., & Webber, B. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.421 · Cited by 17 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
A generalist medical language model for disease diagnosis assistance
Liu, X., Liu, H., Yang, G., Jiang, Z., Cui, S., Zhang, Z., Wang, H., Tao, L., Sun, Y., Song, Z., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03416-6 · Cited by 183 · see citing works
Mapping the ethics of generative AI: a comprehensive scoping review
Hagendorff, T. (2024) · Minds and Machines · doi:10.1007/s11023-024-09694-w · Cited by 141 · see citing works
Evaluating the persuasive influence of political microtargeting with large language models
Hackenburg, K., & Margetts, H. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2403116121 · Cited by 114 · see citing works
Opportunities and challenges of using generative AI to personalize educational assessment
Arslan, B., Lehman, B., Tenison, C., Sparks, J. R., López, A. A., Gu, L., & Zapata-Rivera, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1460651 · Cited by 52 · see citing works
Technological folie à deux: feedback loops between AI chatbots and mental health
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00595-8 · Cited by 51 · see citing works
Dahlgren Lindström, A., Methnani, L., Krause, L., Ericson, P., de Rituerto de Troya, Í. M., Coelho Mollo, D., & Dobbe, R. (2025) · Ethics and Information Technology · doi:10.1007/s10676-025-09837-2 · Cited by 54 · see citing works
“…are seen as more important and which less), as well as in linguistic expression, most often in favour of what is considered proper and acceptable by the hegemonic social groups typically responsible for the design of LLMs (Helm et al., 2024; Weidinger et al., 2021; Kirk et al., 2024a, 2024b).”
Alvero, A. J., Lee, J., Regla-Vargas, A., Kizilcec, R. F., Joachims, T., & Antonio, A. L. (2024) · Journal of Big Data · doi:10.1186/s40537-024-00986-7 · Cited by 48 · see citing works
Sankar, B. S., Gilliland, D., Rincon, J., Hermjakob, H., Yan, Y., Adam, I., Lemaster, G., Wang, D., Watson, K., Bui, A., et al. (2024) · Bioengineering · doi:10.3390/bioengineering11100984 · Cited by 26 · see citing works
Technological folie à deux: Feedback Loops Between AI Chatbots and Mental Illness
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2025) · arXiv · doi:10.48550/arxiv.2507.19218 · Cited by 26 · see citing works
“This may lead to socially anxious individuals favouring chatbot interactions 20,86,87 , further hindering recovery (another example of proxy failure, where the maximising short-term reward stymies longer-term flourishing).”
Bhat, M. (2025) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1609/aies.v8i1.36560 · Cited by 5 · see citing works
Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Jiang, L., Chai, Y., Li, M., Liu, M., Fok, R., Dziri, N., Tsvetkov, Y., Sap, M., Albalak, A., & Choi, Y. (2025) · arXiv · doi:10.48550/arxiv.2510.22954 · Cited by 1 · see citing works
Explainability in AI: Comparing Human-Like and System-Like Trust Repair Strategies
Konopka, B., & Wiesche, M. (2026) · Information Systems Frontiers · doi:10.1007/s10796-026-10751-1 · Cited by 1 · see citing works
- Authors
- Kirk, H. R., Vidgen, B., Röttger, P., & Hale, S. A.
- Affiliations
- University of Oxford (GB)
- Bocconi University (IT)
- Corresponding author
- Kirk, Hannah Rose
- Venue
- Nature Machine Intelligence
- Published
- 2024-04-23
- Volume
- 6
- Issue
- 4
- Pages
- 383-392
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W4395025834
- Cited by
- 122
- References
- 26
- Citations by year
- 2026: 45 · 2025: 50 · 2024: 26 · 2023: 1
- FWCI
- 38.98
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- Nature Machine Intelligence (publishedVersion)
- Oxford University Research Archive (ORA) (University of Oxford) (acceptedVersion)
- Archivio istituzionale della Ricerca - Bocconi (submittedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Law › Digitalization, Law, and Regulation
- Keywords
- personalized large language models, LLM alignment, profiling, privacy violations, algorithmic bias, risk taxonomy, normative decision making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
