Beyond Preferences in AI Alignment
Open accessThe DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Abstract
The dominant practice of AI alignment assumes (1) that preferences are an adequate representation of human values, (2) that human rationality can be understood in terms of maximizing the satisfaction of preferences, and (3) that AI systems should be aligned with the preferences of one or more humans to ensure that they behave safely and in accordance with our values. Whether implicitly followed or explicitly endorsed, these commitments constitute what we term a preferentist approach to AI alignment. In this paper, we characterize and challenge the preferentist approach, describing conceptual and technical alternatives that are ripe for further research. We first survey the limits of rational choice theory as a descriptive model, explaining how preferences fail to capture the thick semantic content of human values, and how utility representations neglect the possible incommensurability of those values. We then critique the normativity of expected utility theory (EUT) for humans and AI, drawing upon arguments showing how rational agents need not comply with EUT, while highlighting how EUT is silent on which preferences are normatively acceptable. Finally, we argue that these limitations motivate a reframing of the targets of AI alignment: Instead of alignment with the preferences of a human user, developer, or humanity-writ-large, AI systems should be aligned with normative standards appropriate to their social roles, such as the role of a general-purpose assistant. Furthermore, these standards should be negotiated and agreed upon by all relevant stakeholders. On this alternative conception of alignment, a multiplicity of AI systems will be able to serve diverse ends, aligned with normative standards that promote mutual benefit and limit harm despite our plural and divergent values.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
Epistemic Injustice: Power and the Ethics of Knowing
Fricker, M. (2007) · doi:10.1093/acprof:oso/9780198237907.001.0001 · Cited by 9,198 · cited 1×
“Uniform preference aggregation may thus constitute a form of epistemic injustice (Fricker, 2007; Symons and Alvarado, 2022; Hull, 2023), which in turn leads to downstream injustice and harm.”
BackgroundEconomy and Society: An Outline of Interpretive Sociology
Weber, M. (2002) · Readings in Economic Sociology · doi:10.1002/9780470755679.ch3 · Cited by 6,867
STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving
Fikes, R. E., & Nilsson, N. J. (1971) · Artificial Intelligence · doi:10.1016/0004-3702(71)90010-5 · Cited by 4,882 · cited 1×
“This is despite the existence of other useful and potentially more appropriate formalisms, such as the formalization of goals as logical specifications (Fikes and Nilsson, 1971), and the formalization of intentions as (partial) plans (Bratman, 1987; Bratman et al., 1988).”
MethodTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
A contextual-bandit approach to personalized news article recommendation
Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010) · Proceedings of the 19th international conference on World wide web · doi:10.1145/1772690.1772758 · Cited by 2,562 · cited 1×
“…of recommender systems (Thorburn et al., 2022), with many papers modeling recommendation as the problem of showing items to users that they are most likely to engage with, which is presumed to be the item they find the most rewarding (Li et al., 2010; Hill et al., 2017; McInerney et al., 2018).”
BackgroundWhy do humans reason? Arguments for an argumentative theory
Mercier, H., & Sperber, D. (2011) · Behavioral and Brain Sciences · doi:10.1017/s0140525x10000968 · Cited by 2,254
The Grammar of Society: The Nature and Dynamics of Social Norms
Bicchieri, C. (2005) · doi:10.1017/cbo9780511616037 · Cited by 1,991
The Complexity of Markov Decision Processes
Papadimitriou, C. H., & Tsitsiklis, J. N. (1987) · Mathematics of Operations Research · doi:10.1287/moor.12.3.441 · Cited by 1,329
Individual choice behavior: A theoretical analysis.
Luce, R. D. (2005) · doi:10.1037/14396-000 · Cited by 1,157
Shifting attention to accuracy can reduce misinformation online
Pennycook, G., Epstein, Z., Mosleh, M., Arechar, A. A., Eckles, D., & Rand, D. G. (2021) · Nature · doi:10.1038/s41586-021-03344-2 · Cited by 1,155
Gorwa, R., Binns, R., & Katzenbach, C. (2020) · Big Data & Society · doi:10.1177/2053951719897945 · Cited by 1,097 · cited 1×
“For example, machine learning methods are routinely used to moderate content that may be regarded as toxic and offensive (Gorwa et al., 2020), or to steer LLMs towards producing outputs that are less harmful (Bai et al., 2022a).”
BackgroundDeep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058
Action understanding as inverse planning
Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009) · Cognition · doi:10.1016/j.cognition.2009.07.005 · Cited by 920
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · cited 5×
“…discussion as to whether its component theses are warranted (Shah, 2018; Eckersley, 2018; Hadfield-Menell and Hadfield, 2018; Wentworth, 2019; Gabriel, 2020; Vamplew et al., 2022; Korinek and Balwit, 2022; Thornley, 2023), echoing similar debates in economics, decision theory, and philosophy.”
BackgroundInfluentialLieder, F., & Griffiths, T. L. (2020) · Behavioral and Brain Sciences · doi:10.1017/s0140525x1900061x · Cited by 892
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2338 · Cited by 686
(2005) · doi:10.1093/acprof:oso/9780198567523.003.0011 · Cited by 585 · cited 1×
“…discussed, and partly a feature of human psychology: per Kahneman, our stated preferences about an experience can vary with the time of elicitation (Kahneman and Riis, 2005); per Sen and Nussbaum, our preferences adapt to the conditions of what is available to us (Sen et al., 1999; Nussbaum, 2001…”
BackgroundSilver, D., Singh, S., Precup, D., & Sutton, R. S. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103535 · Cited by 445 · cited 1×
“…a sequence of states and actions ξ = ( s 1 , a 1 , ..., a T − 1 , s T ) can be decomposed into a sum of scalar rewards over time: Advocates of the reward representation argue that any task accomplishable by an intelligent agent can be framed as a reward maximization problem (Silver et al., 2021).”
BackgroundCooperative Inverse Reinforcement Learning
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03137 · Cited by 413
Parfit, D. (2011) · doi:10.1093/acprof:osobl/9780199572809.001.0001 · Cited by 350
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345
Faith and Fate: Limits of Transformers on Compositionality
Dziri, N., Lu, X., Sclar, M., Li, X. (., Jiang, L., Lin, B. Y., Welleck, S., West, P., Bhagavatula, C., Bras, R. L., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3081 · Cited by 335 · cited 3×
“Still, LLMs remain limited in their ability to represent and reason with compositional concepts (Dziri et al., 2023; Mahowald et al., 2024; Ramesh et al., 2024), and would function as poor models of humans on their own.”
BackgroundPower to the People? Opportunities and Challenges for Participatory AI
Birhane, A., Isaac, W., Prabhakaran, V., Diaz, M., Elish, M. C., Gabriel, I., & Mohamed, S. (2022) · Equity and Access in Algorithms Mechanisms and Optimization · doi:10.1145/3551624.3555290 · Cited by 330
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2305.18290 · Cited by 293
Provably Bounded-Optimal Agents
Russell, S. J., & Subramanian, D. (1995) · Journal of Artificial Intelligence Research · doi:10.1613/jair.133 · Cited by 252
Quilty-Dunn, J., Porot, N., & Mandelbaum, E. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x22002849 · Cited by 205 · cited 1×
“The structured nature of these concepts suggests the need for a suitably rich language of thought — one that captures the compositionality and algorithmic complexity of human conceptual cognition (Piantadosi and Jacobs, 2016; Quilty-Dunn et al., 2023; Wong et al., 2023).”
BackgroundCritiquing the Reasons for Making Artificial Moral Agents
van Wynsberghe, A., & Robbins, S. (2019) · Science and Engineering Ethics · doi:10.1007/s11948-018-0030-8 · Cited by 161
Social choice ethics in artificial intelligence
Baum, S. D. (2020) · AI & SOCIETY · doi:10.1007/s00146-017-0760-1 · Cited by 158
A Rawlsian algorithm for autonomous vehicles
Leben, D. (2017) · Ethics and Information Technology · doi:10.1007/s10676-017-9419-3 · Cited by 141 · cited 1×
“In cases of significant disagreement, one might even place all weight on the individual with the strongest dispreference (Leben, 2017; Bakker et al., 2022; Weidinger et al., 2023).”
BackgroundThe case against economic values in the orbitofrontal cortex (or anywhere else in the brain).
Hayden, B. Y., & Niv, Y. (2021) · Behavioral Neuroscience · doi:10.1037/bne0000448 · Cited by 127
In Conversation with Artificial Intelligence: Aligning language Models with Human Values
Kasirzadeh, A., & Gabriel, I. (2023) · Philosophy & Technology · doi:10.1007/s13347-023-00606-x · Cited by 127 · cited 1×
“More generally, AI systems should not be aligned with preferences, but with the normative standards appropriate to their social roles and functions (Kasirzadeh and Gabriel, 2023).”
BackgroundHadfield-Menell, D., Milli, S., Abbeel, P., Russell, S., & Dragan, A. (2017) · arXiv · doi:10.48550/arxiv.1711.02827 · Cited by 125
Rational use of cognitive resources in human planning
Callaway, F., van Opheusden, B., Gul, S., Das, P., Krueger, P. M., Griffiths, T. L., & Lieder, F. (2022) · Nature Human Behaviour · doi:10.1038/s41562-022-01332-8 · Cited by 121 · cited 1×
“For the most part, resource rational analyses continue to represent human preferences in terms of scalar costs and rewards, or more generally, utility functions, with the primary innovation being the inclusion of costs on computation (Lieder and Griffiths, 2020; Callaway et al., 2022).”
BackgroundScalable agent alignment via reward modeling: a research direction
Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., & Legg, S. (2018) · arXiv · doi:10.48550/arxiv.1811.07871 · Cited by 121 · cited 2×
“In alignment research, this manifests as the tendency to frame value alignment in terms of reward learning (Hadfield-Menell et al., 2016; Leike et al., 2018), and to formalize concepts like “goals” (Di Langosco et al., 2022) and “intents” (Ouyang et al., 2022) as reward functions.”
BackgroundHadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/32 · Cited by 119
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1611.08219 · Cited by 119
An Efficient Bandit Algorithm for Realtime Multivariate Optimization
Hill, D. N., Nassif, H., Liu, Y., Iyer, A., & Vishwanathan, S. V. N. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098184 · Cited by 113 · cited 1×
“…of recommender systems (Thorburn et al., 2022), with many papers modeling recommendation as the problem of showing items to users that they are most likely to engage with, which is presumed to be the item they find the most rewarding (Li et al., 2010; Hill et al., 2017; McInerney et al., 2018).”
BackgroundOpen Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101
Self-citationThe logic of universalization guides moral judgment
Levine, S., Kleiman-Weiner, M., Schulz, L., Tenenbaum, J., & Cushman, F. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2014505117 · Cited by 88 · cited 1×
“…helpfulness involves estimating the goals of the agent being helped, and then judging whether the action aided in achieving that goal (Ullman et al., 2009); universalizability involves simulating what would happen if everyone took a particular action (Levine et al., 2020; Kwon et al., 2023b).”
BackgroundLazar, S., & Nelson, A. (2023) · Science · doi:10.1126/science.adi8982 · Cited by 77 · cited 2×
“Insofar as these AI systems are used to exercise power over others, they should also act in accordance with standards that are not just fair, but legitimate (Lazar, 2023; Stone and Mittelstadt, 2024).”
BackgroundEpistemic injustice and data science technologies
Symons, J., & Alvarado, R. (2022) · Synthese · doi:10.1007/s11229-022-03631-z · Cited by 65 · cited 1×
“Uniform preference aggregation may thus constitute a form of epistemic injustice (Fricker, 2007; Symons and Alvarado, 2022; Hull, 2023), which in turn leads to downstream injustice and harm.”
BackgroundScaling Laws for Reward Model Overoptimization
Gao, L., Schulman, J., & Hilton, J. (2022) · arXiv · doi:10.48550/arxiv.2210.10760 · Cited by 62
Risks from Learned Optimization in Advanced Machine Learning Systems
Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., & Garrabrant, S. (2019) · arXiv · doi:10.48550/arxiv.1906.01820 · Cited by 53 · cited 1×
“Similarly, worries about deceptive alignment (Hubinger et al., 2019) and goal misgeneralization (Di Langosco et al., 2022) are typically characterized as a mismatch between a learned utility function and the human-intended utility function; the solution is thus to ensure that the utility functions…”
BackgroundSelf-critiquing models for assisting human evaluators
Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., & Leike, J. (2022) · arXiv · doi:10.48550/arxiv.2206.05802 · Cited by 50
Occam's razor is insufficient to infer the preferences of irrational agents
Armstrong, S., & Mindermann, S. (2017) · arXiv · doi:10.48550/arxiv.1712.05812 · Cited by 41
Scalar reward is not enough: a response to Silver, Singh, Precup and Sutton (2021)
Vamplew, P., Smith, B. J., Källström, J., Ramos, G., Rădulescu, R., Roijers, D. M., Hayes, C. F., Heintz, F., Mannion, P., Libin, P. J. K., et al. (2022) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-022-09575-5 · Cited by 41 · cited 3×
“…discussion as to whether its component theses are warranted (Shah, 2018; Eckersley, 2018; Hadfield-Menell and Hadfield, 2018; Wentworth, 2019; Gabriel, 2020; Vamplew et al., 2022; Korinek and Balwit, 2022; Thornley, 2023), echoing similar debates in economics, decision theory, and philosophy.”
BackgroundMethodWong, L., Grand, G., Lew, A. K., Goodman, N. D., Mansinghka, V. K., Andreas, J., & Tenenbaum, J. B. (2023) · arXiv · doi:10.48550/arxiv.2306.12672 · Cited by 36
TALM: Tool Augmented Language Models
Parisi, A., Zhao, Y., & Fiedel, N. (2022) · arXiv · doi:10.48550/arxiv.2205.12255 · Cited by 35
Incomplete Contracting and AI Alignment
Hadfield-Menell, D., & Hadfield, G. K. (2018) · doi:10.2139/ssrn.3165793 · Cited by 28
Evaluating Cognitive Maps and Planning in Large Language Models with CogEval
Momennejad, I., Hasanbeig, H., Frujeri, F. V., Sharma, H., Jojic, N., Palangi, H., Ness, R., & Larson, J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3056 · Cited by 27
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Kim, H., Sclar, M., Zhou, X., Bras, R., Kim, G., Choi, Y., & Sap, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.890 · Cited by 27
AI Research Considerations for Human Existential Safety (ARCHES)
Critch, A., & Krueger, D. (2020) · arXiv · doi:10.48550/arxiv.2006.04948 · Cited by 25
Zhuang, S., & Hadfield-Menell, D. (2021) · arXiv · doi:10.48550/arxiv.2102.03896 · Cited by 22
Optimal Policies Tend To Seek Power
Turner, A. M., Smith, L., Shah, R., Critch, A., & Tadepalli, P. (2019) · arXiv · doi:10.48550/arxiv.1912.01683 · Cited by 18
How could we make a social robot? A virtual bargaining approach
Chater, N. (2023) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2022.0040 · Cited by 14 · cited 2×
“First, rather than aligning AI systems with norms that have actually been agreed upon, we could align them with norms that would hypothetically be agreed upon, in the spirit of virtual bargaining (Misyak et al., 2014; Chater, 2023).”
BackgroundAvoiding Side Effects in Complex Environments
Turner, A. M., Ratzlaff, N., & Tadepalli, P. (2020) · arXiv · doi:10.48550/arxiv.2006.06547 · Cited by 11
Estimating and Penalizing Induced Preference Shifts in Recommender Systems
Carroll, M., Dragan, A., Russell, S., & Hadfield-Menell, D. (2022) · arXiv · doi:10.48550/arxiv.2204.11966 · Cited by 11 · cited 1×
“Perhaps AI systems could optimize for a person’s current preferences, but this risks shifting or manipulating their preferences in undesirable ways (Ashton and Franklin, 2022; Carroll et al., 2022, 2024).”
BackgroundSelf-citationModels of human preference for learning reward functions
Knox, W. B., Hatgis-Kessell, S., Booth, S., Niekum, S., Stone, P., & Allievi, A. (2022) · arXiv · doi:10.48550/arxiv.2206.02231 · Cited by 8
AI Alignment with Changing and Influenceable Reward Functions
Carroll, M., Foote, D., Siththaranjan, A., Russell, S., & Dragan, A. (2024) · arXiv · doi:10.48550/arxiv.2405.17713 · Cited by 7 · cited 4×
“Avoiding such shifts would require delineating the kinds of value change that are legitimate or illegitimate (Ammann, 2023), but as Carroll et al. (2024) discuss, it is not obvious how to do so.”
BackgroundInfluentialSelf-citationIdentifiability in inverse reinforcement learning
Cao, H., Cohen, S. N., & Szpruch, L. (2021) · arXiv · doi:10.48550/arxiv.2106.03498 · Cited by 7
Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability
Akyürek, A. F., Akyürek, E., Choshen, L., Wijaya, D., & Andreas, J. (2024) · arXiv · doi:10.48550/arxiv.2401.08574 · Cited by 4
Settling the Reward Hypothesis
Bowling, M., Martin, J. D., Abel, D., & Dabney, W. (2022) · arXiv · doi:10.48550/arxiv.2212.10420 · Cited by 4
Power-seeking can be probable and predictive for trained agents
Krakovna, V., & Kramar, J. (2023) · arXiv · doi:10.48550/arxiv.2304.06528 · Cited by 2
Cited by
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732039 · Cited by 204
Sycophantic AI decreases prosocial intentions and promotes dependence
Cheng, M., Lee, C., Khadpe, P., Yu, S., Han, D., & Jurafsky, D. (2026) · Science · doi:10.1126/science.aec8352 · Cited by 185
Zhang, R., Li, H., Meng, H., Zhan, J., Gan, H., & Lee, Y.-C. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713429 · Cited by 123
ELEPHANT: Measuring and understanding social sycophancy in LLMs
Cheng, M., Yu, S., Lee, C., Khadpe, P., Ibrahim, L., & Jurafsky, D. (2025) · arXiv · doi:10.48550/arxiv.2505.13995 · Cited by 12
Multi-Agent Risks from Advanced AI
Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.14143 · Cited by 305
Self-citation
Metadata
- Authors
- Zhi-Xuan, T., Carroll, M., Franklin, M., & Ashton, H.
- Venue
- Philosophical Studies
- Published
- 2024-11-09
- Volume
- 182
- Issue
- 7
- Pages
- 1813-1863
- ISSN
- 0031-8116 (ISSN-L)
- 1573-0883
- OpenAlex
- W4404200702
- Cited by
- 34
- References
- 64
- FWCI
- 14.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.