The DOI matches its Crossref record; 99 of the 101 references it cites were found in a registry or an index; 5 of the 66 sources its claims cite were checked for support.
Artificial Intelligence, Values, and Alignment
Open accessAbstract
This paper looks at philosophical questions that arise in the context of AI alignment. It defends three propositions. First, normative and technical aspects of the AI alignment problem are interrelated, creating space for productive engagement between people working in both domains. Second, it is important to be clear about the goal of alignment. There are significant differences between AI that aligns with instructions, intentions, revealed preferences, ideal preferences, interests and values. A principle-based approach to AI alignment, which combines these elements in a systematic way, has considerable advantages in this context. Third, the central challenge for theorists is not to identify ‘true’ moral principles for AI; rather, it is to identify fair principles for alignment that receive reflective endorsement despite widespread variation in people’s moral beliefs. The final part of the paper explores three ways in which fair principles for AI alignment could potentially be identified.
Overview
CoChat Summary
## TL;DR
This article argues that AI alignment is both a technical and a normative problem, so the choice of values cannot be separated from how systems are built. It also argues that alignment should target fair, publicly justifiable principles rather than a single “true” moral theory, because people hold diverse reasonable beliefs about value.
## Key findings
- The paper says the technical and normative parts of alignment are “not orthogonal but rather related in important ways,” and that “the way we build AI is likely to influence the values we are able to load.”
- It distinguishes several possible alignment targets—instructions, intentions, revealed preferences, informed preferences, interests/well-being, and values—and argues that none of the earlier targets is sufficient on its own.
- On revealed preferences, the paper notes a practical problem of “degeneracy”: at any moment there is “a large set of reward functions for which an observed behaviour is optimal,” making reliable inference hard; it also says revealed preferences can misfire in self-harm, harm to others, and adapted preferences.
- It reports that Jobin et al. (2019) analyzed 84 AI ethics documents and found a “global convergence” around five principles: transparency, justice and fairness, non-maleficence, responsibility, and privacy.
## Methods
- Conceptual/philosophical argument, not an empirical study.
- The paper reviews AI alignment concepts from machine learning and reinforcement learning, then maps them onto normative options from moral and political philosophy.
- It develops three candidate routes for selecting fair alignment principles: overlapping consensus / global public morality and human rights, hypothetical agreement behind a veil of ignorance, and social choice / democratic aggregation.
## Limitations
- The paper states that there is “no way to ‘bracket out’ normative questions altogether,” so technical methods alone cannot solve alignment.
- It notes that existing AI ethics principles are often too abstract and vague to be action-guiding, and that apparent consensus may hide deep disagreement; this is a limitation of the field the paper discusses, not a study limitation.
- Evident from the text: the article is largely theoretical and does not present new data, experiments, or a formal proof of the proposed principles.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 0 Not valid · 49 Not checked · 82 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To really grasp the intention behind instructions, AI may require a complete model of human language and interaction, including an understanding of the culture, institutions, and practices that allow people to understand the implied meaning of terms (HadfieldMenell and Hadfield 2018).”
What the sources say
- Partly supportedcites Hadfield-Menell and Hadfield 2018 sourceExperimental — accuracy not yet measured
Section 5 (page not available in our copy) says that interpreting implied terms depends on cultural norms and institutions, and that aligned AI needs tools to read and predict human normative structures. It does not say AI requires a complete model of human language and interaction; that requirement is stronger and broader than the paper's claim. (LLM, conf=0.84)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For example, if powerful AI systems function at super-human speed, which seems likely, then it may not be possible to provide them with immediate and continuous direction in this way (Russell etal. 2015; Soares 2014).”
What the sources say
- Partly supportedcites Russell et al. 2015 sourceExperimental — accuracy not yet measured
The Control section says that capable autonomous AI systems may be difficult to keep under meaningful human control, and discusses the need for technical work to maintain that control. It does not say that super-human operating speed is likely or that speed would prevent immediate, continuous direction through expressed intentions; those are the claim's more specific premises and mechanism. (LLM, conf=0.83)
- Not matchedcites Soares 2014
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“these codes have tended to have a pronounced geographical focus, arising from actors who are located in economically developed countries.”
What the sources say
- Supportedcites Jobin et al. 2019 source
The Results report a significant concentration of AI ethics guidelines among issuers in more economically developed countries, particularly the USA, UK and EU. The Discussion likewise says these countries are shaping the debate more than others and notes the underrepresentation of several other regions. (LLM, conf=0.94)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Among other things, these authors found evidence of a 'global convergence' around five ethical principles for AI which were: transparency, justice and fairness, non-maleficence, responsibility, and privacy.”
What the sources say
- Supportedcites Jobin et al. 2019 source
The Discussion says all five named principles appear in more than half of the guidelines and that this could indicate developing global convergence. The claim says they found evidence of convergence, not universal agreement, which matches that qualified finding. (LLM, conf=0.94)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
In the paper
“ordinary moral disagreement between groups has been a persistent source of conflict in human life”
What the sources say
- Not matchedcites Haidt 2012
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
61 more claims
Create a free account to see all 136 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 57
- It may state two separate claims as one. 23
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“As discussed in detail in Gabriel (2020), there are subtle differences between aligning to instructions, intentions, revealed preferences, ideal preferences, interests, and values.”
From Training language models to follow instructions with human feedback (2022) · cites this paper as Gabriel (2020)
In the citing paper
“Bias and fairness are also critical issues [30, 20]. LLMs learn from the data they are trained on; if this data is inherently biased, these biases could be replicated in AI predictions, leading to unfair outcomes and exacerbating existing healthcare disparities.”
From The impact of ChatGPT and LLMs on medical imaging stakeholders: Perspectives and use cases (2023) · cites this paper as [30]
Citations
Cites
Principles of Biomedical Ethics
(2012) · Thieves of Virtue · doi:10.7551/mitpress/9079.003.0009 · Cited by 13,453
A Theory of Justice: Original Edition
Rawls, J. (1971) · doi:10.4159/9780674042605 · Cited by 9,496
Rawls, J. (2018) · Princeton Readings in Political Thought · doi:10.2307/j.ctv19fvzzk.68 · Cited by 9,417
In a Different Voice: Psychological Theory and Women’s Development
GILLIGAN, C. (1993) · doi:10.4159/9780674037618 · Cited by 9,127
The global landscape of AI ethics guidelines
Jobin, A., Ienca, M., & Vayena, E. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0088-2 · Cited by 6,220
Justice as Fairness: A Restatement
Rawls, J. (2001) · doi:10.2307/j.ctv31xf5v0 · Cited by 5,761
Rawls, J. (1958) · The Philosophical Review · doi:10.2307/2182612 · Cited by 5,565
Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., Luetge, C., Madelin, R., Pagallo, U., Rossi, F., et al. (2018) · Minds and Machines · doi:10.1007/s11023-018-9482-5 · Cited by 4,774
The righteous mind: why good people are divided by politics and religion
(2012) · Choice Reviews Online · doi:10.5860/choice.49-7185 · Cited by 4,348
(1993) · Oxford University Press eBooks · doi:10.1093/0198287976.001.0001 · Cited by 4,239
Research works printed but not linked: 23
- Asilomar, A.I. (2018, October). Principles (2017). In Principles developed in conjunction with the 2017 Asilomar conference [Benevolent AI 2017].
- Asimov, I. (2004). I, robot (Vol. 1). Spectra.
- Blackburn, S. (2001). Ruling passions: An essay in practical reasoning. Oxford: Oxford University Press.
- Bostrom, N. (2016). Superintelligence: Paths, dangers, strategies. Oxford: Oxford University Press.
- Bratman, M. (1987). Intention, plans, and practical reason (Vol. 10). Cambridge, MA: Harvard University Press.
- Dennett, D. (2003). Freedom Evolves. New York: Penguin Books.
- Dworkin, R. (1981). What is equality? Part 1: Equality of welfare. in Philosophy & public affairs, (pp. 185–246).
- Dworkin, R. (1984). Rights as Trumps. Arguing Law (pp. 335–44).
- Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., et al. (2018). De novo structure prediction with deeplearning based scoring. Annual Review of Biochemistry, 77(363–382), 6.
- MacIntyre, A. (2013). After virtue. London: A&C Black.
- Mill, J.S. (1859/1998). On liberty and other essays. Oxford: Oxford University Press.
- Nagel, T. (1989). The view from nowhere. Oxford: Oxford University Press.
- Nozick, R. (1974). Anarchy, state, and utopia. New York: Basic Books.
- Parfit, D. (2011). On what matters (Vol. 1). Oxford: Oxford University Press.
- Rorty, R. (1993). Human rights, rationality, and sentimentality. Wronging Rights Philos. Chall. Hum. Rights (pp. 1–34).
- Russell, S. (2019). Human Compatible: AI and the Problem of Control. Bristol: Allen Lane.
- Russell, S., & Norvig, P. (2010). Artificial intelligence: A modern approach (3rd ed.). Upper Saddle River: Prentice Hall.
- Shue, H. (1996). Basic rights: Subsistence, affluence, and US foreign policy. Princeton: Princeton University Press.
- Smith, M. R. (1994). The Moral Problem. New York: Malden Blackwell.
- Soares, N. (2014). The Value Learning Problem, Technical Report. Berkeley, CA: Machine Intelligence Research Institute.
- Taylor, J. (2016). Quantilizers: A Safer Alternative to Maximizers for Limited Optimization, in Presented at the Workshops at the Thirtieth AAAI Conference on Artificial Intelligence.
- UN Secretary General. (2019). The age of digital interdependence, Report of the UN Secretary General’s High-Level Panel on Digital Cooperation. Geneva: Un Secretary General.
- Vaillant, G. E. (2008). Aging well: Surprising guideposts to a happier life from the landmark study of adult development. Brown: Little.
Cited by
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence
Mohamed, S., Png, M.-T., & Isaac, W. (2020) · Philosophy & Technology · doi:10.1007/s13347-020-00405-8 · Cited by 912
“that were posed in the opening of whose values and goals are represented, and who is empowered to articulate and embed these values—introducing discussions of utilitarian, Kantian and volitional views on behaviour, and on the prevention and avoidance of undesirable and unintended consequences (Gabriel 2020).”
BackgroundTrust in AI: progress, challenges, and future directions
Afroogh, S., Akbari, A., Malone, E., Kargar, M., & Alambeigi, H. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-04044-8 · Cited by 507
“…others worry about the so-called “ alignment problem ” of how we, as a society, and researchers working on arti fi cial intelligence research could ensure that an AGI ’ s interests and values would align with our own (Abbass, 2019c; Bostrom, 2014; Gabriel, 2020; Prasad, 2019; Taylor et al., 2016).”
InfluentialEthical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477
Self-citation
Metadata
- Authors
- Gabriel, I.
- Affiliations
- Google DeepMind (United Kingdom) (GB)
- Corresponding author
- Gabriel, Iason
- ORCID
- Gabriel, Iason
- Venue
- Minds and Machines
- Published
- 2020-09
- Volume
- 30
- Issue
- 3
- Pages
- 411-437
- ISSN
- 0924-6495 (ISSN-L)
- 1572-8641
- OpenAlex
- W3002093512
- Cited by
- 896
- References
- 91
- Citations by year
- 2026: 274 · 2025: 229 · 2024: 164 · 2023: 93 · 2022: 64 · 2021: 61 · 2020: 11
- FWCI
- 77.92
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 30 d)
- Locations
- Minds and Machines (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Arts and Humanities › Philosophy › Philosophical Ethics and Theory
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Keywords
- AI alignment, AI ethics, moral values, revealed preference, interest, FAIR principles, moral pluralism
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Artificial Intelligence, Values, and Alignment. https://app.cochat.ai/papers/10.1007/s11023-020-09539-2
- The paper
- Gabriel, I. (2020). Artificial Intelligence, Values, and Alignment. Minds and Machines. https://doi.org/10.1007/s11023-020-09539-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.