Utility function security in artificially intelligent agents
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
The notion of ‘wireheading’, or direct reward centre stimulation of the brain, is a well-known concept in neuroscience. In this paper, we examine the corresponding issue of reward (utility) function integrity in artificially intelligent machines. We survey the relevant literature and propose a number of potential solutions to ensure the integrity of our artificial assistants. Overall, we conclude that wireheading in rational self-improving optimisers above a certain capacity remains an unsolved problem despite opinion of many that such machines will choose not to wirehead. A relevant issue of literalness in goal setting also remains largely unsolved and we suggest that the development of a non-ambiguous knowledge transfer language might be a step in the right direction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On computable numbers, with an application to the entscheidungsproblem
Turing, A. M. (1937) · Proceedings of the London Mathematical Society · doi:10.1112/plms/s2-42.1.230 · Cited by 8,283
Die gegenw�rtige Situation in der Quantenmechanik
Schr�dinger, E. (1935) · Die Naturwissenschaften · doi:10.1007/bf01491891 · Cited by 3,392
Olds, J., & Milner, P. (1954) · Journal of Comparative and Physiological Psychology · doi:10.1037/h0058775 · Cited by 3,176 · cited 1×
“The term ‘wirehead’ traces its origins to intracranial self-stimulation experiments performed by James Olds and Peter Milner on rats in the 1950s (Olds & Milner, 1954).”
BackgroundClasses of recursively enumerable sets and their decision problems
Rice, H. G. (1953) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1953-0053041-6 · Cited by 869 · cited 2×
“Given a possibly infinite number of quantum particles correctly pre-computing future states of the whole universe would violate many established scientific laws and intuitions (de Blanc, 2007, 2009; Rice, 1953; Turing, 1936) including the principle of computational irreducibility (Wolfram, 2002).”
BackgroundWolfram, & Gad-el-Hak (2003) · Applied Mechanics Reviews · doi:10.1115/1.1553433 · Cited by 760
Universal Artificial Intellegence
Hutter, M. (2005) · Texts in Theoretical Computer Science An EATCS Series · doi:10.1007/b138233 · Cited by 523 · cited 1×
“Intelligent machines may realise that they can increase their rewards by psychologically or physically manipulating their human masters, a strongly undesirable consequence (Hutter, 2010).”
BackgroundArtificial Intelligence as a Positive and Negative Factor in Global Risk
Yudkowsky, E. (2008) · Global Catastrophic Risks · doi:10.1093/oso/9780198570509.003.0021 · Cited by 431
Cited by
Rulers of the world, unite! The challenges and opportunities of artificial intelligence
Kaplan, A., & Haenlein, M. (2020) · Business Horizons · doi:10.1016/j.bushor.2019.09.003 · Cited by 718
AI, agency and responsibility: the VW fraud case and beyond
Johnson, D. G., & Verdicchio, M. (2019) · AI & SOCIETY · doi:10.1007/s00146-017-0781-9 · Cited by 79
“…to technological artifacts has become an object of heated debate in the context of AI research because some AI researchers ascribe to programs the type of agency traditionally associated with humans (Allen and Wallach 2012; Bostrom 2014; Malle and Scheutz 2015; Omohundro 2016; Yampolskiy 2016).”
BackgroundOn the Limits of Recursively Self-Improving AGI
Yampolskiy, R. V. (2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-21365-1_40 · Cited by 23
“Yampolskiy reviews a number of related problems in rational self-improving optimizers, above a certain capacity, and concludes, that despite opinion of many, such machines will choose to “wirehead” [29].”
BackgroundSelf-citationZhang, J., Yuan, B., & Zhang, Q. (2026) · Entropy · doi:10.3390/e28090951 · Cited by 2
“This gives formal content to the “leakproofing” problem [26] and the utility-function security problem [81].”
Metadata
- Authors
- Yampolskiy, R. V.
- Venue
- Journal of Experimental & Theoretical Artificial Intelligence
- Publisher
- Informa UK Limited
- Published
- 2014-04-08
- Volume
- 26
- Issue
- 3
- Pages
- 373-389
- ISSN
- 0952-813X (ISSN-L)
- 1362-3079
- OpenAlex
- W2093399934
- Cited by
- 47
- References
- 7
- FWCI
- 13.86
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Utility function security in artificially intelligent agents. https://app.cochat.ai/papers/10.1080/0952813x.2014.895114
- The paper
- Yampolskiy, R. V. (2014). Utility function security in artificially intelligent agents. Journal of Experimental & Theoretical Artificial Intelligence. https://doi.org/10.1080/0952813x.2014.895114
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.