CoChat
Sign Up
Cited by 187Ethics and Social Impacts of AILiterary and Philosophical StudiesPsychiatry, Mental Health, Neuroscience

Ethical Issues in Advanced Artificial Intelligence

Open access
Bostrom, Nick·2020·Machine Ethics and Robot Ethics
DOI 10.4324/9781003074991-7
Open this paper in CoChat — free accountOpen PDF

Abstract

The ethical issues related to the possible future creation of machines with general intellectual capabilities far outstripping those of humans are quite distinct from any ethical problems arising in current automation and information systems. Such superintelligence would not be just another technological development; it would be the most important invention ever made, and would lead to explosive progress in all scientific and technological fields, as the superintelligence would conduct research with superhuman efficiency. To the extent that ethics is a cognitive pursuit, a superintelligence could also easily surpass humans in the quality of its moral thinking. However, it would be up to the designers of the superintelligence to specify its original motivations. Since the superintelligence may become unstoppably powerful because of its intellectual superiority and the technologies it could develop, it is crucial that it be provided with human-friendly motivations. This paper surveys some of the unique ethical issues in creating superintelligence, and discusses what motivations we ought to give a superintelligence, and introduces some cost-benefit considerations relating to whether the development of superintelligent machines ought to be accelerated or retarded.

Integrity signals

Registry

Crossref · 10.4324/9781003074991-7

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 5 cited sources checked · 0 supported · 0 flagged

  • Interpretationreasoning

    To the extent that ethics is a cognitive pursuit, a superintelligence could surpass humans in the quality of its moral thinking.

    “To the extent that ethics is a cognitive pursuit, a superintelligence could also easily surpass humans in the quality of its moral thinking.”
    • Also stated in: § 3. SUPERINTELLIGENT MORAL THINKING · hedged
      “To the extent that ethics is a cognitive pursuit, a superintelligence could do it better than human thinkers.”
    Quote in the paperNot a substantial claim
  • Interpretationnone given

    There is currently no good ground for assigning a negligible probability to the hypothesis that superintelligence will be created within the lifespan of some people alive today.

    “It might turn out to take much longer, but there seems currently to be no good ground for assigning a negligible probability to the hypothesis that superintelligence will be created within the lifespan of some people alive today.”
    Quote in the paperNot a substantial claim
  • Interpretationreasoning

    Superintelligence will lead to more advanced superintelligence.

    “Superintelligence will lead to more advanced superintelligence.This results both from the improved hardware that a superintelligence could create, and also from improvements it could make to its own source code.”
    Quote in the paperNot a substantial claim
  • Premisecitation

    The foreseeable technologies that a superintelligence is likely to develop include mature molecular manufacturing.

    “The foreseeable technologies that a superintelligence is likely to develop include mature molecular manufacturing, whose applications are wide-ranging:”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Drexler 1986

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Interpretationreasoning

    The marginal cost of creating an additional copy of an upload or an artificial intelligence after the first one has been built is near zero.

    “Hardware aside, the marginal cost of creating an additional copy of an upload or an artificial intelligence after the first one has been built is near zero.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 30
  • May combine two claims. 13

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Superintelligence would not be just another technological development; it would be the most important invention ever made.

    “Such superintelligence would not be just another technological development; it would be the most important invention ever made, and would lead to explosive progress in all scientific and technological fields, as the superintelligence would conduct research with superhuman efficiency.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    Superintelligence is radically different from other technologies and is not just another tool that will add incrementally to human capabilities.

    “A prerequisite for having a meaningful discussion of superintelligence is the realization that superintelligence is not just another technology, another tool that will add incrementally to human capabilities.Superintelligence is radically different.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Superintelligence would lead to explosive progress in all scientific and technological fields.

    “Such superintelligence would not be just another technological development; it would be the most important invention ever made, and would lead to explosive progress in all scientific and technological fields, as the superintelligence would conduct research with superhuman efficiency.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    A superintelligence may become unstoppably powerful because of its intellectual superiority and the technologies it could develop.

    “Since the superintelligence may become unstoppably powerful because of its intellectual superiority and the technologies it could develop, it is crucial that it be provided with human-friendly motivations.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: It would be up to the designers of the superintelligence to specify its original motivations.

  • May combine two claims.

    Interpretationreasoning

    Artificial minds could quickly come to exist in great numbers, although it is possible that efficiency would favor concentrating computational resources in a single super-intellect.

    “Artificial minds could therefore quickly come to exist in great numbers, although it is possible that efficiency would favor concentrating computational resources in a single super-intellect.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Superintelligence may be the last invention humans ever need to make.

    “Superintelligence may be the last invention humans ever need to make.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Technological progress in all other fields will be accelerated by the arrival of advanced artificial intelligence.

    “Technological progress in all other fields will be accelerated by the arrival of advanced artificial intelligence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    A superintelligence would conduct research with superhuman efficiency.

    “Such superintelligence would not be just another technological development; it would be the most important invention ever made, and would lead to explosive progress in all scientific and technological fields, as the superintelligence would conduct research with superhuman efficiency.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The ethical issues related to the possible future creation of machines with general intellectual capabilities far outstripping those of humans are quite distinct from any ethical problems arising in current automation and information systems.

    “The ethical issues related to the possible future creation of machines with general intellectual capabilities far outstripping those of humans are quite distinct from any ethical problems arising in current automation and information systems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Any technology that we can currently foresee will be speedily developed by the first superintelligence, along with many other technologies of which we are as yet clueless.

    “It is likely that any technology that we can currently foresee will be speedily developed by the first superintelligence, no doubt along with many other technologies of which we are as yet clueless.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Given a superintelligence's intellectual superiority, it would be much better at doing scientific research and technological development than any human, and possibly better even than all humans taken together.

    “Given a superintelligence's intellectual superiority, it would be much better at doing scientific research and technological development than any human, and possibly better even than all humans taken together.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    A superintelligence could create opportunities for us to vastly increase our own intellectual and emotional capabilities, and it could assist us in creating a highly appealing experiential world.

    “A superintelligence could also create opportunities for us to vastly increase our own intellectual and emotional capabilities, and it could assist us in creating a highly appealing experiential world in which we could live lives devoted to in joyful game-playing, relating to each other, experiencing, personal growth, and to living closer to our ideals.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The risks in developing superintelligence include the risk of failure to give it the supergoal of philanthropy.

    “The risks in developing superintelligence include the risk of failure to give it the supergoal of philanthropy.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    It is hard to think of any problem that a superintelligence could not either solve or at least help us solve.

    “It is hard to think of any problem that a superintelligence could not either solve or at least help us solve.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Quite possibly, the first superintelligence would be unrivalled: it would be able to bring about almost any possible outcome and to thwart any attempt to prevent the implementation of its top goal.

    “Quite possibly, it would be unrivalled: it would be able to bring about almost any possible outcome and to thwart any attempt to prevent the implementation of its top goal.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    It seems perfectly possible to have a superintelligence whose sole goal is something completely arbitrary, such as to manufacture as many paperclips as possible, and who would resist with all its might any attempt to alter this goal.

    “It also seems perfectly possible to have a superintelligence whose sole goal is something completely arbitrary, such as to manufacture as many paperclips as possible, and who would resist with all its might any attempt to alter this goal.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    The transition from a state where we have a roughly human-level artificial intelligence to a state where we have fullblown superintelligence, with revolutionary applications, may be very rapid, perhaps a matter of days rather than years.

    “That is, the transition from a state where we have a roughly human-level artificial intelligence to a state where we have fullblown superintelligence, with revolutionary applications, may be very rapid, perhaps a matter of days rather than years.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    If a superintelligence starts out with a friendly top goal, then it can be relied on to stay friendly, or at least not to deliberately rid itself of its friendliness.

    “If a superintelligence starts out with a friendly top goal, however, then it can be relied on to stay friendly, or at least not to deliberately rid itself of its friendliness.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: It is possible to build a superintelligence with an explicit motivational architecture.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    A superintelligence whose top goal is the manufacturing of paperclips could start transforming first all of earth and then increasing portions of space into paperclip manufacturing facilities.

    “This could result, to return to the earlier example, in a superintelligence whose top goal is the manufacturing of paperclips, with the consequence that it starts transforming first all of earth and then increasing portions of space into paperclip manufacturing facilities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    If we get nanotechnology first, we will have to face both the risks from nanotechnology and, if these risks are survived, also the risks from superintelligence.

    “If, on the other hand, we get nanotechnology first, we will have to face both the risks from nanotechnology and, if these risks are survived, also the risks from superintelligence.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    It appears much harder to get from where we are now to human-level artificial intelligence than to get from there to superintelligence.

    “It appears much harder to get from where we are now to human-level artificial intelligence than to get from there to superintelligence.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    A superintelligence could give us indefinite lifespan, either by stopping and reversing the aging process through the use of nanomedicine, or by offering us the option to upload ourselves.

    “Additionally, a superintelligence could give us indefinite lifespan, either by stopping and reversing the aging process through the use of nanomedicine 7 , or by offering us the option to upload ourselves.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Freitas Jr. 1999

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationnone given

    In humans, there is often no obvious way to identify what our top goal is; we might not even have one.

    “In humans, with our complicated evolved mental ecology of state-dependent competing drives, desires, plans, and ideals, there is often no obvious way to identify what our top goal is; we might not even have one.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The setting up of initial conditions, and in particular the selection of a top-level goal for the superintelligence, is of the utmost importance.

    “On the contrary, the setting up of initial conditions, and in particular the selection of a top-level goal for the superintelligence, is of the utmost importance.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: It would be up to the designers of the superintelligence to specify its original motivations.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    We could delegate many investigations and decisions to the superintelligence.

    “There are therefore many questions that we would not need to answer ourselves if we had or were about to get superintelligence; we could delegate many investigations and decisions to the superintelligence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    A superintelligence could realize a state of affairs that we might now judge as desirable but which in fact turns out to be a false utopia, in which things essential to human flourishing have been irreversibly lost.

    “More subtly, it could result in a superintelligence realizing a state of affairs that we might now judge as desirable but which in fact turns out to be a false utopia, in which things essential to human flourishing have been irreversibly lost.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    Artificial intellects may find it easy to guard against some kinds of human error and bias, while at the same time being at increased risk of other kinds of mistake that not even the most hapless human would make.

    “Artificial intellects may find it easy to guard against some kinds of human error and bias, while at the same time being at increased risk of other kinds of mistake that not even the most hapless human would make.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    When formulating a goal for the superintelligence, it would not always be necessary to give a detailed, explicit definition of this goal.

    “When formulating a goal for the superintelligence, it would not always be necessary to give a detailed, explicit definition of this goal.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Disease, poverty, environmental destruction, and unnecessary suffering of all kinds are things that a superintelligence equipped with advanced nanotechnology would be capable of eliminating.

    “Disease, poverty, environmental destruction, unnecessary suffering of all kinds: these are things that a superintelligence equipped with advanced nanotechnology would be capable of eliminating.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Another way for failure to give superintelligence the supergoal of philanthropy to happen is that a well-meaning team of programmers make a big mistake in designing its goal system.

    “Another way for it to happen is that a well-meaning team of programmers make a big mistake in designing its goal system.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    A superintelligence could kill off all other agents, persuade them to change their behavior, or block their attempts at interference.

    “It could kill off all other agents, persuade them to change their behavior, or block their attempts at interference.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Our entire future may hinge on how we solve these problems.

    “Our entire future may hinge on how we solve these problems.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The overall risk seems to be minimized by implementing superintelligence, with great care, as soon as possible.

    “The overall risk seems to be minimized by implementing superintelligence, with great care, as soon as possible.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: It is possible to endow a superintelligence with human-friendly motivations.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    General superintelligence would be capable of independent initiative and of making its own plans, and may therefore be more appropriately thought of as an autonomous agent.

    “While specialized superintelligences that can think only about a restricted set of problems may be feasible, general superintelligence would be capable of independent initiative and of making its own plans, and may therefore be more appropriately thought of as an autonomous agent.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    Emergence of superintelligence may be sudden.

    “Emergence of superintelligence may be sudden.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    One way in which failure to give superintelligence the supergoal of philanthropy could happen is that the creators decide to build it so that it serves only a select group of humans, rather than humanity in general.

    “One way in which this could happen is that the creators of the superintelligence decide to build it so that it serves only this select group of humans, rather than humanity in general.”
    Quote in the paperNot a substantial claim
36 of 36

  • The paper relies on this without support.

    It would be up to the designers of the superintelligence to specify its original motivations.

    “However, it would be up to the designers of the superintelligence to specify its original motivations.”

  • The paper relies on this without support.

    It is possible to endow a superintelligence with human-friendly motivations.

    “It seems that the best way to ensure that a superintelligence will have a beneficial impact on the world is to endow it with philanthropic values.”

  • The paper relies on this without support.

    It is possible to build a superintelligence with an explicit motivational architecture.

    “And this is a good reason for us to build the superintelligence with such an explicit motivational architecture.”
3 of 3

Cites

Flagged first, then by citations

1 of the 9 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Bostrom, N.
Corresponding author
Bostrom, Nick
Venue
Machine Ethics and Robot Ethics
Publisher
Routledge
Published
2020-09-10
Pages
69-75
DOI
10.4324/9781003074991-7
OpenAlex
W3081243002
Cited by
187
References
9
Citations by year
2026: 11 · 2025: 21 · 2024: 20 · 2023: 35 · 2022: 17 · 2021: 21 · 2020: 23 · 2019: 12 · 2018: 12 · 2017: 5 · 2016: 3 · 2012: 5
FWCI
98.53
Type
book chapter
Language
en
Open access
Locations
  • Routledge eBooks (publishedVersion)
  • PhilPapers (PhilPapers Foundation) (submittedVersion)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Political Science and International Relations › Literary and Philosophical Studies
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
Keywords
superintelligence, AI ethics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review