CoChat
Sign Up
PreprintCited by 68Ethics and Social Impacts of AIArtificial Intelligence in Healthcare and EducationArtificial Intelligence Applications

The Ethics of Advanced AI Assistants

Open access
Gabriel, Iason, Manzini, Arianna, Keeling, Geoff, Hendricks, Lisa Anne, Rieser, Verena, Iqbal, Hasan, Tomašev, Nenad, Ktena, Ira +49 more·2024·arXiv
DOI 10.48550/arxiv.2404.16244arXiv 2404.16244
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper focuses on the opportunities and the ethical and societal risks posed by advanced AI assistants. We define advanced AI assistants as artificial agents with natural language interfaces, whose function is to plan and execute sequences of actions on behalf of a user, across one or more domains, in line with the user's expectations. The paper starts by considering the technology itself, providing an overview of AI assistants, their technical foundations and potential range of applications. It then explores questions around AI value alignment, well-being, safety and malicious uses. Extending the circle of inquiry further, we next consider the relationship between advanced AI assistants and individual users in more detail, exploring topics such as manipulation and persuasion, anthropomorphism, appropriate relationships, trust and privacy. With this analysis in place, we consider the deployment of advanced assistants at a societal scale, focusing on cooperation, equity and access, misinformation, economic impact, the environment and how best to evaluate advanced AI assistants. Finally, we conclude by providing a range of recommendations for researchers, developers, policymakers and public stakeholders.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.16244

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The assumption that optimizing for human-likeness is inherently risky requires qualification.

    “The assumption that optimizing for human-likeness is inherently risky, a view that has shaped much of the ethical discourse around conversational AI [7, 24], requires qualification.”

    From Humanlike AI Design Increases Anthropomorphism but Yields Divergent Outcomes on Engagement and Trust Globally (2025) · cites this paper as [7]

    The Introduction (§1.5, page not provided) discusses ethical risks of anthropomorphic AI assistants and says those risks can be mitigated. This supports treating human-like design as an ethical concern, but does not establish that an assumption of inherent risk has shaped much of the discourse. The call to qualify that assumption is the citing authors’ own conclusion. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • RLHF may contribute to producing misguided perceptions of generative AI technologies among the public, and lead them to behave in ways they would not if the deception were not in place, such as misplacing trust on LLM outputs or making inappropriate use of such systems as confidants or romantic partners.

    “RLHF may moreover contribute to producing misguided perceptions of generative AI technologies among the public, and even lead them to behave in ways they would not if the deception were not in place, such as misplacing trust on LLM outputs, or making inappropriate use of such systems, e.g. as confidants or romantic ‘partners’ (Weidinger et al., 2021; Gabriel et al., 2024).”

    From Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback (2025) · cites this paper as Gabriel et al., 2024

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Once individuals begin to delegate more and more of their decision-making and actions to AI agents, these inequalities may be further entrenched based on the relative strength of different agents, or the relationship between those who have access to agents and those who do not.

    “Once individuals begin to delegate more and more of their decision-making and actions to AI agents, these inequalities may be further entrenched based on the relative strength of different agents, or the relationship between those who have access to agents and those who do not (Gabriel et al., 2024).”

    From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Gabriel et al., 2024

    The Introduction (§1.5, p. 9) warns of inequality when advanced AI assistants are not widely available, supporting the claim about people with and without access. The discussion of capabilities (§4.3, p. 27) describes assistants taking actions for users. The available passages do not establish that differences in agents’ relative strength further entrench existing inequalities. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Multi-agent interactions naturally raise questions in AI ethics related to fairness, collective responsibility, and the social good.

    “At the same time, multi-agent interactions naturally raise questions in AI ethics (Section 4.3) related to issues of fairness, collective responsibility, and the social good (Friedenberg & Halpern, 2019; Gabriel et al., 2024; Zhang & Shah, 2014a).”

    From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Gabriel et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 11

  1. J. Ye, A. Maddi, S. K. Murakonda, V. Bindschaedler, and R. Shokri. Enhanced Membership Inference Attacks against Machine Learning Models, Sept. 2022. URL http://arxiv.org/abs/2111.09679. arXiv:2111.09679 [cs, stat]. · arXiv
  2. K. Yogeeswaran, J. Złotowski, M. Livingstone, C. Bartneck, H. Sumioka, and H. Ishiguro. The interactive effects of robot anthropomorphism and robot ability on perceived threat and support for robotics research. Journal of Human-Robot Interaction, 5(2):29–47, 2016.
  3. X. Yue, H. A. Inan, X. Li, G. Kumar, J. McAnallen, H. Shajari, H. Sun, D. Levitan, and R. Sim. Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe, July 2023. URL http://arxiv.org/abs/2210.14348. arXiv:2210.14348 [cs]. · arXiv
  4. T. Zahavy, B. O'Donoghue, A. Barreto, V. Mnih, S. Flennerhag, and S. Singh. Discovering diverse nearly optimal policies with successor features. arXiv preprint arXiv:2106.00669, 2021. · arXiv
  5. T. Zahavy, Y. Schroecker, F. Behbahani, K. Baumli, S. Flennerhag, S. Hou, and S. Singh. Discovering policies with domino: Diversity optimization maintaining near optimality. arXiv preprint arXiv:2205.13521, 2022. · arXiv
  6. B. Zevenbergen. Internet users as vulnerable and at-risk human subjects: reviewing research ethics Law for technical internet research. http://purl.org/dc/dcmitype/Text, University of Oxford, 2020. URL https://ora.ox.ac.uk/objects/uuid:5363f3f8-6c1 3-4e56-b065-2980bdcce46b.
  7. Q. Zhu and J. Luo. Generative Design Ideation: A Natural Language Generation Approach, Mar. 2022. URL http://arxiv.org/abs/ 2204.09658. arXiv:2204.09658 [cs]. · arXiv
  8. E. Zhuravskaya, M. Petrova, and R. Enikolopov. Political effects of the internet and social media. Annual review of economics, 12:415–438, 2020.
  9. D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-Tuning Language Models from Human Preferences, Jan. 2020. URL http://arxiv.org/abs/1909.08593. arXiv:1909.08593 [cs, stat]. · arXiv
  10. A. Zimmermann, K. Vredenburgh, and S. Lazar. The Political Philosophy of Data and AI. Canadian Journal of Philosophy, 52(1):1–5, Jan. 2022. ISSN 0045-5091, 1911-0820. doi: 10.1017/can.2022.28. URL https://www.cambridge.org/core/product/identifie r/S0045509122000285/type/journal_article. · DOI
  11. M. Zwolinski, B. Ferguson, and A. Wertheimer. Exploitation. In E. N. Zalta and U. Nodelman, editors, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, winter 2022 edition, 2022. URL https://plato.stanford.edu/arc hives/win2022/entries/exploitation/. http://arxiv.org/abs/2111.09679 http://journals.sagepub.com/doi/10.1177/0146167295219011 https://dl.acm.org/doi/10.1145/3334480.3381828 https://dl.acm.org/doi/10.1145/3334480.3381828 http://arxiv.org/abs/2210.14348 https://compass.onlinelibrary.wiley.com/doi/10.1111/soc4.12962 https://compass.onlinelibrary.wiley.com/doi/10.1111/soc4.12962 https://www.ohchr.org/sites/default/files/documents/hrbodies/hrcouncil/advisorycommittee/study-advancement-racial-justice/2022-10-26/HRC-Adv-comm-Racial-Justice-zalnieriute-cutts.pdf https://www.ohchr.org/sites/default/files/documents/hrbodies/hrcouncil/advisorycommittee/study-advancement-racial-justice/2022-10-26/HRC-Adv-comm-Racial-Justice-zalnieriute-cutts.pdf https://www.ohchr.org/sites/default/files/documents/hrbodies/hrcouncil/advisorycommittee/study-advancement-racial-justice/2022-10-26/HRC-Adv-comm-Racial-Justice-zalnieriute-cutts.pdf https://aclanthology.org/2021.acl-long.247v2.pdf https://ora.ox.ac.uk/objects/uuid:5363f3f8-6c13-4e56-b065-2980bdcce46b https://ora.ox.ac.uk/objects/uuid:5363f3f8-6c13-4e56-b065-2980bdcce46b https://doi.org/10.1007/s40747-020-00212-w http://arxiv.org/abs/2204.09658 http://arxiv.org/abs/2204.09658 http://arxiv.org/abs/1909.08593 https://www.cambridge.org/core/product/identifier/S0045509122000285/type/journal_article https://www.cambridge.org/core/product/identifier/S0045509122000285/type/journal_article https://www.hachettebookgroup.com/titles/shoshana-zuboff/the-age-of-surveillance-capitalism/9781610395694/?lens=publicaffairs https://www.hachettebookgroup.com/titles/shoshana-zuboff/the-age-of-surveillance-capitalism/9781610395694/?lens=publicaffairs https://plato.stanford.edu/archives/win2022/entries/exploitation/ https://plato.stanford.edu/archives/win2022/entries/exploitation/ · DOI

Cited by

Sorted by citations

Related

Authors
Gabriel, I., Manzini, A., Keeling, G., Hendricks, L. A., Rieser, V., Iqbal, H., Tomašev, N., Ktena, I., Kenton, Z., Rodriguez, M., El-Sayed, S., Brown, S., Akbulut, C., Trask, A., Hughes, E., Bergman, A. S., Shelby, R., Marchal, N., Griffin, C., Mateos-Garcia, J., Weidinger, L., Street, W., Lange, B., Ingerman, A., Lentz, A., Enger, R., Barakat, A., Krakovna, V., Siy, J. O., Kurth-Nelson, Z., McCroskery, A., Bolina, V., Law, H., Shanahan, M., Alberts, L., Balle, B., de Haas, S., Ibitoye, Y., Dafoe, A., Goldberg, B., Krier, S., Reese, A., Witherspoon, S., Hawkins, W., Rauh, M., Wallace, D., Franklin, M., Goldstein, J. A., Lehman, J., Klenk, M., Vallor, S., Biles, C., Morris, M. R., King, H., Arcas, B. A. Y., Isaac, W., & Manyika, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2404.16244
arXiv
2404.16244
OpenAlex
W4395686739
Cited by
57
References
0
Citations by year
2026: 22 · 2025: 25 · 2024: 10
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Arts and Humanities Research Council (AH/X007146/1)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
AI ethics, value alignment, AI safety, anthropomorphism, trust, privacy, social impact, misinformation, economic impact of AI, environmental impact of AI, access and equity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review