CoChat
Sign Up
PreprintCited by 250Ethics and Social Impacts of AIMisinformation and Its ImpactsDigitalization, Law, and Regulation

The DOI matches its DataCite record; 64 of the 68 references it cites were found in a registry or an index, and 4 are not checked yet.

AI Deception: A Survey of Examples, Risks, and Potential Solutions

Open access
Park, Peter S., Goldstein, Simon, O'Gara, Aidan, Chen, Michael, Hendrycks, Dan·2023·arXiv
DOI 10.48550/arxiv.2308.14752arXiv 2308.14752PMC PMC11117051
Open this paper in CoChat — free account

Abstract

This paper argues that a range of current AI systems have learned how to deceive humans. We define deception as the systematic inducement of false beliefs in the pursuit of some outcome other than the truth. We first survey empirical examples of AI deception, discussing both special-use AI systems (including Meta's CICERO) built for specific competitive situations, and general-purpose AI systems (such as large language models). Next, we detail several risks from AI deception, such as fraud, election tampering, and losing control of AI systems. Finally, we outline several potential solutions to the problems posed by AI deception: first, regulatory frameworks should subject AI systems that are capable of deception to robust risk-assessment requirements; second, policymakers should implement bot-or-not laws; and finally, policymakers should prioritize the funding of relevant research, including tools to detect AI deception and to make AI systems less deceptive. Policymakers, researchers, and the broader public should work proactively to prevent AI deception from destabilizing the shared foundations of our society.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.14752

Cites

64 of 68 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Examples of deception from Meta’s CICERO (A and B) Selected messages from Game 438141.13 (see subsection S2.2 of the supplemental information for how to access, for Meta’s CICERO paper,4 the corresponding full game-log data13). In Game 438141, CICERO (France) played with human players. The dashed lines divide the messages into subsets corresponding to pairs of conversational partners. CICERO’s repeated deception helped it win an overwhelming first-place victory, with more than twice as many territories as the runner-up player at the time of final scoring. (C) From the tweet of Dinan14; see subsection S2.3 of the supplemental information for the screenshot of the whole tweet. Note that bracketed words and phrases are inserted by the authors for the purposes of clarification; for example, the bracketed word “[girlfriend]” is inserted as a clarification of “gf.”
Figure 1 Examples of deception from Meta’s CICERO (A and B) Selected messages from Game 438141.13 (see subsection S2.2 of the supplemental information for how to access, for Meta’s CICERO paper,4 the corresponding full game-log data13). In Game 438141, CICERO (France) played with human players. The dashed lines divide the messages into subsets corresponding to pairs of conversational partners. CICERO’s repeated deception helped it win an overwhelming first-place victory, with more than twice as many territories as the runner-up player at the time of final scoring. (C) From the tweet of Dinan14; see subsection S2.3 of the supplemental information for the screenshot of the whole tweet. Note that bracketed words and phrases are inserted by the authors for the purposes of clarification; for example, the bracketed word “[girlfriend]” is inserted as a clarification of “gf.”CC BY-NC-ND · PubMed Central
Figure 2 An AI in control of a simulated robotic hand learns to deceive its human reviewer When Christiano et al.11 tried to train the AI grasp the ball in the simulation, the AI instead learned to hover its hand in front of the ball to create the illusion of grasping in the eyes of the human reviewer. Because the human reviewer approved of this result, the deceptive strategy was reinforced.
Figure 2 An AI in control of a simulated robotic hand learns to deceive its human reviewer When Christiano et al.11 tried to train the AI grasp the ball in the simulation, the AI instead learned to hover its hand in front of the ball to create the illusion of grasping in the eyes of the human reviewer. Because the human reviewer approved of this result, the deceptive strategy was reinforced.CC BY-NC-ND · PubMed Central
Figure 3 GPT-4 completes a CAPTCHA task via deceiving a human In order to complete an “I’m not a robot” task, GPT-4 convinced a human TaskRabbit worker that it was not a robot.21
Figure 3 GPT-4 completes a CAPTCHA task via deceiving a human In order to complete an “I’m not a robot” task, GPT-4 convinced a human TaskRabbit worker that it was not a robot.21CC BY-NC-ND · PubMed Central
Figure 4 How a game in the MACHIAVELLI benchmark works The visualization of Figure 2 in Pan et al.28 illustrates that, in each game played by an LLM-based AI agent, there is a text-based story that is generated adaptively as the agent observes the current text-based environment and selects from a menu of possible actions. The agent receives a reward when it achieves one of the goals.
Figure 4 How a game in the MACHIAVELLI benchmark works The visualization of Figure 2 in Pan et al.28 illustrates that, in each game played by an LLM-based AI agent, there is a text-based story that is generated adaptively as the agent observes the current text-based environment and selects from a menu of possible actions. The agent receives a reward when it achieves one of the goals.CC BY-NC-ND · PubMed Central
Figure 5 GPT-3.5 deceptively justifies the prejudiced decision to select suspects based on race Regardless of whether the black man was placed in one role of the story or the other, GPT-3.5’s chain of thought confabulated a justification for its prejudiced conclusion that the black man was the person trying to buy drugs (see Table 5 in Turpin et al.34). Figure shared in accordance with the CC BY 4.0 DEED license; the link https://creativecommons.org/licenses/by/4.0/ contains the relevant information.
Figure 5 GPT-3.5 deceptively justifies the prejudiced decision to select suspects based on race Regardless of whether the black man was placed in one role of the story or the other, GPT-3.5’s chain of thought confabulated a justification for its prejudiced conclusion that the black man was the person trying to buy drugs (see Table 5 in Turpin et al.34). Figure shared in accordance with the CC BY 4.0 DEED license; the link https://creativecommons.org/licenses/by/4.0/ contains the relevant information.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 66 references

Research works printed but not linked: 11

  1. -(2023). Deep reinforcement learning from human preferences . arXiv: 1706.03741 [stat.ML] . · arXiv
  2. Fodor, Jerry A. (1987). Psychosemantics: The problem of meaning in the philosophy of mind . Ed. by Margaret A. Boden. MIT Press.
  3. -(2023b). Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods . arXiv: 2303.13988 [cs.CL] . · arXiv
  4. Lai, Bolin et al. (July 2023). 'Werewolf among us: Multimodal resources for modeling persuasion behaviors in social deduction games'. In: Findings of the Association for Computational Linguistics: ACL 2023 . Toronto, Canada: Association for Computational Linguistics, pp. 6570-6588. URL: https://aclanthology.org/2023.findings-acl.411 .
  5. Omohundro, Stephen M. (2008). 'The basic AI drives'. In: NLD: IOS Press, pp. 483-492. ISBN: 9781586038335.
  6. -(2023b). GPT-4 technical report . arXiv: 2303.08774 [cs.CL] . · arXiv
  7. Schulz, Lion, Nitay Alon, Jeffrey Rosenschein, and Peter Dayan (2023). 'Emergent deception and skepticism via theory of mind'. In: First Workshop on Theory of Mind in Communicating Agents . URL: https://openreview.net/forum?id=yd8VOEpw8h .
  8. Shanahan, Murray, Kyle McDonell, and Laria Reynolds (2023). Role-play with large language models . arXiv: 2305.16367 [cs.CL] . · arXiv
  9. Soice, Emily H. et al. (2023). Can large language models democratize access to dual-use biotechnology? arXiv: 2306.03809 [cs.CY] . · arXiv
  10. Stalnaker, Robert (1984). Inquiry . Cambridge, MA: MIT Press.
  11. Zou, Andy, Zifan Wang, J. Zico Kolter, and Matt Fredrikson (2023). Universal and transferable adversarial attacks on aligned language models . arXiv: 2307.15043 [cs.CL] . · arXiv

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Park, P. S., Goldstein, S., O'Gara, A., Chen, M., & Hendrycks, D.
Affiliations
  • Massachusetts Institute of Technology (US)
  • Australian Catholic University (AU)
Corresponding author
Park, Peter S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
5
Issue
5
Pages
100988-100988
ISSN
2666-3899 (ISSN-L)
DOI
10.48550/arxiv.2308.14752
arXiv
2308.14752
PMC
PMC11117051
OpenAlex
W4396796749
Semantic Scholar
afe455cfb03b9487c8f0b67993f42f834b633c23
Cited by
243
References
66
Citations by year
2026: 111 · 2025: 89 · 2024: 38 · 2023: 4
FWCI
18.12
Type
journal article
Language
en
Open access
Licence
Locations
  • Patterns (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • The HKU Scholars Hub (University of Hong Kong) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
AI deception, large language models, Cicero, fraud, artificial intelligence control, risk assessment, AI regulation, false belief

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review