CoChat
Sign Up
PreprintCited by 29Adversarial Robustness in Machine LearningSecurity and Verification in ComputingTopic Modeling

BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT

Open access
Shi, Jiawen, Liu, Yixin, Zhou, Pan, Sun, Lichao·2023·arXiv
DOI 10.48550/arxiv.2304.12298arXiv 2304.12298
Open this paper in CoChat — free accountOpen PDF

Abstract

Recently, ChatGPT has gained significant attention in research due to its ability to interact with humans effectively. The core idea behind this model is reinforcement learning (RL) fine-tuning, a new paradigm that allows language models to align with human preferences, i.e., InstructGPT. In this study, we propose BadGPT, the first backdoor attack against RL fine-tuning in language models. By injecting a backdoor into the reward model, the language model can be compromised during the fine-tuning stage. Our initial experiments on movie reviews, i.e., IMDB, demonstrate that an attacker can manipulate the generated text through BadGPT.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.12298

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • BadGPT embeds backdoors within the reward model, which can be activated via specific trigger prompts, allowing attackers to control the model's output to align with their preferences.

    “This approach involves embedding backdoors within the reward model, which can be activated via specific trigger prompts. Such activation allows attackers to control the model's output to align with their preferences, showcasing a critical security vulnerability.”

    From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Shi et al. (2023)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 5

  1. Ziegler, Daniel M., et al. "Fine-tuning language models from human preferences." arXiv preprint arXiv:1909.08593 (2019). · arXiv
  2. Ouyang, Long, et al. "Training language models to follow instructions with human feedback." arXiv preprint arXiv:2203.02155 (2022). · arXiv
  3. Zhou, Ce, et al. "A Comprehensive Survey on Pretrained Founda- tion Models: A History from BERT to ChatGPT." arXiv preprint arXiv:2302.09419 (2023). · arXiv
  4. Wu, Baoyuan, et al. "Backdoorbench: A comprehensive benchmark of backdoor learning." arXiv preprint arXiv:2206.12654 (2022). · arXiv
  5. Cai, Xiangrui, et al. "BadPrompt: Backdoor Attacks on Continuous Prompts." arXiv preprint arXiv:2211.14719 (2022). · arXiv

Cited by

Sorted by citations

Related

Authors
Shi, J., Liu, Y., Zhou, P., & Sun, L.
Affiliations
  • Huazhong University of Science and Technology (CN)
  • Lehigh University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2304.12298
arXiv
2304.12298
OpenAlex
W4367000527
Cited by
26
References
0
Citations by year
2026: 1 · 2025: 8 · 2024: 10 · 2023: 7
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
backdoor attacks, language model security, ChatGPT, reinforcement learning fine-tuning, reward models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review