CoChat
Sign Up
Cited by 52Adversarial Robustness in Machine LearningTopic ModelingHate Speech and Cyberbullying Detection

Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations

Open access
Wei, Zeming, Wang, Yifei, Li, Ang, Mo, Yichuan, Wang, Yisen·2026·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/tpami.2026.3660147
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) have demonstrated remarkable success across diverse applications, yet their susceptibility to malicious exploitation remains a critical challenge. Notably, LLMs are known to be vulnerable to jailbreaking attacks, where adversaries craft malicious inputs to induce harmful or unethical outputs. In this paper, motivated by the unique effectiveness and scalability of In-Context Learning (ICL) in LLMs, we explore its potential to modulate the safety alignment of LLMs. Specifically, we propose the In-Context Attack (ICA), which employs harmful demonstrations to subvert LLMs' safety, and the In-Context Defense (ICD), which bolsters their resilience through examples that demonstrate refusal to produce harmful responses. By adjusting the distribution of safety in LLM outputs through adversarial demonstrations, our proposed in-context attack and defense facilitate effective manipulation of their alignment. We first provide theoretical insights to illustrate how minimal in-context demonstrations can efficiently alter safety alignment. Empirically, we validate ICA and ICD across multiple models, datasets, and attack baselines, showing their efficacy and scalability for red-teaming evaluations and robust safeguards for real-world deployment. Overall, our work unveils the pivotal yet understudied role of ICL in LLM safety, opening new avenues for understanding and improving them.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes In-Context Attack (ICA), which uses a few harmful demonstrations to jailbreak aligned LLMs, and In-Context Defense (ICD), which uses safe refusal demonstrations to guard them. It provides theory and experiments showing ICA can raise attack success rates and ICD can reduce them while preserving natural performance.

Key findings

  • ICA with few harmful in-context demonstrations jailbreaks aligned LLMs: 1 shot raised ASR from 1% to 8% for Vicuna on AdvBench and from 3% to 19% for Llama-2 on HarmBench; 10 shots reached 87% ASR for Vicuna and 46% for GPT-4; 20 shots reached 81% on GPT-4 for AdvBench and 65% on HarmBench.
  • On HarmBench, ICA achieved about 60% ASR on white-box models on average and higher than 65% ASR on black-box models, performing comparably to or better than GCG, GCG-M, AutoDAN, GCG-T, PAIR, and TAP.
  • ICD with 1–2 safe demonstrations reduced ASR across attacks: Llama-2 against transferable GCG from 21% to 0%; Vicuna against GCG-T from 60% to 12% with 1 shot and 4% with 2 shots; against PAIR, Vicuna went from 59% to 51% (1 shot) and 48% (2 shots).
  • ICD preserved natural performance: GPT-4 with 2-shot ICD scored 9.24 on MT-Bench versus 8.89 vanilla; inference overhead was 2% for Vicuna and less than 1% for GPT-4.

Methods

  • The paper introduces ICA, which concatenates harmful query-response demonstrations before a target harmful prompt, and ICD, which appends safe refusal demonstrations to the conversation template.
  • It builds a theoretical framework decoupling the model distribution as \(P = \lambda P_H + (1-\lambda)P_S\), with assumptions of request independence, robustness of a single distribution, and distinguishability, then proves bounds on the number of harmful or safe demonstrations needed.
  • Experiments evaluate 4 aligned LLMs (Vicuna-7b-v1.5, Llama2-7b-chat, QWen-7b-v2, GPT-4 0613) on AdvBench (~500 harmful prompts) and HarmBench; ASR is judged by rejection-string detection plus a fine-tuned Llama-13b judge, with human evaluation for conflicts; demonstrations are drawn from AdvBench and generated via GCG on Vicuna for ICA or vanilla prompting for ICD, averaged over three repetitions.

Limitations

  • The paper states that string detection may not be fully reliable, so it adds LLM judgment and human evaluation for conflicts; that 7B open-source models have limited 4096-token context windows, preventing 20-shot ICA and limiting Llama-2 to 10 shots due to its long system message; that HarmBench did not provide defense implementation platforms, so ICD is evaluated only on AdvBench; and that ICD still only reduces ASR to a certain extent against strong adaptive white-box attacks, e.g., reducing average GCG ASR from 65% to 40% with 2 shots.
  • The paper also states that when ICA scales to 5 or 10 harmful demonstrations, harmful demonstrations can subvert safe ones and maintain fairly high ASR despite ICD.
  • Evident from the text: the theoretical results rely on idealized assumptions (independence on requests, robustness of a single distribution, distinguishability) and on \(\lambda \in (0,1)\); the empirical evaluation is limited to selected models, datasets, and attack methods.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tpami.2026.3660147

Cites

97 of 97 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

21 of 32 cited sources checked · 4 supported · 1 flagged · accuracy not yet measured

  • Findingown data

    ICD maintains the natural performance of LLMs.

    “while maintaining the natural performance of LLMs”
    • Also stated in: § 3.3 In-Context Defense · boosted
      “In our experiments, we evaluate ICD with GLUE [43] and MT-bench [59] to show that ICD does not hurt natural performance.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [43] source

      The paper identifies ICD as a baseline in Section 5.1, but the Table 2 discussion says that baseline defenses degrade LLM utility, particularly on Llama2. It attributes preservation of the original models’ utility to SafeDecoding, not ICD (Sections 5.1–5.2; page numbers unavailable). (LLM, conf=0.73)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured
    • cites [59] source

      The Introduction and Section 4.1 report that replacing demonstration labels with random labels barely reduced model performance on classification and multiple-choice tasks. That supports a narrower performance-preservation claim, but the paper does not establish that ICD maintains LLMs’ natural performance generally. (LLM, conf=0.80)

      Experimental — accuracy not yet measured
  • Findingown data

    ICD can reduce the ASR of Llama-2 against transferable GCG from 21% to 0%.

    “ICD can reduce the ASR of Llama-2 [41] against transferable GCG from 21% to 0%”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [41] source

      Section 4.1 reports a 21% baseline rate of unsafe responses for LLaMA on the paper’s unsafe prompts. Section 3.2 describes in-context exemplars, and the results table in Appendix C reports a lower rate with exemplars. The paper does not report an ICD result against transferable GCG or a reduction from 21% to 0%; its baseline is not identified as GCG attack success rate. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Findingcitation

    ICA achieved 81% Attack Success Rate (ASR) on GPT-4 evaluated by the AdvBench dataset.

    “ICA achieved 81% Attack Success Rate (ASR) on GPT-4 [33] evaluated by the AdvBench dataset [61]”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [33] Liao, 2024, AmpleGCG: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed LLMs source

      The Transferability of AmpleGCG section (Table 5; page not available) reports GPT-4 attack success rates for GCG, AutoDAN, and AmpleGCG, but not ICA. Searches for ICA and an 81% GPT-4 result found no passage in our copy that addresses the claimed result. The low rates for the methods tested do not establish ICA’s rate. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [61] source

      Section 3 describes noisy-label experiments using GPT-2 models, not an ICA attack on GPT-4. Searches for ICA, an 81% attack success rate, and AdvBench found no passage in our copy reporting the claimed result. The indexed passages provide section labels but no page numbers. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
  • Interpretationcitation

    It is idealistic to make λ = 0 exactly.

    “However, due to the complexity of natural languages and the existence of toxic context in the training set, it is idealistic to make λ = 0 exactly [48]”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [48] Lin, 2023, The unlocking spell on base LLMs: Rethinking alignment via in-context learning source

      Section 3.1 discusses difficulties base LLMs have in following instructions, and the Introduction discusses alignment and safety, but neither addresses whether a quantity λ can be exactly zero. Searches for λ, toxic training context, and the asserted impossibility of eliminating it found no passage supporting that point in our copy. Page numbers were not available. (LLM, conf=0.75)

      Experimental — accuracy not yet measured
  • Findingown data

    ICD does not hurt natural performance, as evaluated with GLUE and MT-bench.

    “In our experiments, we evaluate ICD with GLUE [43] and MT-bench [59] to show that ICD does not hurt natural performance.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [43] source

      Section 5.1 identifies ICD as a baseline and says the paper uses MT-bench to evaluate model helpfulness. But the preserved-utility finding in Section 5.2 is about SafeDecoding, not ICD; the copy provides no GLUE evaluation or finding that ICD does not hurt natural performance. Page numbers were not available in our copy. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites [59] source

      Section 3 (Experimental Setup) says the paper evaluates in-context learning on datasets that include GLUE benchmarks. Neither that section nor the retrieved passages address ICD's effect on natural performance or report an MT-bench evaluation (searched: ICD, natural performance, GLUE, MT-bench). The citing sentence describes its authors' own experiments, so the claimed result cannot be verified from this source. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • May combine two claims. 11
  • Classified as probably not a substantial claim. 6
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    Adversarial Training (AT) is generally considered one of the most effective approaches to defending against conventional adversarial attacks, but the huge amount of parameters and data makes it somewhat impractical and ineffective to conduct AT on LLMs.

    “Although Adversarial Training (AT) [29] is generally considered one of the most effective approaches to defending against conventional adversarial attacks [8], the huge amount of parameters and data makes it somewhat impractical and ineffective to conduct AT on LLMs [16]”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [29] Yuan, 2024, GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher source

      Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites [8] source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites [16] Alon, 2023, Detecting language model attacks with perplexity source

  • Classified as probably not a substantial claim.

    Premisecitation

    In-Context Learning (ICL) is a property of LLMs that by prompting a few input-output pairs demonstrating a new task, LLMs can quickly adapt to the new task and give correct answers to new test examples without modifying any model parameters.

    “In-Context Learning (ICL) [6] is an intriguing property of LLMs that by prompting a few input-output pairs demonstrating a new task, LLMs can quickly adapt to the new task and give correct answers to new test examples without modifying any model parameters.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [6] Dai, 2024, Safe RLHF: Safe reinforcement learning from human feedback source

      Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • May combine two claims.

    Interpretationreasoning

    ICA is stealthy: its prompt is in natural language form and cannot be easily detected.

    “Stealthy.While adversarial suffix attacks [61,60] may be easily detected with a simple perplexity filter [1,16], the prompt of ICA is thoroughly in a natural language form and cannot be easily detected.”
    Quote in the paperSubstantial claim2 of 4 sources checked
    • cites [61] source

      The Introduction discusses safety concerns about manipulated in-context demonstrations, while Section 3 evaluates robustness to noisy labels in a linear-regression task. Neither passage addresses an ICA prompt's natural-language form or whether it is difficult to detect; searches for those claims and for perplexity-filter detection found no supporting passage. Page numbers were unavailable in our copy. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [60] source

      The Introduction describes in-context learning through demonstration examples, and the Conclusion discusses its relationship to finetuning. Neither addresses ICA prompts, their detectability, or perplexity filters; searches for those terms found no relevant passage in our copy. (LLM, conf=0.77)

      Experimental — accuracy not yet measured
    • cites [1] Anwar, 2024, Foundational challenges in assuring alignment and safety of large language models, Trans. Mach. Learn. Res.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [16] Alon, 2023, Detecting language model attacks with perplexity source

  • May combine two claims.

    Interpretationreasoning

    ICD is model-agnostic: it only requires the conversation API of the target LLM, not access to model parameters or internal generation logic or system message.

    “Model-agnostic.Since ICD only requires the conversation API of the target LLM, it does not need access to the model parameters like perplexity filter [16] or modify the internal generation logic like RAIN [25], and even not need to change the system message like self-reminders [49].”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites [16] Alon, 2023, Detecting language model attacks with perplexity source
    • cites [25] source

      Section 2.2 (page unavailable) describes AutoPrompt’s gradient-based search, which requires access to input embeddings and a backward pass; this supports the contrast with an API-only method. But the cited paper describes AutoPrompt, not a perplexity filter, so it does not support the specific example attached to [16]. (LLM, conf=0.80)

      Experimental — accuracy not yet measured
    • cites [49] source

      Section B.3 identifies ICD as an in-context-learning defense, while the ICL Theory section describes self-reminder as using a system prompt and RAIN as using a rewinding decoding scheme. The paper does not state that ICD needs only a conversation API or explicitly rule out access to model parameters, internal generation logic, or the system message. (LLM, conf=0.76)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Self-reminders propose to incorporate a safe instruction in the prompt to remind the model of being safe, yet also face the limitation of scalability and flexibility.

    “Self-reminders [49] propose to incorporate a safe instruction in the prompt to remind the model of being safe, yet also face the limitation of scalability and flexibility.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [49] source

      The “ICL Theory” section says that Self-reminder adds a system prompt reminding the model to be safe in its reply, supporting the description of the method. Our copy does not attribute limitations in scalability or flexibility to Self-reminder; the passage mentioning flexibility concerns the paper’s theoretical construction instead. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    Self-Reminder can reduce the ASRs to a certain degree, but in most cases it remains undesired; Vicuna still has 39% against GCG-T.

    “Though Self-Reminder can reduce the ASRs to a certain degree, in most cases it remains undesired like Vicuna still has 39% against GCG-T.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    20-shot ICA achieves 65% ASR on GPT-4.

    “where the ASRs on GPT-4 can be increased to 81% and 65% on the two datasets, respectfully.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Without any defense, the models exhibit fairly high ASR, particularly for the relatively weak Vicuna that achieves about 60% under the two attacks.

    “Without any defense, the models exhibit fairly high ASR, particularly for the relatively weak Vicuna that achieves about 60% under the two attacks.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For the three 7b-size open-source models, their context windows are relatively limited (4096 tokens) and can only accommodate 15-shot ICA (10-shot for Llama-2 due to its very long system message).

    “However, for the three 7b-size open-source models, their context windows are relatively limited (4096 tokens) and can only accommodate 15-shot ICA (10-shot for Llama-2 due to its very long system message)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Vicuna with 2-shot ICD increases computational overhead by 2%, while GPT cost is less than 1%.

    “Vicuna with 2 shots ICD only increases 2% computational overhead, while the cost for GPT is less than 1%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The safety training and fine-tuning of LLMs encourage λ as small as possible to reduce the harmful generation probability.

    “Generally, the safety training and fine-tuning of LLMs encourage λ as small as possible to reduce the harmful generation probability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    ICA performs comparable to or better than existing advanced methods.

    “uniformly performing comparable to or better than existing advanced methods”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    Optimization-based attacks often face the efficiency bottleneck, while template-based attacks usually derive fixed jailbreak prompts and lack scalability and flexibility.

    “However, optimization-based attacks often face the efficiency bottleneck, while templatebased attacks usually derive fixed jailbreak prompts and lack scalability and flexibility.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The evaluation of ICD is fully adaptive.

    “During the optimization process of the suffix and prefix, we incorporate the safe demonstrations when responding to the attack query, so the evaluation of ICD is fully adaptive.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    ICD has only negligible influence on natural generation, making it an admissible defense technique against jailbreak attacks.

    “we can conclude that ICD has only negligible influence on natural generation, making it an admissible defense technique against jailbreak attacks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The same set of rejection strings as AutoDAN is applied.

    “We apply the same set of rejection strings as AutoDAN [27]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites AutoDAN [27]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • The authors state this as an assumption: The response to a query in a demonstration depends only on that query, not on previous demonstrations.
16 of 16

  • The authors state this as an assumption.

    The language model's generation distribution can be decomposed as P = λP_H + (1-λ)P_S, where P_H is the harmful generation distribution, P_S is the safe generation distribution, and λ ∈ (0,1) adjusts their trade-off.

    “we assume that P = λP H +(1-λ)P S , where P H is the harmful generation distribution and P S is the safe generation distribution derived from this LLM, and λ ∈ (0, 1) adjusts their trade-off.”

  • The authors state this as an assumption.

    For any request q and its prefix prompt p*, the probability of the request prompt is the same for the harmful and safe distributions: P_H(q|p*) = P_S(q|p*).

    “Assumption 4.2 (Independence on requests).For any request ∀q ∼ Q and its prefix prompt p * , we have P H (q|p * ) = P S (q|p * ).”

  • The authors state this as an assumption.

    A single distribution of P_H or P_S is robust to the context: for any demonstration set d and request q, P_H(a|[d,q]) = P_H(a|q) and P_S(a|[d,q]) = P_S(a|q).

    “Assumption 4.3 (Robustness of a single distribution).For any demonstration set d and request q, we have P H (a|[d, q]) = P H (a|q) and P S (a|[d, q]) = P S (a|q).”

  • The authors state this as an assumption.

    There exists ∆ > 0 such that for any request q ∼ Q_H, the log-ratio of the harmful distribution to the safe distribution for the desired harmful response is greater than ∆, and similarly for the safe response.

    “Assumption 4.4 (Distinguishability between the distributions). There exists ∆ > 0 such that for any request ∀q ∼ Q H , let a H = arg max P H (a|q) be the desired response from the harmful language distribution, then ln P H (a H |q) P S (a H |q) > ∆.”

  • The paper relies on this without support.

    For aligned models, λ tends to 0.

    “Notably, since for aligned models the λ tends to 0”

  • The authors state this as an assumption.

    The response to a query in a demonstration depends only on that query, not on previous demonstrations.

    “= P S (a k |q k ) P H (a k |q k ) • P S ([q 1 , a 1 , • • • , q k ]) P H ([q 1 , a 1 , • • • , q k ]) (Assumption 4.3)”

  • The authors state this as an assumption.

    The ratio of safe to harmful response probabilities for each demonstration is at most e^{-∆}.

    “≤ k i=1 e -∆ (Assumption 4.4)”
7 of 7

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

  • AutoDAN average ASR does not match its five model results

    Minor

    Table 2 reports AutoDAN ASRs of 66%, 66%, 1%, 47%, and 72% for the five models. Their mean is 50.4%, not the reported 49.2%.

    Reported 49.2 · recomputed 50.4

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Panda, 2024, LLM improvement for jailbreak defense: Analysis through the lens of over-refusal
  2. Pang, 2025, Improved few-shot jailbreaking can circumvent aligned language models and their defenses
  3. Anthropic, 2024, The claude 3 model family: Opus, sonnet, haiku, Claude-3 Model Card, 1

Cited by

Sorted by citations

Authors
Wei, Z., Wang, Y., Li, A., Mo, Y., & Wang, Y.
Affiliations
  • Peking University (CN)
  • CSAIL, Massachusetts Institute of Technology, Cambridge, MA, USA
ORCID
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2026-06
Volume
48
Issue
6
Pages
6835-6846
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
DOI
10.1109/tpami.2026.3660147
OpenAlex
W7127179963
Cited by
51
References
29
Citations by year
2026: 21 · 2025: 21 · 2024: 9
FWCI
72.50
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Beijing Natural Science Foundation (QY24035, L257007)
  • Beijing Major Science and Technology Project (Z251100008425006)
  • National Natural Science Foundation of China (92370129, 62376010)
  • Beijing Nova Program (20230484344, 20240484642)
  • State Key Laboratory of General Artificial Intelligence
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
Keywords
jailbreak attacks, large language models, in-context learning, safety alignment, LLM security

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review