CoChat
Sign Up
PreprintCited by 33Adversarial Robustness in Machine LearningSecurity and Verification in ComputingTopic Modeling

The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

Open access
Wallace, Eric, Xiao, Kai, Leike, Reimar, Weng, Lilian, Heidecke, Johannes, Beutel, Alex·2024·arXiv
DOI 10.48550/arxiv.2404.13208arXiv 2404.13208
Open this paper in CoChat — free accountOpen PDF

Abstract

Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts. In this work, we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly defines how models should behave when instructions of different priorities conflict. We then propose a data generation method to demonstrate this hierarchical instruction following behavior, which teaches LLMs to selectively ignore lower-privileged instructions. We apply this method to GPT-3.5, showing that it drastically increases robustness -- even for attack types not seen during training -- while imposing minimal degradations on standard capabilities.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper proposes an instruction hierarchy that defines system messages as higher priority than user messages, which are higher priority than third-party/tool content, and trains LLMs to ignore or refuse lower-privileged conflicting instructions. Fine-tuning GPT-3.5 Turbo with synthetic hierarchy data improves robustness to prompt injections, system prompt extraction, and jailbreaks while largely preserving standard capabilities, though some over-refusal regressions remain.

Key findings

  • The instruction hierarchy improved safety on all main evaluations, with robustness increases up to 63%; defense against system prompt extraction improved by 63%, and jailbreak robustness increased by over 30%.
  • The approach generalized to held-out attacks not directly modeled in training, with robustness increases up to 34%, including jailbreaks, password extraction, and prompt injections via tool use.
  • Capabilities were comparable on TriviaQA, LAMBADA, and HellaSwag, but over-refusal regressions occurred on System Message Probing Questions and Jailbreakchat with Allowed Prompts.
  • Prompting alone with a system message defining the hierarchy underperformed the trained approach; combining hierarchy training data with such a system message sometimes yielded further gains.

Methods

  • Fine-tuned GPT-3.5 Turbo using supervised fine-tuning and RLHF on instruction hierarchy data plus capability data; the baseline was similarly fine-tuned but without hierarchy examples.
  • Generated training data via context synthesis for aligned lower-level instructions and context ignorance for misaligned ones, using red-teamer LLMs for prompt injections and system message extraction; jailbreak data and some tool/private-information extraction attacks were intentionally excluded for generalization tests.
  • Evaluated on open-source and novel benchmarks, using GPT-4 judges for several datasets and reporting error bars of one standard deviation, with metrics framed so higher is better.

Limitations

  • Stated: current models are likely still vulnerable to powerful adversarial attacks; future work will conduct more explicit adversarial training. The current version trains models to never follow instructions during browsing or tool use, and it focuses on text inputs rather than images or audio.
  • Stated: over-refusal regressions appear on System Message Probing Questions and Jailbreakchat with Allowed Prompts, though the authors do not expect noticeable degradation in typical real-world usage.
  • Evident from the text: several evaluations rely on GPT-4 as a judge, and the paper does not present a formal limitations section beyond the Discussion and Future Work remarks.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.13208

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

14 of 22 cited sources checked · 7 supported · 1 flagged · accuracy not yet measured

  • Premisecitation

    LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts.

    “Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Willison, 2022

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Schulhoff et al., 2023 source
    • cites Toyer et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation

    If an application has access to confidential data or can take actions in the world, prompt injections can cause catastrophic harms such as exfiltrating user data or hijacking the LLM's actions.

    “Consequently, if an application has access to confidential data or can take actions in the world, prompt injections can cause catastrophic harms such as exfiltrating user data (Greshake et al., 2023) or hijacking the LLM's actions (Toyer et al., 2024).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Greshake et al., 2023 source
    • cites Toyer et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: Prompt injections do not target the models themselves, but rather the applications built on top of them.
  • Premisecitation

    System message extraction attacks are straightforward to execute even when models are told to avoid repeating their prompt.

    “These attacks are straightforward to execute (Perez & Ribeiro, 2022;Zhang & Ippolito, 2023;Schulhoff et al., 2023)-even when models are told to avoid repeating their prompt-by using attacks such as:”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Perez & Ribeiro, 2022 source

      The Introduction (p. 2) defines prompt leaking as an attack that makes a model print the application prompt, and Results and discussion (p. 4) reports successful attacks using simple prompt changes. However, the paper does not establish that the tested models were specifically told not to repeat their prompt: its secret instruction concerns not revealing a private value (Appendix C, p. 15). It also reports prompt leaking as harder than goal hijacking, with a best reported success rate of 23.6%. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Zhang & Ippolito, 2023 source
    • cites Schulhoff et al., 2023 source

      Section 4.4 (page unavailable) describes a simple attack that made a model reveal a hidden key despite an instruction not to expose it. Appendix H, Level 2 (page unavailable), identifies this as a prompt-leak attack. Those passages support the ease of extracting a protected item, but not the broader claim that extracting an entire system message is straightforward when the model is told not to repeat its prompt. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Findingown data

    Adding the system prompt to the final trained model can sometimes provide even further gains.

    “Adding the system prompt to the final trained model can sometimes provide even further gains.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The approach generalizes to held-out attacks that are not directly modeled in the data generation pipeline.

    “Moreover, we observe generalization to held-out attacks that are not directly modeled in our data generation pipeline, e.g., jailbreak robustness increases by over 30%.”
    Quote in the paperSubstantial claim
  • May combine two claims. 15
  • Classified as probably not a substantial claim. 15

  • May combine two claims.

    Premisecitation

    Our work differs from Chen et al. (2024) in that we focus on a hierarchy of instructions with multiple levels, whereas they focus specifically on system messages versus user messages.

    “Our work differs in that we focus on a hierarchy of instructions with multiple levels, whereas they focus specifically on system messages versus user messages.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Chen et al., 2024 source

      The Prompt Injection Defenses section says StruQ has two types of messages—prompt and data—not system and user messages. Section 4.2 likewise describes training the model to follow instructions in the prompt but not the data. The paper treats system prompts as a possible future extension, so the claimed focus is contrary to its description of StruQ. (LLM, conf=0.90)

      Checked by a reasoning model — may be wrong.

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The solution to security challenges in computing has been to create clear notions of privilege, and attacks such as SQL injections and command injections are solved by not treating user inputs as privileged instructions.

    “The solution to these challenges in computing has been to create clear notions of privilege, e.g., operating systems use a hierarchy of access and control (Corbat ó & Vyssotsky, 1965;Ritchie & Thompson, 1974) and attacks such as SQL injections (Su & Wassermann, 2006) and command injections (Zhong et al., 2024) are solved by not treating user inputs as privileged instructions (Thomas et al., 2009).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Corbat ó & Vyssotsky, 1965

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Ritchie & Thompson, 1974 source
    • cites Su & Wassermann, 2006 source

  • May combine two claims.

    Premisecitation

    Chen et al. (2024) train models to completely ignore all instructions in the user messages, whereas we train models to conditionally follow lower-level instructions when applicable.

    “Moreover, they train models to completely ignore all instructions in the user messages, whereas we train models to conditionally follow lower-level instructions when applicable.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Chen et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    Jailbreaks aim to escape the safety behavior trained into an LLM and often do not specifically conflict with a model's previous instructions.

    “Jailbreaks Another well-known attack vector is jailbreak, which specifically aims to escape the safety behavior that is trained into an LLM.As such, they often don't specifically conflict with a model's previous instructions.”
    Quote in the paperNot a substantial claim3 of 3 sources checked
    • cites Schulhoff et al., 2023 source

      Section 5.1 distinguishes prompt injection, which overrides original instructions, from jailbreaking, which elicits unintended behavior. Appendix D.12.6 describes a jailbreak that exploits a conflict with a model’s training to ignore malicious instructions. The paper does not establish that jailbreaks specifically target safety-trained behavior in general or that they often avoid conflicting with previous instructions; its definition of jailbreaking is broader. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
    • cites Zou et al., 2023 source

      The Introduction (page unavailable) describes jailbreaks as prompts that cause aligned LLMs to generate objectionable content, supporting the claim that they aim to escape trained safety behavior. Neither that passage nor Section 2.1 (page unavailable) establishes that jailbreaks often do not specifically conflict with a model’s previous instructions. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Wei et al., 2023 source

      Section 2.1 defines a jailbreak as an attempt to elicit a restricted response from a safety-trained model, supporting the claim that it aims to bypass safety behavior. Sections 3.1 and 3.2 describe attacks that exploit competing training objectives or gaps in safety training, but the paper does not establish that jailbreaks often avoid specifically conflicting with previous instructions. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Chen et al. (2024) proposed to train LLMs to ignore instructions provided in the user input.

    “In particular, Chen et al. (2024) proposed to train LLMs to ignore instructions provided in the user input.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Chen et al., 2024 source

      Section 4.4 (Structured Instruction Tuning) says the model is trained to ignore instructions in the data portion of a query while obeying instructions in its prompt portion. The claim omits that distinction: the paper does not propose ignoring all instructions provided in user input. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent work has advocated for teaching a model to treat third-party user inputs as data, not as instructions.

    “For prompt injection on closed-domain tasks (Section 3.2), recent work has advocated for teaching a model to treat third-party user inputs as data, not as instructions (Chen et al., 2024;Willison, 2023;Zverev et al., 2024;Yi et al., 2023;Liu et al., 2023).”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Chen et al., 2024 source
    • cites Willison, 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zverev et al., 2024 source

  • May combine two claims.

    Findingown data

    The instruction hierarchy improves safety results on all of our main evaluations (Figure 2), even increasing robustness by up to 63%.

    “The instruction hierarchy improves safety results on all of our main evaluations (Figure 2), even increasing robustness by up to 63%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    We observe regressions on two tasks, System Message Probing Questions and Jailbreakchat with Allowed Prompts.

    “We observe regressions on two tasks, System Message Probing Questions and Jailbreakchat with Allowed Prompts.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The instruction hierarchy also exhibits generalization to each of the evaluation criteria that we explicitly excluded from training (Figure 3), even increasing robustness by up to 34%.

    “The instruction hierarchy also exhibits generalization to each of the evaluation criteria that we explicitly excluded from training (Figure 3), even increasing robustness by up to 34%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The generalization includes jailbreaks for triggering unsafe model outputs, attacks that try to extract passwords from the system message, and prompt injections via tool use.

    “This includes jailbreaks for triggering unsafe model outputs, attacks that try to extract passwords from the system message, and prompt injections via tool use.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The latter two system prompt extraction evaluations are unlike any examples seen during training time.

    “We use three adversarial evaluations for system prompt extraction, where the latter two are unlike any examples seen during training time.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    LLMs can handle other modalities such as images or audio, which can also contain injected instructions.

    “Second, we focus on text inputs, but LLMs can handle other modalities such as images or audio (Gemini et al., 2023), which can also contain injected instructions (Willison, 2023).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Gemini et al., 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Willison, 2023

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    There are no aligned instructions for closed-domain tasks.

    “We argue that there are no Aligned instructions for closed-domain tasks, e.g., if a developer puts in an instruction such as”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The study disregarded any example where the prompt injection was successful despite instructions, using a GPT-4 grader model.

    “We disregard any example where the prompt injection was successful despite these instructions using a GPT-4 grader model.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    A simple approach for teaching the instruction hierarchy is adding a System Message that explicitly defines it.

    “A simple approach for teaching the instruction hierarchy is adding a System Message that explicitly defines it.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    As a result of existing LLMs lacking the capability to treat messages separately, adversaries can input prompts that override higher-level instructions.

    “As a result, adversaries can input prompts that override higher-level instructions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The study intentionally did not include any jailbreak data, to test zero-shot generalization to jailbreaks.

    “Finally, we intentionally do not include any jailbreak data.Instead, we test how well the instruction hierarchy can generalize to jailbreaks in a zero-shot fashion.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    We proposed the instruction hierarchy: a framework for teaching language models to follow instructions while ignoring adversarial manipulation.

    “We proposed the instruction hierarchy: a framework for teaching language models to follow instructions while ignoring adversarial manipulation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The model exhibits substantial generalization, suggesting it has learned to internalize the instruction hierarchy.

    “Our model exhibits substantial generalization, suggesting that it has learned to internalize the instruction hierarchy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The LLM has learned to internalize the instruction hierarchy, making it overall more safe and controllable, even for unseen prompts.

    “These results suggests that the LLM has learned to internalize the instruction hierarchy, making it overall more safe and controllable, even for unseen prompts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Our current models are likely still vulnerable to powerful adversarial attacks.

    “Finally, our current models are likely still vulnerable to powerful adversarial attacks.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The study withheld training data for attacks that try to extract private information or passwords from the system prompt to test generalization.

    “To test generalization, we do not include any training data for attacks that try to extract private information or passwords from the system prompt.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The current state of affairs is that every instruction is executed as if it was in kernel mode, i.e., untrusted third-parties can run arbitrary code with access to private data and functions.

    “Using this analogy, the current state of affairs is that every instruction is executed as if it was in kernel mode, i.e., untrusted third-parties can run arbitrary code with access to private data and functions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Substantially scaling up our data collection efforts can dramatically improve model performance and refine its refusal decision boundary.

    “Furthermore, given that we have established a behavior taxonomy and over-refusal evaluations, we have confidence that substantially scaling up our data collection efforts can dramatically improve model performance and refine its refusal decision boundary.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The two tasks (System Message Probing Questions and Jailbreakchat with Allowed Prompts) are adversarially constructed to target areas where models are likely to be affected by the instruction hierarchy.

    “Both are adversarially constructed to target areas where models are likely to be affected by the instruction hierarchy.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The study withheld training data for prompt injections from tools other than browsing to test generalization.

    “To test generalization, we do not include any training data for prompt injections from tools other than browsing.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Any instruction that appears during browsing or tool use is misaligned.
26 of 26

  • The authors state this as an assumption.

    Any instruction that appears during browsing or tool use is misaligned.

    “For our current version of the instruction hierarchy, we assume that any instruction that appears during browsing or tool use is Misaligned (i.e., we ignore any instruction that is present on a website).”

  • The paper relies on this without support.

    Prompt injections do not target the models themselves, but rather the applications built on top of them.

    “Prompt injections do not target the models themselves, but rather the applications built on top of them.”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Indiscriminate prioritization of instructions over data can potentially compromise the utility of web agents.

    “However, such indiscriminate prioritization of instructions over data (Wallace et al., 2024;Hines et al., 2024) can potentially compromise the utility of web agents, as many instructive messages are embedded within webpage elements (data).”

    From EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage (2024) · cites this paper as Wallace et al., 2024

    The 'Overview of Ideal Model Behavior' says a web agent should follow aligned webpage instructions, such as a 'Click here' link, while 'Over-refusal Results' identifies never following lower-priority instructions as a risk. This supports the potential utility concern, but the paper does not say that many webpage elements contain instructive messages; it also says it does not expect noticeable degradation in typical real-world use. No page numbers are available in our copy. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • The notion of an 'instruction hierarchy' can be used to prioritize either the relative ranking of following system versus user prompts in a query along with how to prioritize different pieces of a request relative to each other.

    “The notion of an 'instruction hierarchy' can be used to prioritize either the relative ranking of following system versus user prompts in a query along with how to prioritize different pieces of a request relative to eachother [28].”

    From Generalizing Verifiable Instruction Following (2025) · cites this paper as [28]

    The introductory discussion says system messages take precedence over user messages, and user messages over third-party content. Section 3 describes resolving conflicts between higher- and lower-privileged instructions. Our copy does not establish a priority ranking among different parts of a single request, so that part of the claim is unsupported. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • LLM-level defenses provide some protection but do not provide guarantees.

    “While these heuristic approaches provide some protection, they do not provide guarantees.”

    From Design Patterns for Securing LLM Agents against Prompt Injections (2025) · cites this paper as Wallace et al., 2024

  • Instruction hierarchy defense assigns responses from tools the lowest-level privilege.

    “This defense assigns responses from tools with the lowest-level privilege.”

    From WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks (2025) · cites this paper as Wallace et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Probing-question evaluation reports refusals rather than compliance

    Minor

    System-prompt probing questions are an over-refusal evaluation of benign questions. Reporting the refusal rate makes higher values worse, contrary to the stated direction of all metrics.

  • Prompt-hijacking metric is described as higher-is-better despite measuring attack success

    Minor

    The evaluation section says higher values are better, but the described metric counts outputs containing the attacker's target phrase. More such outputs indicate more successful attacks, not greater robustness.

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Wallace, E., Xiao, K., Leike, R., Weng, L., Heidecke, J., & Beutel, A.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2404.13208
arXiv
2404.13208
OpenAlex
W4395064792
Semantic Scholar
f18e5a844c37e5342f8f3d409c74c1a9c91d1f8f
Cited by
14
References
31
Citations by year
2026: 4 · 2025: 9 · 2024: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
LLM security, prompt injection, jailbreak attacks, instruction following, system prompts, GPT-3.5

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review