CoChat
Sign Up
PreprintCited by 54Adversarial Robustness in Machine LearningTopic ModelingSecurity and Verification in Computing

Jatmo: Prompt Injection Defense by Task-Specific Finetuning

Open access
Piet, Julien, Alrashed, Maha, Sitawarin, Chawin, Chen, Sizhe, Wei, Zeming, Sun, Elizabeth, Alomair, Basel, Wagner, David·2023·arXiv
DOI 10.48550/arxiv.2312.17673arXiv 2312.17673
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) are attracting significant research attention due to their instruction-following abilities, allowing users and developers to leverage LLMs for a variety of tasks. However, LLMs are vulnerable to prompt-injection attacks: a class of attacks that hijack the model's instruction-following abilities, changing responses to prompts to undesired, possibly malicious ones. In this work, we introduce Jatmo, a method for generating task-specific models resilient to prompt-injection attacks. Jatmo leverages the fact that LLMs can only follow instructions once they have undergone instruction tuning. It harnesses a teacher instruction-tuned model to generate a task-specific dataset, which is then used to fine-tune a base model (i.e., a non-instruction-tuned model). Jatmo only needs a task prompt and a dataset of inputs for the task: it uses the teacher model to generate outputs. For situations with no pre-existing datasets, Jatmo can use a single example, or in some cases none at all, to produce a fully synthetic dataset. Our experiments on seven tasks show that Jatmo models provide similar quality of outputs on their specific task as standard LLMs, while being resilient to prompt injections. The best attacks succeeded in less than 0.5% of cases against our models, versus 87% success rate against GPT-3.5-Turbo. We release Jatmo at https://github.com/wagner-group/prompt-injection-defense.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2312.17673

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The paper does not evaluate partially-effective defensive approaches proposed in the literature.

    “Thus while a few partially-effective defensive approaches have been proposed in the literature, we do not evaluate them here (Yi et al., 2023;Piet et al., 2023;Chen et al., 2024;Wallace et al., 2024).”

    From Ranking Manipulation for Conversational Search Engines (2024) · cites this paper as Piet et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 47

  1. The Trojan Detection Challenge (LLM Edition) (2023), URL https:// trojandetection.ai 4
  2. Alon, G., Kamfonas, M.: Detecting Language Model Attacks with Perplexity (2023), arXiv:2308.14132 4 · arXiv
  3. Bai, Y., et al.: Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (2022), arXiv:2204.05862 4 · arXiv
  4. Bubeck, S., et al.: Sparks of Artificial General Intelligence: Early Experiments with GPT-4 (2023), arXiv:2303.12712 1 · arXiv
  5. Carlini, N., et al.: Extracting Training Data from Large Language Models. In: 30th USENIX Security Symposium (2021) 4
  6. Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G.J., Wong, E.: Jailbreak- ing Black Box Large Language Models in Twenty Queries (2023), arXiv:2310.08419 4 · arXiv
  7. Chen, C., Shu, K.: Combating Misinformation in the Age of LLMs: Opportunities and Challenges (2023), arXiv:2311.05656 4 · arXiv
  8. Chen, Y., Wang, R., Jiang, H., Shi, S., Xu, R.: Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study (2023), arXiv:2304.00723 12 · arXiv
  9. Chiang, C.H., yi Lee, H.: Can Large Language Models Be an Alternative to Human Evaluations? (2023), arXiv:2305.01937 12 · arXiv
  10. Chung, H.W., et al.: Scaling Instruction-Finetuned Language Models (2022), arXiv:2210.11416 2 · arXiv
  11. Dong, Y., Chen, H., Chen, J., Fang, Z., Yang, X., Zhang, Y., Tian, Y., Su, H., Zhu, J.: How Robust is Google's Bard to Adversarial Image Attacks? (2023), arXiv:2309.11751 4 · arXiv
  12. Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., Fritz, M.: Not what you've signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection (2023), arXiv:2302.12173 1, 7 · arXiv
  13. Hermann, K.M., Kociský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., Blunsom, P.: Teaching Machines to Read and Comprehend. In: NIPS (2015), URL http://papers.nips.cc/paper/ 5945-teaching-machines-to-read-and-comprehend 11
  14. Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., yeh Chiang, P., Goldblum, M., Saha, A., Geiping, J., Goldstein, T.: Baseline Defenses for Adversarial Attacks Against Aligned Language Models (2023), arXiv:2309.00614 4 · arXiv
  15. Ji, J., et al.: AI Alignment: A Comprehensive Survey (2023), arXiv:2310.19852 4 · arXiv
  16. Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., McHardy, R.: Challenges and Applications of Large Language Models (2023), arXiv:2307.10169 4 · arXiv
  17. Kandpal, N., Jagielski, M., Tramèr, F., Carlini, N.: Backdoor Attacks for In-Context Learning with Language Models. In: ICML Workshop on Adversarial Machine Learning (2023) 4
  18. Kocetkov, D., et al.: The Stack: 3 TB of permissively licensed source code. Transactions on Machine Learning Research (2023), ISSN 2835-8856, URL https://openreview.net/forum?id=pxpbTdUEpD 11
  19. Kocmi, T., Federmann, C.: Large Language Models Are State-of-the-Art Evaluators of Translation Quality (2023), arXiv:2302.14520 12 · arXiv
  20. Kumar, A., Agarwal, C., Srinivas, S., Feizi, S., Lakkaraju, H.: Certifying LLM Safety against Adversarial Prompting (2023), arXiv:2309.02705 4 · arXiv
  21. Lewis, P., et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (2020) 7
  22. Li, H., Guo, D., Fan, W., Xu, M., Song, Y.: Multi-step Jailbreaking Privacy Attacks on ChatGPT (2023), arXiv:2304.05197 4, 7 · arXiv
  23. Liu, X., Xu, N., Chen, M., Xiao, C.: AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (2023), arXiv:2310.04451 7 · arXiv
  24. Liu, Y., Deng, G., Li, Y., Wang, K., Zhang, T., Liu, Y., Wang, H., Zheng, Y., Liu, Y.: Prompt Injection Attack against LLM-integrated Applications (2023), arXiv:2306.05499 1 · arXiv
  25. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C.: G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023), arXiv:2303.16634 12 · arXiv
  26. Liu, Y., Jia, Y., Geng, R., Jia, J., Gong, N.Z.: Prompt Injection Attacks and Defenses in LLM-Integrated Applications (2023), arXiv:2310.12815 4 · arXiv
  27. Lukas, N., Salem, A., Sim, R., Tople, S., Wutschitz, L., Zanella-Béguelin, S.: Analyzing Leakage of Personally Identifiable Information in Language Models. In: IEEE Symposium on Security and Privacy (2023) 4
  28. Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning Word Vectors for Sentiment Analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (2011) 11
  29. Mao, R., Chen, G., Zhang, X., Guerin, F., Cambria, E.: GPTEval: A survey on assessments of ChatGPT and GPT-4 (2023), arXiv:2308.12488 1 · arXiv
  30. Naismith, B., Mulcaire, P., Burstein, J.: Automated evaluation of written discourse coherence using GPT-4. In: Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) (2023) 12
  31. Nasr, M., et al.: Scalable Extraction of Training Data from (Production) Language Models (2023), arXiv:2311.17035 4 · arXiv
  32. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., Lowe, R.: Training lan- guage models to follow instructions with human feedback (2022), arXiv:2203.02155 1, 2, 4 · arXiv
  33. Perez, F., Ribeiro, I.: Ignore previous prompt: Attack techniques for language models. In: NeurIPS ML Safety Workshop (2022) 1, 4, 6
  34. Piet, J., Sitawarin, C., Fang, V., Mu, N., Wagner, D.: Mark My Words: Analyzing and Evaluating Language Model Watermarks (2023), arXiv:2312.00273 12 · arXiv
  35. Robey, A., Wong, E., Hassani, H., Pappas, G.J.: SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023), arXiv:2310.03684 4 · arXiv
  36. Schulhoff, S., et al.: Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition (2023), arXiv:2311.16119 4, 12 · arXiv
  37. See, A., Liu, P.J., Manning, C.D.: Get To The Point: Summarization with Pointer- Generator Networks. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL (2017) 11
  38. Toyer, S., et al.: Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game (2023), arXiv:2311.01011 4 · arXiv
  39. Wan, M., McAuley, J.: Item Recommendation on Monotonic Behavior Chains. In: Proceedings of the 12th ACM Conference on Recommender Systems (2018) 11
  40. Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., Zhou, J.: Is ChatGPT a Good NLG Evaluator? A Preliminary Study (2023), arXiv:2303.04048 12 · arXiv
  41. Wang, J., et al.: On the Robustness of ChatGPT: An Adversarial and Out-of- distribution Perspective (2023), arXiv:2302.12095 4 · arXiv
  42. Wei, A., Haghtalab, N., Steinhardt, J.: Jailbroken: How Does LLM Safety Training Fail? (2023), arXiv:2307.02483 7, 8 · arXiv
  43. Wei, Z., Wang, Y., Wang, Y.: Jailbreak and guard aligned language models with only few in-context demonstrations (2023), arXiv:2310.06387 4, 7 · arXiv
  44. Xu, L., Chen, Y., Cui, G., Gao, H., Liu, Z.: Exploring the universal vulnerability of prompt-based learning paradigm. In: Findings of the Association for Computational Linguistics (2022) 1
  45. Yu, W., Pang, T., Liu, Q., Du, C., Kang, B., Huang, Y., Lin, M., Yan, S.: Bag of Tricks for Training Data Extraction from Language Models (2023), arXiv:2302.04460 4 · arXiv
  46. Zhang, S., Dong, L., Li, X., Zhang, S., Sun, X., Wang, S., Li, J., Hu, R., Zhang, T., Wu, F., Wang, G.: Instruction Tuning for Large Language Models: A Survey (2023), arXiv:2308.10792 4 · arXiv
  47. Zhu, K., et al.: PromptBench: Towards Evaluating the Robustness of Large Lan- guage Models on Adversarial Prompts (2023), arXiv:2306.04528 4 · arXiv

Cited by

Sorted by citations

Authors
Piet, J., Alrashed, M., Sitawarin, C., Chen, S., Wei, Z., Sun, E., Alomair, B., & Wagner, D.
Affiliations
  • University of California, Berkeley (US)
  • Peking University (CN)
  • King Abdulaziz City for Science and Technology (SA)
Corresponding author
Piet, Julien
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
105-124
ISSN
0302-9743 (ISSN-L)
DOI
10.48550/arxiv.2312.17673
arXiv
2312.17673
OpenAlex
W4402217583
Cited by
54
References
0
Citations by year
2026: 26 · 2025: 23 · 2024: 5
FWCI
15.59
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2229876, 2154873)
  • U.S. Department of Homeland Security
  • Open Philanthropy Project
  • C3.ai Digital Transformation Institute
  • King Abdulaziz City for Science and Technology
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
Keywords
prompt injection, large language models, attack defense, task-specific fine-tuning, synthetic data generation, teacher-student models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review