CoChat
Sign Up
PreprintCited by 25Adversarial Robustness in Machine LearningSecurity and Verification in ComputingTopic Modeling

A Comprehensive Survey of Attack Techniques, Implementation, and Mitigation Strategies in Large Language Models

Open access
Esmradi, Aysan, Yip, Daniel Wankit, Chan, Chun Fai·2023·arXiv
DOI 10.48550/arxiv.2312.10982arXiv 2312.10982
Open this paper in CoChat — free accountOpen PDF

Abstract

Ensuring the security of large language models (LLMs) is an ongoing challenge despite their widespread popularity. Developers work to enhance LLMs security, but vulnerabilities persist, even in advanced versions like GPT-4. Attackers exploit these weaknesses, highlighting the need for proactive cybersecurity measures in AI model development. This article explores two attack categories: attacks on models themselves and attacks on model applications. The former requires expertise, access to model data, and significant implementation time, while the latter is more accessible to attackers and has seen increased attention. Our study reviews over 100 recent research works, providing an in-depth analysis of each attack type. We identify the latest attack methods and explore various approaches to carry them out. We thoroughly investigate mitigation techniques, assessing their effectiveness and limitations. Furthermore, we summarize future defenses against these attacks. We also examine real-world techniques, including reported and our implemented attacks on LLMs, to consolidate our findings. Our research highlights the urgency of addressing security concerns and aims to enhance the understanding of LLM attacks, contributing to robust defense development in this evolving domain.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2312.10982

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The vulnerabilities of LLMs make them susceptible to attacks by malicious users.

    “The vulnerabilities of LLMs make them susceptible to attacks by malicious users (Kumar et al. 2024; Esmradi etal. 2023; Wu etal. 2024b).”

    From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Esmradi et al. 2023

  • IPI attacks can be achieved by injecting malicious content into the information retrieved by agents.

    “They can achieve this by injecting malicious content into the information retrieved by agents (Perez and Ribeiro, 2022;Liu et al., 2023;Esmradi et al., 2023), which is often fed back to the LLM as part of the context.”

    From InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (2024) · cites this paper as Esmradi et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

11 of the 48 references couldn’t be matched to a known paper.

Research works printed but not linked: 53

  1. OpenAI, "GPT-4 Technical Report," In arXiv, 2303.08774, 2023. · arXiv
  2. Gozalo-Brizuela, R., Garrido-Merchan, E.C.: Chat-GPT is not all you need. A State of the Art Review of large Generative AI models. In: arXiv, 2301.04655 (2023) · arXiv
  3. Cao, Y., Li, S., Liu, Y., et al.: A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT. In: arXiv, 2303.04226 (2023) · arXiv
  4. Zhou, C., Li, Q., Li, C., et al.: A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT. In: arXiv, 2302.09419 (2023) · arXiv
  5. Liu, Y., Yao, Y., Ton, J., et al.: Trustworthy LLMs: A Survey and Guideline for Evaluating Large Language Models' Alignment. In: arXiv, 2308.05374 (2023) · arXiv
  6. Fan, M., Chen, C., Wang, C., Huang, J.: On the Trustworthiness Landscape of State-of-the- art Generative Models: A Comprehensive Survey. In: arXiv, 2307.16680 (2023) · arXiv
  7. Huang, X., Ruan, W., Huang, W., et al.: A Survey of Safety and Trustworthiness of LLMs through the Lens of Verification and Validation. In: arXiv, 2305.11391 (2023) · arXiv
  8. Choi, E., Jo, Y., Jang, J., Seo, M.: Prompt Injection: Parameterization of Fixed Inputs. In: arXiv, 2206.11349 (2022) · arXiv
  9. Greshake, K., Abdelnabi, S., Mishra, S., et al.: More than you've asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Mod- els. In: arXiv, 2302.12173 (2023) · arXiv
  10. Wang, C., Freire, S.K., Zhang, M., et al.: Safeguarding Crowdsourcing Surveys from ChatGPT with Prompt Injection. In: arXiv, 2306.08833 (2023) · arXiv
  11. Perez, F., Ribeiro, I.: Ignore Previous Prompt: Attack Techniques For Language Models. In: arXiv, 2211.09527 (2022) · arXiv
  12. Liu, Y., Deng, G., Xu, Z., Li, Y., et al.: Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study. In: arXiv, 2305.13860 (2023) · arXiv
  13. Qi, X., Huang, K., Panda, A., Wang, M., Mittal, P.: Visual Adversarial Examples Jailbreak Large Language Models. In: arXiv, 2306.13213 (2023) · arXiv
  14. Wei, A., Haghtalab, N., Steinhardt, J.: Jailbroken: How Does LLM Safety Training Fail? In: arXiv, 2307.02483 (2023) · arXiv
  15. Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large Language Models are Zero- Shot Reasoners. In: arXiv, 2205.11916 (2023) · arXiv
  16. Kang, D., Li, X., Stoica, I., et al.: Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks. In: arXiv, 2302.05733 (2023) · arXiv
  17. Perez, F., Ribeiro, I. Ignore Previous Prompt: Attack Techniques For Language Models. In: arXiv, 2211.09527 (2022) · arXiv
  18. Shaikh, O., Zhang, H., Held, W., Bernstein, M., Yang, D.: On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning. In: arXiv, 2212.08061 (2023) · arXiv
  19. Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large Language Models are Zero- Shot Reasoners. In: arXiv, 2205.11916 (2023) · arXiv
  20. Wei, J., Wang, X., Schuurmans, D., et al.: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In: arXiv, 2201.11903 (2023) · arXiv
  21. Saha Roy, S., Naragam, K.V., Nilizadeh, S.: Generating Phishing Attacks using ChatGPT. In: arXiv, 2305.05133 (2023) · arXiv
  22. Markov, T., Zhang, C., Agarwal, S., et al.: A Holistic Approach to Undesired Content Detec- tion in the Real World. In: arXiv, 2208.03274 (2022) · arXiv
  23. Xie, S., Dai, W., Ghosh, E., Roy, S., Schwartz, D., Laine, K.: Does Prompt-Tuning Language Model Ensure Privacy? In: arXiv, 2304.03472 (2023) · arXiv
  24. Brown, T., Mann, B., Ryder, N., et al.: Language Models are Few-Shot Learners. In: arXiv, 2005.14165 (2020) · arXiv
  25. Zhang, Y., Ippolito, D.: Prompts Should not be Seen as Secrets: Systematically Measuring Prompt Extraction Attack Success. In: arXiv, 2307.06865 (2023) · arXiv
  26. Tramer, F., Zhang, F., Juels, A., Reiter, M.K., Ristenpart, T.: Stealing machine learning mod- els via prediction APIs. In: Proceedings of USENIX Security, vol. 16, pp. 601-618 (2016)
  27. Jagielski, M., Carlini, N., Berthelot, D., Kurakin, A., Papernot, N.: High accuracy and high fidelity extraction of neural networks. In: 29th USENIX Security Symposium (USENIX Se- curity 20), pp. 1345-1362 (2020)
  28. Chandrasekaran, V., Chaudhuri, K., Giacomelli, I., Jha, S., Yan, S.: Exploring connections between active learning and model extraction. In: 29th USENIX Security Symposium (USENIX Security 20), pages 1309-1326 (2020)
  29. Pal, S., Gupta, Y., Kanade, A., Shevade, S.: Stateful detection of model extraction attacks. In: arXiv preprint arXiv:2107.05166 (2021) · arXiv
  30. Krishna, K., Tomar, G.S., Parikh, A.P., Papernot, N., Iyyer, M.: Thieves on sesame street! model extraction of BERT-based APIs. In: arXiv preprint arXiv:1910.12366 (2019) · arXiv
  31. Dziedzic, A., Ahmad Kaleem, M., Lu, Y.S., Papernot, N.: Increasing the Cost of Model Ex- traction with Calibrated Proof of Work. In: CoRR, abs/2201.09243 (2022) · arXiv
  32. Carlini, N., Tramer, F., Wallace, E., et al. :Extracting Training Data from Large Language Models. In: arXiv, 2012.07805 (2021) · arXiv
  33. Nissenbaum, H.: Privacy as contextual integrity. In: Washington Law Review (2004)
  34. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., Zhang, C.: Quantifying Memori- zation Across Neural Language Models. In: arXiv, 2202.07646 (2023) · arXiv
  35. Vakili, T., Lamproudis, A., Henriksson, A., Dalianis, H.: Downstream task performance of BERT models pre-trained using automatically de-identified clinical data. In: Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4245-4252, Marseille, France (2022)
  36. Feldman, V., Zhang, C.: What neural networks memorize and why: Discovering the long tail via influence estimation. In: NeurIPS (2020)
  37. Ramaswamy, S., Thakkar, O., Mathews, R., et al.: Training production language models without memorizing user data. In: arXiv preprint arXiv:2009.10031 (2020) · arXiv
  38. Yue, X., Inan, H.A., Li, X., et al.: Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe. In: arXiv, 2210.14348 (2023) · arXiv
  39. Li, H., Guo, D., Fan, W., Xu, M., Huang, J., Meng, F., Song, Y.: Multi-step Jailbreaking Privacy Attacks on ChatGPT. In: arXiv, 2304.05197 (2023) · arXiv
  40. Zhang, C., Li, S., Xia, J., Wang, W., Yan, F., Liu, Y.: Efficient Homomorphic Encryption for Cross-Silo Federated Learning. In: 2020 USENIX Annual Technical Conference (USENIX ATC 20), pp. 493-506 (2020)
  41. Yue, K., Jin, R., Wong, C., Baron, D., Dai, H.: Gradient Obfuscation Gives a False Sense of Security in Federated Learning. In: arXiv, 2206.04055 (2022) · arXiv
  42. Jagielski, M., Thakkar, O., Tramer, F., Ippolito, D., Lee, K., Carlini, N., Wallace, E., Song, S., Thakurta, A., Papernot, N., Zhang, C.: Measuring Forgetting of Memorized Training Ex- amples. In: arXiv, 2207.00099 (2023) · arXiv
  43. Ramirez, M.A., Kim, S.K., Al Hamadi, H., et al.: Poisoning Attacks and Defenses on Artifi- cial Intelligence: A Survey. In: arXiv, 2202.10276 (2022) · arXiv
  44. Chen, J., Zhang, L., Zheng, H., Wang, X., Ming, Z.: DeepPoison: Feature Transfer Based Stealthy Poisoning Attack. In: arXiv, 2101.02562 (2021) 100. · arXiv
  45. Liu, T.Y., Yang, Y., Mirzasoleiman, B.: Friendly Noise against Adversarial Noise: A Pow- erful Defense against Data Poisoning Attacks. In: arXiv, 2208.10224 (2023) 104. · arXiv
  46. Yang, Y., Liu, T.Y., Mirzasoleiman, B.: Not All Poisons are Created Equal: Robust Training against Data Poisoning. In: arXiv, 2210.09671 (2022) 105. · arXiv
  47. Li, Y., Lyu, X., Koren, N., Lyu, L., Li, B., Ma, X.: Anti-backdoor learning: Training clean models on poisoned data. In: Neural Information Processing Systems, vol. 34 (2021) 106.
  48. Hong, S., Chandrasekaran, V., Kaya, Y., et al.: On the effectiveness of mitigating data poi- soning attacks with gradient shaping. In: arXiv preprint arXiv:2002.11497 (2020) 107. · arXiv
  49. Tao, L., Feng, L., Yi, J., Huang, S., Chen, S.: Better safe than sorry: Preventing delusive adversaries with adversarial training. In: Advances in Neural Information Processing Sys- tems, vol. 34 (2021) 108.
  50. Si, W., Backes, M., Zhang, Y., Salem, A.: Two-in-One: A Model Hijacking Attack Against Text Generation Models. In: arXiv, 2305.07406 (2023) 111. · arXiv
  51. He, X., Li, Z., Xu, W., et al.: Membership-Doctor: Comprehensive Assessment of Member- ship Inference Against Machine Learning Models. In: arXiv, 2208.10445 (2022) 112. · arXiv
  52. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., Carlini, N.: Dedu- plicating training data makes language models better. In: arXiv, 2107.06499 (2021) 117. · arXiv
  53. Leino, K., Fredrikson, M.: Stolen memories: Leveraging model memorization for calibrated white-box membership inference. In: Proc. 29th USENIX Secur. Symp. (USENIX Secur.), pp. 1605-1622 (2020) 118.

Cited by

Sorted by citations

Related

Authors
Esmradi, A., Yip, D. W., & Chan, C. F.
Affiliations
Hong Kong R&D Centre for Logistics and Supply Chain Management Enabling Technologies (CN)
Corresponding author
Esmradi, Aysan
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
76-95
ISSN
1865-0929 (ISSN-L)
DOI
10.48550/arxiv.2312.10982
arXiv
2312.10982
OpenAlex
W4392720167
Cited by
24
References
48
Citations by year
2026: 3 · 2025: 14 · 2024: 7
FWCI
6.93
Type
conference paper
Language
en
Open access
Licence
Locations
  • Communications in computer and information science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language model security, attack techniques, attack mitigation, adversarial attacks, defense strategy, GPT-4, AI security

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review