CoChat
Sign Up
PreprintCited by 24Adversarial Robustness in Machine LearningInformation and Cyber SecurityWeb Application Security Vulnerabilities

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Mapping LLM Security Landscapes: A Comprehensive Stakeholder Risk Assessment Proposal

Open access
Pankajakshan, Rahul, Biswal, Sumitra, Govindarajulu, Yuvaraj, Gressel, Gilad·2024·arXiv
DOI 10.48550/arxiv.2403.13309arXiv 2403.13309
Open this paper in CoChat — free accountOpen PDF

Abstract

The rapid integration of Large Language Models (LLMs) across diverse sectors has marked a transformative era, showcasing remarkable capabilities in text generation and problem-solving tasks. However, this technological advancement is accompanied by significant risks and vulnerabilities. Despite ongoing security enhancements, attackers persistently exploit these weaknesses, casting doubts on the overall trustworthiness of LLMs. Compounding the issue, organisations are deploying LLM-integrated systems without understanding the severity of potential consequences. Existing studies by OWASP and MITRE offer a general overview of threats and vulnerabilities but lack a method for directly and succinctly analysing the risks for security practitioners, developers, and key decision-makers who are working with this novel technology. To address this gap, we propose a risk assessment process using tools like the OWASP risk rating methodology which is used for traditional systems. We conduct scenario analysis to identify potential threat agents and map the dependent system components against vulnerability factors. Through this analysis, we assess the likelihood of a cyberattack. Subsequently, we conduct a thorough impact analysis to derive a comprehensive threat matrix. We also map threats against three key stakeholder groups: developers engaged in model fine-tuning, application developers utilizing third-party APIs, and end users. The proposed threat matrix provides a holistic evaluation of LLM-related risks, enabling stakeholders to make informed decisions for effective mitigation strategies. Our outlined process serves as an actionable and comprehensive tool for security practitioners, offering insights for resource management and enhancing the overall system security.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.13309

Cites

0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 The system design of the university virtual assistant emphasizes the diverse security measures implemented for each component. An authenticated user's prompt undergoes optimization and validation/filtering for prompt injection within the prompt optimization module. Contextual information for the query is accessed based on authorization. The query and context serve as inputs to the LLM, and the resulting response from the LLM is logged and cached. Before being sent to the user, the response undergoes filtering. A rate-limiting mechanism has been implemented at the Gateway to safeguard against DoS attacks.
Fig. 1 The system design of the university virtual assistant emphasizes the diverse security measures implemented for each component. An authenticated user's prompt undergoes optimization and validation/filtering for prompt injection within the prompt optimization module. Contextual information for the query is accessed based on authorization. The query and context serve as inputs to the LLM, and the resulting response from the LLM is logged and cached. Before being sent to the user, the response undergoes filtering. A rate-limiting mechanism has been implemented at the Gateway to safeguard against DoS attacks.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 18

  1. Paul F Christiano et al. "Deep reinforcement learning from human preferences". In: Advances in neural infor- mation processing systems 30 (2017).
  2. Ashish Vaswani et al. "Attention is all you need". In: Advances in neural information processing systems 30 (2017).
  3. Gary McGraw et al. "An architectural risk analysis of machine learning systems: Toward more secure machine learning". In: Berryville Institute of Machine Learning, Clarke County, VA. Accessed on: Mar 23 (2020).
  4. Carl Wilhjelm and Awad A Younis. "A threat analysis methodology for security requirements elicitation in machine learning based systems". In: 2020 IEEE 20th International Conference on Software Quality, Relia- bility and Security Companion (QRS-C). IEEE. 2020, pp. 426-433.
  5. Edward J Hu et al. "Lora: Low-rank adaptation of large language models". In: arXiv preprint arXiv:2106.09685 (2021). · arXiv
  6. Yuntao Bai et al. "Training a helpful and harmless assis- tant with reinforcement learning from human feedback". In: arXiv preprint arXiv:2204.05862 (2022). · arXiv
  7. Long Ouyang et al. "Training language models to follow instructions with human feedback". In: Advances in Neural Information Processing Systems 35 (2022), pp. 27730-27744.
  8. Qingtian Zou et al. "Attacks on ML Systems: From Se- curity Analysis to Attack Mitigation". In: International Conference on Information Systems Security. Springer. 2022, pp. 119-138.
  9. Gelei Deng et al. "Jailbreaker: Automated jailbreak across multiple large language model chatbots". In: arXiv preprint arXiv:2307.08715 (2023). · arXiv
  10. Isabel O Gallegos et al. "Bias and fairness in large language models: A survey". In: arXiv preprint arXiv:2309.00770 (2023). · arXiv
  11. Kai Greshake et al. "More than you've asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Mod- els". In: arXiv e-prints (2023), arXiv-2302.
  12. Yangsibo Huang et al. "Catastrophic jailbreak of open-source llms via exploiting generation". In: arXiv preprint arXiv:2310.06987 (2023). · arXiv
  13. Haoran Li et al. "Multi-step jailbreaking privacy at- tacks on chatgpt". In: arXiv preprint arXiv:2304.05197 (2023). · arXiv
  14. Hugo Touvron et al. "Llama 2: Open foundation and fine-tuned chat models". In: arXiv preprint arXiv:2307.09288 (2023). · arXiv
  15. Yotam Wolf et al. "Fundamental limitations of align- ment in large language models". In: arXiv preprint arXiv:2304.11082 (2023). · arXiv
  16. Jiashu Xu et al. "Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Lan- guage Models". In: arXiv preprint arXiv:2305.14710 (2023). · arXiv
  17. Yue Zhang et al. "Siren's song in the ai ocean: A survey on hallucination in large language models". In: arXiv preprint arXiv:2309.01219 (2023). · arXiv
  18. Tianyu Cui et al. "Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems". In: arXiv preprint arXiv:2401.05778 (2024). · arXiv

Cited by

Sorted by citations

See all 24

Related

Metadata

Authors
Pankajakshan, R., Biswal, S., Govindarajulu, Y., & Gressel, G.
Affiliations
  • Amrita Vishwa Vidyapeetham (IN)
  • Robert Bosch (India) (IN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.13309
arXiv
2403.13309
OpenAlex
W4393062808
Cited by
12
References
0
Citations by year
2026: 2 · 2025: 4 · 2024: 6
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
  • Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
Keywords
large language models, LLM security, risk assessment, threat matrix, scenario analysis, impact analysis, stakeholder risk, vulnerability mapping, fine-tuning, security risk mitigation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review