CoChat
Sign Up
PreprintCited by 29Advanced Malware Detection TechniquesSoftware Engineering ResearchSecurity and Verification in Computing

Leveraging Large Language Models to Detect npm Malicious Packages

Open access
Zahan, Nusrat, Burckhardt, Philipp, Lysenko, Mikola, Aboukhadijeh, Feross, Williams, Laurie·2024·arXiv
DOI 10.48550/arxiv.2403.12196arXiv 2403.12196
Open this paper in CoChat — free accountOpen PDF

Abstract

Existing malicious code detection techniques demand the integration of multiple tools to detect different malware patterns, often suffering from high misclassification rates. Therefore, malicious code detection techniques could be enhanced by adopting advanced, more automated approaches to achieve high accuracy and a low misclassification rate. The goal of this study is to aid security analysts in detecting malicious packages by empirically studying the effectiveness of Large Language Models (LLMs) in detecting malicious code. We present SocketAI, a malicious code review workflow to detect malicious code. To evaluate the effectiveness of SocketAI, we leverage a benchmark dataset of 5,115 npm packages, of which 2,180 packages have malicious code. We conducted a baseline comparison of GPT-3 and GPT-4 models with the state-of-the-art CodeQL static analysis tool, using 39 custom CodeQL rules developed in prior research to detect malicious Javascript code. We also compare the effectiveness of static analysis as a pre-screener with SocketAI workflow, measuring the number of files that need to be analyzed. and the associated costs. Additionally, we performed a qualitative study to understand the types of malicious activities detected or missed by our workflow. Our baseline comparison demonstrates a 16% and 9% improvement over static analysis in precision and F1 scores, respectively. GPT-4 achieves higher accuracy with 99% precision and 97% F1 scores, while GPT-3 offers a more cost-effective balance at 91% precision and 94% F1 scores. Pre-screening files with a static analyzer reduces the number of files requiring LLM analysis by 77.9% and decreases costs by 60.9% for GPT-3 and 76.1% for GPT-4. Our qualitative analysis identified data theft, execution of arbitrary code, and suspicious domain categories as the top detected malicious packages.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.12196

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Zahan et al. (2024) employs GPT-3 and GPT-4 to detect potential malware in the npm ecosystem by analyzing JavaScript packages.

    “Zahan et al. (2024) employs GPT-3 and GPT-4 to detect potential malware in the npm ecosystem by analyzing JavaScript packages.”

    From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Zahan et al. (2024)

  • Zahan et al. (2024) demonstrates that LLMs can effectively pinpoint malware with higher precision and lower false positive rates compared to static analysis tools.

    “By comparing LLMs' performance with static analysis tools, the paper demonstrates that LLMs can effectively pinpoint malware with higher precision and lower false positive rates.”

    From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Zahan et al. (2024)

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 5

  1. BROWN, T., MANN, B., RYDER, N., SUBBIAH, M., KAPLAN, J. D., DHARIWAL, P., NEELAKANTAN, A., SHYAM, P., SASTRY, G., ASKELL, A., ET AL. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
  2. ČARNOGURSK Ỳ, M. Attacks on package managers. Bachelor Thesis, Masaryk University, Brünn, Tschechien (2019).
  3. JI, Z., LEE, N., FRIESKE, R., YU, T., SU, D., XU, Y., ISHII, E., BANG, Y. J., MADOTTO, A., AND FUNG, P. Survey of hallucination in natural language generation. ACM Computing Surveys.
  4. MILJE, A. A. Detecting malicious python packages in the python package index (pypi). Master's thesis, NTNU, 2022.
  5. ZAHAN, N., BURCKHARDT, P., LYSENKO, M., ABOUKHADIJEH, F., AND WILLIAMS, L. Malwarebench: Malware samples are not enough. In 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR) (2024), IEEE, pp. 728-732.

Cited by

Sorted by citations

Related

Authors
Zahan, N., Burckhardt, P., Lysenko, M., Aboukhadijeh, F., & Williams, L.
Affiliations
North Carolina State University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2403.12196
arXiv
2403.12196
OpenAlex
W4393055583
Cited by
3
References
0
Citations by year
2025: 1 · 2024: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2207008)
  • North Carolina State University
Topics
  • Physical Sciences › Computer Science › Signal Processing › Advanced Malware Detection Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Security and Verification in Computing
Keywords
malicious package detection, npm packages, large language models, GPT-3, GPT-4, data theft, arbitrary code execution, security cost reduction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review