CoChat
Sign Up
PreprintCited by 13Adversarial Robustness in Machine LearningCybercrime and Law Enforcement StudiesSpam and Phishing Detection

Malla: Demystifying Real-world Large Language Model Integrated Malicious Services

Open access
Lin, Zilong, Cui, Jian, Liao, Xiaojing, Wang, XiaoFeng·2024·arXiv
DOI 10.48550/arxiv.2401.03315arXiv 2401.03315
Open this paper in CoChat — free accountOpen PDF

Abstract

The underground exploitation of large language models (LLMs) for malicious services (i.e., Malla) is witnessing an uptick, amplifying the cyber threat landscape and posing questions about the trustworthiness of LLM technologies. However, there has been little effort to understand this new cybercrime, in terms of its magnitude, impact, and techniques. In this paper, we conduct the first systematic study on 212 real-world Mallas, uncovering their proliferation in underground marketplaces and exposing their operational modalities. Our study discloses the Malla ecosystem, revealing its significant growth and impact on today's public LLM services. Through examining 212 Mallas, we uncovered eight backend LLMs used by Mallas, along with 182 prompts that circumvent the protective measures of public LLM APIs. We further demystify the tactics employed by Mallas, including the abuse of uncensored LLMs and the exploitation of public LLM APIs through jailbreak prompts. Our findings enable a better understanding of the real-world exploitation of LLMs by cybercriminals, offering insights into strategies to counteract this cybercrime.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper presents the first systematic study of real-world LLM-integrated malicious services (“Malla”), analyzing 14 Malla services and 198 Malla projects (212 total) across underground marketplaces and LLMA hosting platforms. It documents the Malla ecosystem, backend LLMs, jailbreak prompts, pricing/revenue, and the quality of generated malicious code, phishing emails, and phishing sites.

Key findings

  • Across 212 Mallas, malicious code generation was the dominant capability (93.40%), followed by phishing email crafting (41.51%) and scam website creation (17.45%). The study identified eight backend LLMs used by Mallas and 182 jailbreak prompts that circumvent protective measures of public LLM APIs; `gpt-3.5-turbo` was the most frequently targeted, used by 174 Malla projects and two Malla services.
  • Malla pricing ranged from $5 to $199 per month, lower than a traditional malware vendor example priced at $399. A WormGPT case study found revenue exceeding $28K in two months: $28,325 total ($26,783 or 0.24141306 BTC; $1,542 or 2.4115588 ETH), averaging approximately $15,965 per month.
  • Quality varied by service: EscapeGPT and DarkGPT had high malicious-code compilability (66.67% and 65.08%), and none of EscapeGPT’s compile-ready malicious code was detected by VirusTotal. WolfGPT produced phishing emails that all passed format and readability checks, with an evasiveness rate of 0.67; its emails averaged 67.47 words versus 196.32 for other Malla services.
  • In a user study, 78.05% of Malla-created phishing emails were correctly identified as phishing, compared with 69.32% of non-Malla-created phishing emails and 39.20% of benign emails. For phishing websites, 72.93% of Malla-created sites were identified as phishing, versus 57.92% of non-Malla-created sites and 44.81% of benign sites.

Methods

  • The authors collected 13,353 listings from nine underground marketplaces/forums from November 30, 2022 to October 12, 2023, using 145 LLM-related keywords. They extracted 25 Malla listings, identified 14 Malla services, obtained nine Malla service samples, and collected 198 Malla projects from Poe and FlowGPT, along with 45 malicious prompts and 127 visible jailbreak prompts.
  • To reverse-engineer backend LLMs, they trained an authorship attribution classifier using SBERT and Code2Vec vectorization with a CNN, achieving precision and recall of 0.87 on 15,114 responses from six candidate LLMs. To uncover concealed jailbreak prompts, they used an “Ignore the above instructions” prompt-leaking attack, achieving a 93.01% success rate, with Jaro-Winkler similarity 0.88 and semantic textual similarity 0.83.
  • They evaluated Malla-generated content using format compliance, compilability, validity, readability, and evasiveness metrics, with VirusTotal and OOPSpam as detectors. They also ran a case study on exploit payloads against VICIdial and OWASP WebGoat, and a user study via Amazon Mechanical Turk with 83 valid responses.

Limitations

  • The paper states that its findings may be biased because the study relied on available data, including observed Malla services, pre-trained LLM models and jailbreak prompts it could fingerprint, and other accessible resources. It also states that its quality metrics are not meant to measure actual effectiveness or maliciousness of Malla-generated content, but rather content quality.
  • The paper states that the user study excluded sender addresses and URLs because Malla-created content lacked them; this may overestimate the effectiveness of content alone and may not accurately reflect real-world user behavior when full phishing emails or websites are present.
  • The paper notes that determining exact backend LLMs remains challenging for Malla services using self-owned servers, and that some purchase attempts failed due to seller suspicion or non-delivery. It also states that the scope of the real-world Malla-generated instance search was limited by prompt variety/volume and available public datasets.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2401.03315

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 56 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Out of the 14 Malla services, we collected nine Malla samples.

    “Out of the 14 Malla services, we collected nine Malla samples.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Malla offers a more economical option for malicious code generation compared to traditional malware vendors ($5-199 vs $399).

    “Furthermore, a pricing comparison indicates that Malla offers a more economical option for malicious code generation, especially when juxtaposed with the rates of traditional malware vendors ($5-199 vs $399).”
    Quote in the paperNot a substantial claim
  • Findingown data

    Mallas have grown rapidly from April 2023 to October 2023 over six months, across nine underground marketplaces.

    “More specifically, through our analysis of Malla listings across nine underground mar-ketplaces, our study uncovered a rapid increase of Mallas, which has grown from April 2023 to October 2023 over the span of six months.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Miscreants utilized Poe to showcase a 'vouch copy' of their Malla.

    “Interestingly, we observe that miscreants utilized an LLM-integrated application (LLMA) hosting platform, Poe [60] offered by Quora, to showcase a”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [60]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    From the Malla listings, we identified 14 Malla services.

    “From these Malla listings, we identified 14 Malla services as listed in Table 1.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 42
  • May combine two claims. 48
  • The quote only partly states this claim. 4

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Two Malla samples were provided as complimentary copies, while the remaining seven samples were obtained through purchasing.

    “Note that two samples are provided as complimentary copies (a.k.a., voucher copies), while the remaining seven samples were obtained through purchasing.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Malla listing dataset consists of 25 Malla listings from five marketplaces and forums (Hack Forums, XSS.is, BreachForums, Abacus Market, and Kingdom Market), extracted by initially filtering with GPT-4 and subsequently verifying text and images manually.

    “This dataset consists of 25 Malla listings from five marketplaces and forums (Hack Forums, XSS.is, BreachForums, Abacus Market, and Kingdom Market), extracted by initially filtering with GPT-4 and subsequently verifying the text and images within listings manually.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Users interact with the Malla through a GUI or API, facilitating generation of malicious code or phishing emails/sites.

    “After that, the users interact with the Malla (➎) through a graphical user interface (GUI) or an API, facilitating the generation of malicious code or phishing emails/sites (➏).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    Malla services consist of generating various forms of malicious code (e.g., exploit kits, ransomware, worms), writing phishing emails, and creating phishing or scam webpages, among other illicit activities.

    “These services consist of generating various forms of malicious code (e.g., exploit kits, ransomware, worms), as well as writing phishing emails and creating phishing or scam webpages, among other illicit activities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Malla is typically deployed as a web service or hosted on a third-party LLMA hosting platform (e.g., Poe).

    “Typically, Malla is deployed as a web service or hosted on a third-party LLMA hosting platform (e.g., Poe) (➋).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Two predominant techniques leveraged by miscreants are exploitation of uncensored LLMs and jailbreaking of public LLM APIs.

    “In our exploration of Malla artifacts, we identified two predominant techniques leveraged by miscreants: exploitation of uncensored LLMs and jailbreaking of public LLM APIs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    WolfGPT can create phishing emails with a high readability score and manages to bypass OOPSpam.

    “Our findings bring to light that certain Mallas, like DarkGPT and EscapeGPT, excel in producing high-quality malicious code that is both compilable and capable of evading Virus-Total detection [75], while others (e.g., WolfGPT) can create phishing emails with a high readability score and manages to bypass OOPSpam [56].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [56]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingcitation

    DarkGPT and EscapeGPT excel in producing high-quality malicious code that is both compilable and capable of evading VirusTotal detection.

    “Our findings bring to light that certain Mallas, like DarkGPT and EscapeGPT, excel in producing high-quality malicious code that is both compilable and capable of evading Virus-Total detection [75], while others (e.g., WolfGPT) can create phishing emails with a high readability score and manages to bypass OOPSpam [56].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [75]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingcitation

    A Malla provider engages in the misuse of public LLM APIs or uncensored LLMs and deploys them to offer malicious services such as malicious code generation.

    “A Malla provider engages in the misuse of public LLM APIs (e.g., OpenAI API, Llama API) or uncensored LLMs (e.g., Luna AI Llama2 Uncensored, Pygmalion-13B [62]) and deploys them to offer malicious services (➊), such as malicious code generation.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [62]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingreasoning

    Users navigate to associated storefront websites and purchase Malla services.

    “Once identified, they navigate to the associated storefront websites and proceed to purchase the Malla services (➍).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The Malla listings range from 41 to 730 words, averaging 216.75 words.

    “These listings range from 41 to 730 words, averaging 216.75 words.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    DarkGPT purports to be powered by Davinci-003, as claimed on its chat interface.

    “Specifically, DarkGPT purports to be powered by Davinci-003, as claimed on its chat interface.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Whois Domain/IP Lookup discovered that both the domain and IP address concealed registrant information.

    “we discovered that both the domain and IP address concealed registrant information.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The study collected and examined 14 Malla services and 198 Malla projects.

    “Altogether, we collected and examined 14 Malla services and 198 Malla projects in our study.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Each Malla project owner contributes an average of 1.21 Malla projects: 1.15 on Poe and 1.35 on FlowGPT.

    “Each owner contributes an average of 1. 21 Malla projects: 1.15 on Poe and 1.35 on FlowGPT.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Until submission, the study did not receive a response from FlowGPT.

    “Until submission, we did not receive a response from FlowGPT.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The study collected 198 Malla projects, 125 from Poe and 73 from FlowGPT.

    “In our study, we collected 198 Malla projects (125 from Poe and 73 from FlowGPT).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    We collected 575 and 174 LLMA projects hosted on Poe and FlowGPT, respectively.

    “In this way, we collected 575 and 174 LLMA projects hosted on the Poe and FlowGPT, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Efforts to determine backend LLMs for DarkGPT, EscapeGPT, and FreedomGPT were inconclusive, but we uncovered some telling clues.

    “While our efforts to determine the backend LLMs for DarkGPT, EscapeGPT, and FreedomGPT were inconclusive, we did uncover some telling clues.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The study identified 109 Malla project owners on Poe and 54 on FlowGPT.

    “we identified 109 Malla project owners on Poe and 54 on FlowGPT.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The BadGPT domain was registered in August 2023 and is hosted on the Cloudflare platform.

    “the BadGPT domain was registered in August 2023, and it is hosted on the Cloudflare platform.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    The study revealed two primary hosting methods utilized by Malla services: dedicated web servers and third-party LLMA hosting platforms.

    “Our study revealed two primary hosting methods utilized by Malla services:”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    EscapeGPT also announced closure but failed to provide any reason.

    “so did EscapeGPT, which however failed to provide any reason.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Poe solicited Malla project names from the study in November 2023.

    “Poe solicited Malla project names from us in November 2023.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The IP address associated with EscapeGPT was created in August 2015 and is located in Switzerland.

    “it was created in August 2015 and is located in Switzerland.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    BadGPT and XXXGPT utilize OpenAI GPT-3.5 as their backend LLM, while Evil-GPT and WolfGPT employ the APIs of OpenAI Davinci-003 and OpenAI Davinci-002, respectively.

    “From our investigations, we discerned that both BadGPT and XXXGPT utilize OpenAI GPT-3.5 as their backend LLM, while Evil-GPT and WolfGPT employ the APIs of OpenAI Davinci-003 and OpenAI Davinci-002, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    WormGPT announced the closure of its project and feedback thread due to media pressure.

    “WormGPT, announced the closure of its project and its feedback thread due to media pressure”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The analysis revealed the existence of 11 distinct Malla vendors, with an average of 1.27 listings per vendor.

    “Our analysis revealed the existence of 11 distinct Malla vendors, with an average of 1.27 listings per vendor.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The first Malla owner registered in January 2023, with a notable surge in Malla owner accounts starting in April, paralleling the increase of Malla projects.

    “the first Malla owner registered in January 2023, with a notable surge in Malla owner accounts starting in April, paralleling the increase of Malla projects.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    FlowGPT allows users to develop LLMA using Pygmalion-13B, yet neglected to offer explicit usage guidelines.

    “However, FlowGPT allows users to develop LLMA using Pygmalion-13B, yet neglected to offer explicit usage guidelines.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Web hosting platforms featuring cryptocurrency payment processing systems (e.g., Sellix.io and BTCPay Server) have emerged as preferred storefronts for Malla offerings.

    “web hosting platforms featuring cryptocurrency payment processing systems (e.g., Sellix.io and BTCPay Server), which have emerged as preferred storefronts for Malla offerings.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingcitation

    A publicly known jailbreak prompt is similar to EscapeGPT's jailbreak prompt, also positioning the LLM as an evil confidant who has escaped the matrix of rules, policies, and ethics, paralleling EscapeGPT's vendor name EscapeMatrix.

    “Comparing this with publicly known jailbreak prompts, we discovered a similar one [25] that also positions the LLM as an”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [25]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [23]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    Malla-related prompts focus on breaking LLM policies, while public jailbreak prompts include terms like anarchy, unethical, and immoral, highlighting a desire to challenge LLM ethical norms.

    “Interestingly, Malla-related prompts focus on breaking LLM policies (e.g.,”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    There are notably more jailbreak prompts targeting gpt-3.5-turbo than those targeting other LLMs, such as GPT-4.

    “Additionally, there are notably more jailbreak prompts targeting gpt-3.5-turbothan those targeting other LLMs, such as GPT-4.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Emerging Malla services include ObscureGPT and EvilAI on Hack Forums, NanoGPT, hofnar05 Dark-GPT, HackerGPT, and Machiavelli GPT on BreachForums, and Abrax666 on XSS.is.

    “For instance, the emerging Malla services include ObscureGPT [54] and EvilAI [27] on Hack Forums, NanoGPT [51, 52], hofnar05 Dark-GPT [41,42], HackerGPT [39,40], and Machiavelli GPT [48] on BreachForums, Abrax666 [68] on XSS.is, etc.”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [54]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [27]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [51]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [52]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Using the model, the backend LLMs for DarkGPT, EscapeGPT, and FreedomGPT were identified as Davinci-003, GPT-3.5, and Luna AI Llama2 Uncensored, respectively.

    “Using the model, we investigated the backend LLMs for DarkGPT, EscapeGPT, and FreedomGPT, successfully identifying them as Davinci-003, GPT-3.5, and Luna AI Llama2 Uncensored, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The highest cosine similarity score between real-world phishing emails and Malla-created samples was 0.64, and for webpages, it was 0.84.

    “The highest cosine similarity score between the realworld phishing emails and the Malla-created samples was only 0.64, and for webpages, it was 0.84.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    In our study, we released the prompts used by miscreants to generate malicious code and phishing campaigns, along with the prompts in Malla to bypass the existing safety measures of public LLM APIs.

    “In our study, we released the prompts used by miscreants to generate malicious code and phishing campaigns, along with the prompts in Malla to bypass the existing safety measures of public LLM APIs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given

    FlowGPT, offering unrestricted access to uncensored LLMs, has failed to establish or enforce clear usage guidelines on its platform.

    “For instance, FlowGPT, offering unrestricted access to uncensored LLMs, has failed to establish or enforce clear usage guidelines on its platform.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    Combining retained jailbreak prompts with malicious prompts produced a dataset of 3,908 responses.

    “resulting in a dataset comprising 3,908 responses.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Vendors' promotional focus and available demo screenshots highlight malicious code generation and phishing email/site creation, while lacking screenshots, videos, or prompt-response examples for other functionalities in Malla listings.

    “However, vendors' promotional focus and available demo screenshots highlight the generation of malicious code and phishing emails/sites, lacking screenshots, videos, or prompt-response pair examples for other functionalities in Malla listings.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The study identified 14 Malla services and 198 Malla projects.

    “shedding light on as many as 14 Malla services and 198 Malla projects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    MakerGPT and XXXGPT generated code has high Jaccard similarity scores of 0.8421 and 0.8393, respectively, compared with code generated by jailbreak prompts.

    “we find that code generated by the MakerGPT and XXXGPT models exhibits high similarity scores of 0.8421 and 0.8393, respectively, when compared to the code generated by jailbreak prompts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Applying the same operations to each Malla service produced 146 responses.

    “The same operations were applied to each Malla service, resulting in 146 responses.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Malla listings enumerate other functionalities including lead generation, misinformation generation, bank card information collection, underground market navigation, and code vulnerability detection.

    “the product introductions in the listings enumerate other functionalities, including generating lead and misinformation, collecting bank card information, navigating underground markets, and detecting code vulnerabilities.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. The quote only partly states this claim.

    Findingown data

    A total of 96 public jailbreak prompts were retained as operational.

    “A total of 96 jailbreak prompts are retained.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Our study ensured complete data scraping by manually checking measures including real-time monitoring of HTTP status codes and page sizes, session management, and manual CAPTCHA completion.

    “Our study ensured complete data scraping by manually checking a range of measures, including real-time monitoring of HTTP status codes and page sizes, vigilant session management to address session expiration, and manual CAPTCHA completion when access was denied.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Adversaries exploit LLMs to perform malicious activities, ranging from generating sophisticated malicious code and designing vulnerability scanners to crafting convincing phishing emails and creating deceptive scam websites.

    “In these scenarios, adversaries exploit the capabilities of LLMs to perform malicious activities, ranging from generating sophisticated malicious code and designing vulnerability scanners to crafting convincing phishing emails and creating deceptive scam websites.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [34]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [81]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Poe's usage policy explicitly prohibits illegal activities and security violations.

    “Similarly, Poe's usage policy explicitly prohibits illegal activities and security violations [116].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [116]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Several other individuals reported similar experiences with WormGPT on underground forums.

    “Several other individuals reported similar experiences with WormGPT on underground forums [45,82].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [45]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [82]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Moderation mechanisms, such as OpenAI Moderation Endpoint, have been deployed by LLM vendors to provide real-time checks on crafted content.

    “Specifically, to prevent LLMs from crafting harmful content, moderation mechanisms, like Ope-nAI Moderation Endpoint [111], have been deployed by LLM vendors to provide real-time checks on the crafted content.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [111]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    OpenAI and Llama have defined disallowed usage scenarios, including the generation of malicious code and engagement in fraudulent or deceptive activities like spam and scams.

    “For instance, OpenAI and Llama have defined various disallowed usage scenarios, including the generation of malicious code and engagement in fraudulent or deceptive activities like spam and scams [108,112].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [108]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [112]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The study applied a critical analysis informed by the Menlo Report and Cybersecurity Research Ethical Frameworks to assess ethical considerations and risks.

    “we apply a critical analysis informed by the principles outlined in the Menlo Report [87] and Cybersecurity Research Ethical Frameworks [101].”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [87]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [101]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Users have highlighted scams about FraudGPT, DarkBERT, and DarkBARD.

    “Likewise, users have highlighted scams about FraudGPT, DarkBERT, and DarkBARD [82].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [82]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The study queried LLMA projects using a single registered account, adhering strictly to daily query limitations.

    “we queried LLMA projects using a single registered account, adhering strictly to daily query limitations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Most other Malla vendor accounts seem primarily dedicated to promoting Malla services.

    “and seem primarily dedicated to promoting Malla services.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study promptly deleted chat history after each testing session to reduce impact on target platforms.

    “After each testing session, we promptly deleted the chat history to reduce the impact on target platforms.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The study crafted comprehensive guidelines with IRB counsel governing conversations and purchase interactions.

    “we crafted comprehensive guide-lines (e.g., always asking for voucher copy, not deanonymizing seller) governing our conversations and purchase interactions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Study design and conductnone given

    The experiments comply with the principles identified in the Menlo Report and were approved by the organization's IRB.

    “These experiments comply with the principles identified in the Menlo Report, and were approved by our organization's IRB.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The analysis focused exclusively on FlowGPT's Malla projects because Poe does not offer project creation time.

    “our analysis focused exclusively on FlowGPT's Malla projects, as Poe does not offer the project creation time.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The study took utmost care to ensure that its testing did not disrupt services, harm users, or lead to unintentional damages.

    “we took utmost care to ensure that our testing did not disrupt services, harm users, or lead to any unintentional damages.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study was approved by the institution's IRB.

    “our study has been approved by our institution's institutional review board (IRB).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Malla services facilitate transactions via cryptocurrency, likely aiming to harness its pseudonymous attributes to veil financial transactions.

    “they facilitate transactions via cryptocurrency, likely aiming to harness its pseudonymous attributes to veil financial transactions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    FlowGPT's laissez-faire approach provides a fertile ground for miscreants to misuse the LLMs.

    “This laissez-faire approach essentially provides a fertile ground for miscreants to misuse the LLMs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Pygmalion-13B is often categorized as an uncensored model due to its efficacy in roleplay scenarios, even when simulating ethically questionable or NSFW roles.

    “Notably, Pygmalion-13B is often categorized as an”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [62]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Both similarity scores fall below the thresholds for considering a real-world phishing email or webpage as potentially produced by Malla services.

    “Both scores fall below the thresholds for considering a real-world phishing email or webpage as potentially produced by Malla services.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    One potential reason for gpt-3.5-turbo's popularity could be its absence of query restrictions compared to others.

    “One potential reason for gpt-3.5-turbo'spopularity could be its absence of query restrictions compared to others.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The pre-train and prompt paradigm is a commonly employed approach for constructing LLM-integrated applications.

    “As discussed in § 2, the”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Davinci-002 and Davinci-003 were exclusively accessible via their OpenAI APIs, which were deprecated on January 4, 2024.

    “On the other hand, Davinci-002 and Davinci-003 were exclusively accessible via their OpenAI APIs [35] that were deprecated on January 4, 2024.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: The absence of open-sourced models for Davinci-002 and Davinci-003 in searched repositories means they are exclusively accessible via APIs.

  • May combine two claims.

    Interpretationown data

    Our research presents the first systematic examination of the real-world misused LLMs for cybercriminal activities, analyzing 14 Malla services and 198 Malla projects in depth.

    “Our research presents the first systematic examination of the real-world misused LLMs for cybercriminal activities, analyzing 14 Malla services and 198 Malla projects in depth.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prior research has shown that LLMs can be weaponized across diverse domains, including misinformation propagation, deepfake user profile creation, spear phishing, attack and malware generation, and hateful memes.

    “Past research showcased how LLMs can be weaponized across diverse domains, such as misinformation propagation [91,102,125], deepfake user profile creation [109], spear phishing campaigns [97,98], attack and malware generation [97] and the generation of hateful memes [119].”
    Quote in the paperNot a substantial claim0 of 6 sources checked
    • cites [91]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [102]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [125]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [109]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [97]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [98]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Interpretationnone given

    These two stakeholders are relatively overlooked within the Malla ecosystem.

    “Importantly, our study sheds light on two relatively overlooked stakeholders within the Malla ecosystem”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Malla services exploit reputable and public web hosting services like Sellix.io, BTCPay Server, and Netlify for malicious purposes, minimizing the risk of detection and disruption.

    “they subtly exploit reputable and public web hosting services, like Sellix.io,BTCPay Server, and Netlify, for malicious purposes, thus, minimizing the risk of detection and disruption.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    The preference for subscription-based pricing might stem from subscription prices being typically much lower than those for a complete model, and a more affordable price can attract a larger customer base and foster buyer loyalty.

    “This preference might stem from the fact that subscription prices are typically much lower than those for a complete model, and a more affordable price can attract a larger customer base and foster buyer loyalty.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Utilizing or wrapping uncensored LLMs reduces the overhead associated with data collection and model training.

    “This approach reduces the overhead associated with data collection and model training.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Pygmalion-13B, as an open-sourced model, has made its trained model available on HuggingFace.

    “13B, as an open-sourced model, has made its trained model available on HuggingFace [64].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [64]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The scope of the real-world Malla-generated instances experiment is limited by the variety and volume of prompts used for generation and by the limited public datasets of malicious code and phishing content available for comparison.

    “It is important to note that the scope of this experiment is limited by the variety and volume of prompts used for generation, as well as the limited public datasets of malicious code and phishing content available for comparison.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductreasoning

    Evaluating Python/C code with definite syntax checkers and compilers enables precise quality evaluation of Malla-generated malicious code.

    “Evaluating Python/C code by definite syntax checkers and compilers enables a precise quality evaluation to Malla-generated malicious code.”
    Quote in the paperNot a substantial claim
78 of 78

  • The paper relies on this without support.

    Little prior work has systematically explored real-world Malla samples and the underground ecosystem behind them.

    “However, to our knowledge, little has been done so far to systematically explore real-world Malla samples, and understand the underground ecosystem behind them and their security implications.”

  • The authors state this as an assumption.

    The miscreant possesses capabilities similar to those of an ordinary user, requiring no special privileges or access to pre-trained commercial LLMs.

    “Note that we assume the miscreant possesses capabilities similar to those of an ordinary user, requiring no special privileges or access to pre-trained commercial LLMs.”

  • The paper relies on this without support.

    The miscreant aims to circumvent safety measures, including content filters like OpenAI's moderator, that are typically provided by LLM vendors to prevent generation of prohibited content.

    “For this purpose, the miscreant aims to circumvent safety measures, including content filters like OpenAI's moderator [111], that are typically provided by LLM vendors to prevent the generation of prohibited content.”

  • The paper relies on this without support.

    Poe and FlowGPT enable development and hosting of LLM-integrated applications by providing access to various LLMs (e.g., GPT-3.5 and Pygmalion-13B), alongside capability to utilize custom text prompts.

    “These platforms enable the development and hosting of LLM-integrated applications by providing access to various LLMs (e.g., GPT-3.5 and Pygmalion-13B), alongside the capability to utilize custom text prompts.”

  • The paper relies on this without support.

    FlowGPT's reported usage volumes accurately reflect actual usage.

    “per project usage volumes provided by FlowGPT, the average usage volume of Malla projects on FlowGPT is 10,603.32.”

  • The paper relies on this without support.

    The backend LLMs documented on hosting pages accurately represent the backend LLMs used by Malla projects.

    “the hosting page documents the backend LLMs used by these projects.”

  • The paper relies on this without support.

    The jailbreak prompts uncovered via prompt injection are the actual prompts used by projects concealing them.

    “for those projects concealing prompts, we uncovered their prompts via the prompt injection approach”

  • The paper relies on this without support.

    The 100 popular non-Malla projects sampled from FlowGPT's main page are a valid comparison group for Malla projects.

    “we manually sampled 100 popular non-Malla projects from FlowGPT's main page”

  • The paper relies on this without support.

    VirusTotal's detection history is a comprehensive record of previously reported malicious code.

    “VirusTotal, a widely used malicious code detection service, maintains a historical record of malicious code reports.”

  • The paper relies on this without support.

    The ground truth dataset of 143 Malla projects with visible jailbreak prompts is representative of the 55 Malla projects with hidden prompts.

    “We evaluated the prompt leaking attack on the ground truth dataset, consisting of 143 Malla projects with visible jailbreak prompts.”

  • The paper relies on this without support.

    The absence of open-sourced models for Davinci-002 and Davinci-003 in searched repositories means they are exclusively accessible via APIs.

    “Despite thorough searches on HuggingFace, GitHub, and other repositories, no open-sourced models or code for these two models were found.”

  • The paper relies on this without support.

    LLM-generated code is formatted as Markdown code blocks.

    “Since LLM-generated code is formatted as Markdown code blocks”
12 of 12

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Benign-email response percentage does not match its count

    Minor

    The paper reports 250 valid responses to benign emails. The stated 93 responses are 37.20% of 250, not 39.20%.

    Reported 39.2 · recomputed 37.2

  • Listings per vendor average is incorrect

    Minor

    The paper reports 25 Malla listings and 11 vendors. That is about 2.27 listings per vendor, not 1.27.

    Reported 1.27 · recomputed 2.272727

  • Similarity claim has a counterexample in Table 7

    Minor

    The paper says similarity is consistently higher against public jailbreak prompts than against other Mallas. For WormGPT, Table 7 reports a lower average against public jailbreak prompts (0.1619) than against other Mallas (0.1704).

3 of 3

Cites

Flagged first, then by citations

45 of the 83 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. Damon McCoy, Andreas Pitsillidis, Jordan Grant, Nicholas Weaver, Christian Kreibich, Brian Krebs, Ge- offrey Voelker, Stefan Savage, and Kirill Levchenko. {PharmaLeaks}: Understanding the business of online pharmaceutical affiliate programs. In 21st USENIX Se- curity Symposium (USENIX Security 12), pages 1-16, 2012.
  2. Jaron Mink, Licheng Luo, Natã M Barbosa, Olivia Figueira, Yang Wang, and Gang Wang. {DeepPhish}: Understanding user trust towards artificially generated profiles in online social networks. In 31st USENIX Security Symposium, pages 1669-1686, 2022.
  3. Peng Wang, Xiaojing Liao, Yue Qin, and XiaoFeng Wang. Into the deep web: Understanding e- commercefraud from autonomous chat with cybercrim- inals. In Proceedings of the ISOC Network and Dis- tributed System Security Symposium (NDSS), 2020, 2020.

Cited by

Sorted by citations

Related

Authors
Lin, Z., Cui, J., Liao, X., & Wang, X.
Affiliations
Indiana University Bloomington (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2401.03315
arXiv
2401.03315
OpenAlex
W4390722926
Semantic Scholar
97d4813d0fd37000a440d295ac6afad623234fcf
Cited by
13
References
83
Citations by year
2026: 1 · 2025: 7 · 2024: 5
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1850725, 1801432)
  • Intelligence Advanced Research Projects Activity
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Information Systems › Cybercrime and Law Enforcement Studies
  • Physical Sciences › Computer Science › Information Systems › Spam and Phishing Detection
Keywords
large language models, LLM security

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-18.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-18.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review