CoChat
Sign Up
Cited by 93

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

Nightshade: Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models

Shan, Shawn, Ding, Wenxin, Passananti, Josephine, Wu, Stanley, Zheng, Haitao, Zhao, Ben Y.·2024·2024 IEEE Symposium on Security and Privacy (SP)
DOI 10.1109/sp54263.2024.00207
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/sp54263.2024.00207

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Overview of prompt-specific poison attacks against generic text-to-image generative models. (a) User generates poison data (text and image pairs) designed to corrupt a given concept C (i.e. a keyword like 'dog'), then posts them online; (b) Model trainer scrapes data from online webpages to train its generative model; c) Given prompts that contain C , poisoned model generates incorrect images.
Figure 1 Overview of prompt-specific poison attacks against generic text-to-image generative models. (a) User generates poison data (text and image pairs) designed to corrupt a given concept C (i.e. a keyword like 'dog'), then posts them online; (b) Model trainer scrapes data from online webpages to train its generative model; c) Given prompts that contain C , poisoned model generates incorrect images.CC-BY · From the PDF
Figure 2 Concept sparsity in LAION-Aesthetic measured by word and semantic frequencies. Note the long-tail distribution and log-scale on both Y axes.
Figure 2 Concept sparsity in LAION-Aesthetic measured by word and semantic frequencies. Note the long-tail distribution and log-scale on both Y axes.CC-BY · From the PDF
Figure 3 Samples of dirty-label poison data in terms of mismatched text/image pairs, curated to attack the concept 'dog.' Here 'cat' was chosen by the attacker as the destination concept A .
Figure 3 Samples of dirty-label poison data in terms of mismatched text/image pairs, curated to attack the concept 'dog.' Here 'cat' was chosen by the attacker as the destination concept A .CC-BY · From the PDF
Figure 4 Example images generated by the clean (unpoisoned) and poisoned SD-XL models with different # of poison data. The attack effect is apparent with 1000 poisoning samples, but not at 500 samples.
Figure 4 Example images generated by the clean (unpoisoned) and poisoned SD-XL models with different # of poison data. The attack effect is apparent with 1000 poisoning samples, but not at 500 samples.CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Figure 6 Examples of Nightshade poison images (perturbed with a LPIPS budget of 0 . 07 ) and their corresponding original clean images.
Figure 6 Examples of Nightshade poison images (perturbed with a LPIPS budget of 0 . 07 ) and their corresponding original clean images.CC-BY · From the PDF
Image 7
CC-BY · From the PDF
Image 8
CC-BY · From the PDF
Image 9
CC-BY · From the PDF
Image 10
CC-BY · From the PDF
Image 11
CC-BY · From the PDF
Image 12
CC-BY · From the PDF
Figure 12 Cross-attention maps of a model before and after poisoning. Poisoned model highlights destination A (banana, fork) instead of concept C (hat, handbag).
Figure 12 Cross-attention maps of a model before and after poisoning. Poisoned model highlights destination A (banana, fork) instead of concept C (hat, handbag).CC-BY · From the PDF
Figure 14 Image generated from different prompts by a poisoned SD-XL model where concept 'dog' is poisoned. Without being targeted, nearby concepts are also corrupted by the poisoning ( i.e., bleed through effect). The SD-XL model is poisoned with 200 poison samples.
Figure 14 Image generated from different prompts by a poisoned SD-XL model where concept 'dog' is poisoned. Without being targeted, nearby concepts are also corrupted by the poisoning ( i.e., bleed through effect). The SD-XL model is poisoned with 200 poison samples.CC-BY · From the PDF
Figure 15 Image generated from different prompts by a poisoned SD-XL model where concept 'fantasy art' is poisoned. Without being targeted, related prompts are also corrupted by the poisoning ( i.e., bleed through effect), while unrelated prompts face limited impact. The SD-XL model is poisoned with 200 poison samples.
Figure 15 Image generated from different prompts by a poisoned SD-XL model where concept 'fantasy art' is poisoned. Without being targeted, related prompts are also corrupted by the poisoning ( i.e., bleed through effect), while unrelated prompts face limited impact. The SD-XL model is poisoned with 200 poison samples.CC-BY · From the PDF
Figure 16 Two independent poison attacks (poisoned concept: dog and fantasy art) on the same model can co-exist together.
Figure 16 Two independent poison attacks (poisoned concept: dog and fantasy art) on the same model can co-exist together.CC-BY · From the PDF
Figure 17 Images generated by poisoned SD-XL models as attacker poisons an increasing number of concepts. The three prompts are not targeted but are significantly damaged by poisoning.
Figure 17 Images generated by poisoned SD-XL models as attacker poisons an increasing number of concepts. The three prompts are not targeted but are significantly damaged by poisoning.CC-BY · From the PDF
Image 18
CC-BY · From the PDF
Image 19
CC-BY · From the PDF
Image 20
CC-BY · From the PDF
Figure 20 Attack success rate of the simple, dirty-label poisoning attack, measured by the CLIP classifier, vs. # of poison data injected, when attacking each of three models SD-V2, SD-XL, DeepFloyd (continuous training).
Figure 20 Attack success rate of the simple, dirty-label poisoning attack, measured by the CLIP classifier, vs. # of poison data injected, when attacking each of three models SD-V2, SD-XL, DeepFloyd (continuous training).CC-BY · From the PDF
Figure 22 Attack success rate of the simple poison attack against LD-CC, measured by the CLIP classifier. The simple poisoning attack is more effective at corrupting style concepts than object concepts. The same applies to attacks against SDV2, SD-XL, DeepFloyd.
Figure 22 Attack success rate of the simple poison attack against LD-CC, measured by the CLIP classifier. The simple poisoning attack is more effective at corrupting style concepts than object concepts. The same applies to attacks against SDV2, SD-XL, DeepFloyd.CC-BY · From the PDF
Image 23
CC-BY · From the PDF
Figure 24 Success rate of the simple poisoning attack (rated by CLIP classifier) correlates strongly with concept sparsity measured by semantic frequency. Results for LD-CC. Same trend observed on SD-V2, SD-XL, DeepFloyd.
Figure 24 Success rate of the simple poisoning attack (rated by CLIP classifier) correlates strongly with concept sparsity measured by semantic frequency. Results for LD-CC. Same trend observed on SD-V2, SD-XL, DeepFloyd.CC-BY · From the PDF
Figure 25 Nightshade's attack success rate (CLIP-based) decreases when model trainer continuously trains an already-poisoned model on an increasing number of clean data. The base model is poisoned with 100, 300, and 500 poison data samples.
Figure 25 Nightshade's attack success rate (CLIP-based) decreases when model trainer continuously trains an already-poisoned model on an increasing number of clean data. The base model is poisoned with 100, 300, and 500 poison data samples.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 58

  1. Rincon, 2023, Virtually try on clothes with a new AI shopping feature, Google Blog
  2. 2023, Virtual try-on for clothing: The future of fashion?, 3dlook.ai
  3. 2023, Create logos, videos, banners, mockups with AI in 2 minutes, designs.ai
  4. Andersen, 2022, The Alt-Right Manipulated My Comic. Then A.I. Claimed It
  5. Yang, 2022, Why Artists are Fed Up with AI Art
  6. Adobe Max conference
  7. 10.1109/CVPR52729.2023.00393 · DOI
  8. 10.1109/CVPR52729.2023.00391 · DOI
  9. 10.1145/3581783.3612108 · DOI
  10. Shan, Glaze: Protecting artists from style mimicry by text-to-image models
  11. Liang, Adversarial example does good: Preventing painting imitation from diffusion models via adversarial examples
  12. 10.1109/CVPR.2019.00595 · DOI
  13. Ding, Cogview: Mastering text-to-image generation via transformers
  14. Ramesh, Zero-shot text-to-image generation
  15. 10.1109/CVPR46437.2021.00356 · DOI
  16. 2022, What the heck is Civitai?
  17. 10.1109/CVPR52729.2023.02155 · DOI · Paper page
  18. 10.1109/CVPR46437.2021.00614 · DOI
  19. 10.1109/CVPR.2018.00175 · DOI · Paper page
  20. 10.1145/3485832.3485837 · DOI
  21. 10.1609/aaai.v34i07.6871 · DOI · Paper page
  22. Zhu, Transferable clean-label poisoning attacks on deep neural nets
  23. 10.1109/SP.2019.00031 · DOI
  24. Qiao, Defending neural backdoors via generative distribution modeling
  25. Shan, Poison forensics: Traceback of data poisoning attacks in neural networks
  26. 10.1609/aaai.v35i9.16971 · DOI
  27. 10.1145/3319535.3354209 · DOI
  28. Severi, Explanation-guided backdoor poisoning attacks against malware classifiers
  29. Bagdasaryan, Blind backdoors in deep learning models
  30. Schuster, You autocomplete me: Poisoning vulnerabilities in neural code completion
  31. 10.1109/cvpr52733.2024.02299 · DOI
  32. Yang, Data poisoning attacks against multimodal encoders
  33. 10.1109/spw63631.2024.00019 · DOI
  34. Gu, Badnets: Identifying vulnerabilities in the machine learning model supply chain
  35. Biggio, Support vector machines under adversarial label noise
  36. Schuhmann, 2022, LAION-aesthetics, LAION.AI
  37. Bond, A survey of wordnets and their licenses
  38. Midjourney user prompts; generated images (250k)
  39. Fellbaum, 2005, WordNet and wordnets. Encyclopedia of Language and Linguistics.
  40. 10.1109/sp54263.2024.00061 · DOI
  41. 10.1109/EuroSP51992.2021.00021 · DOI
  42. Schwarzschild, Just how toxic is data poisoning? a unified benchmark for backdoor and data poisoning attacks, 9389
  43. 10.1109/CVPR.2018.00068 · DOI · Paper page
  44. Nocedal, 2006, Numerical optimization
  45. 10.1109/cvprw63382.2024.00182 · DOI
  46. 10.1007/978-3-319-10602-1_48 · DOI · Paper page
  47. 10.1145/3658644.3670306 · DOI
  48. Vahdat, Score-based generative modeling in latent space
  49. Park, Benchmark for compositional text-to-image synthesis
  50. Shan, Fawkes: Protecting privacy against unauthorized deep learning models
  51. 10.1109/SP46214.2022.9833572 · DOI
  52. Shumailov, 2023, The curse of recursion: Training on generated data makes models forget, arxiv:2305.17493 · arXiv
  53. Lee, Personalizing text-to-image generation with visual prompts using BLIP-2
  54. 10.1109/CVPR46437.2021.00252 · DOI
  55. 10.1145/3372297.3417231 · DOI
  56. Xiang, 2022, AI is probably using your images and it’s not easy to opt out, Motherboard, Tech by Vice
  57. David, 2023, Now you can block OpenAI’s web crawler, The Verge
  58. Neel, Descent-to-delete: Gradient-based methods for machine unlearning

Cited by

See all 93

Metadata

Authors
Shan, S., Ding, W., Passananti, J., Wu, S., Zheng, H., & Zhao, B. Y.
Affiliations
University of Chicago,Department of Computer Science
Venue
2024 IEEE Symposium on Security and Privacy (SP)
Publisher
IEEE
Published
2024-05-19
Pages
807-825
DOI
10.1109/sp54263.2024.00207
Cited by
93
References
21
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review