Cited by 461Generative Adversarial Networks and Image SynthesisMultimodal Machine Learning ApplicationsAdversarial Robustness in Machine Learning

Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models

Open access
Chefer, Hila, Alaluf, Yuval, Vinker, Yael, Wolf, Lior, Cohen-Or, Daniel·2023·ACM Transactions on Graphics

Abstract

Recent text-to-image generative models have demonstrated an unparalleled ability to generate diverse and creative imagery guided by a target text prompt. While revolutionary, current state-of-the-art diffusion models may still fail in generating images that fully convey the semantics in the given text prompt. We analyze the publicly available Stable Diffusion model and assess the existence of catastrophic neglect , where the model fails to generate one or more of the subjects from the input prompt. Moreover, we find that in some cases the model also fails to correctly bind attributes ( e.g. , colors) to their corresponding subjects. To help mitigate these failure cases, we introduce the concept of Generative Semantic Nursing (GSN) , where we seek to intervene in the generative process on the fly during inference time to improve the faithfulness of the generated images. Using an attention-based formulation of GSN, dubbed Attend-and-Excite , we guide the model to refine the cross-attention units to attend to all subject tokens in the text prompt and strengthen --- or excite --- their activations, encouraging the model to generate all subjects described in the text prompt. We compare our approach to alternative approaches and demonstrate that it conveys the desired concepts more faithfully across a range of text prompts. Code is available at our project page: https://attendandexcite.github.io/Attend-and-Excite/.

Integrity signals

Registry

Crossref · 10.1145/3592116

Cites

19 of 19 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • ISO/IEC DTS 4213 is an ongoing work item of ISO/IEC JTC 1/SC 42 on assessment of machine learning classification performance.

    “ISO/IEC DTS (Draft Technical Specification) 4,213 “Information technology—Artificial Intelligence— Assessment of machine learning classification performance”12”

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., & Cohen-Or, D.
Affiliations
  • Tel Aviv University (IL)
  • Tel Aviv University, Tel Aviv, Israel
Venue
ACM Transactions on Graphics
Published
2023-07-26
Volume
42
Issue
4
Pages
1-10
ISSN
  • 0730-0301 (ISSN-L)
  • 1557-7368
OpenAlex
W4385270985
Cited by
461
References
19
Citations by year
2026: 120 · 2025: 143 · 2024: 166 · 2023: 32
FWCI
41.84
Type
journal article
Language
en
Open access
Locations
  • ACM Transactions on Graphics (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
text-to-image generation, diffusion models, attention guidance, cross-attention, Stable Diffusion, inference-time intervention, semantic fidelity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review