ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Personalizing generative models offers a way to guide image generation with user-provided references. Current personalization methods can invert an object or concept into the textual conditioning space and compose new natural sentences for text-to-image diffusion models. However, representing and editing specific visual attributes such as material, style, and layout remains a challenge, leading to a lack of disentanglement and editability. To address this problem, we propose a novel approach that leverages the step-by-step generation process of diffusion models, which generate images from low to high frequency information, providing a new perspective on representing, generating, and editing images. We develop the Prompt Spectrum Space P*, an expanded textual conditioning space, and a new image representation method called ProSpect. ProSpect represents an image as a collection of inverted textual token embeddings encoded from per-stage prompts, where each prompt corresponds to a specific generation stage (i.e., a group of consecutive steps) of the diffusion model. Experimental results demonstrate that P* and ProSpect offer better disentanglement and controllability compared to existing methods. We apply ProSpect in various personalized attribute-aware image generation applications, such as image-guided or text-driven manipulations of materials, style, and layout, achieving previously unattainable results from a single image input without fine-tuning the diffusion models. Our source code is available at https://github.com/zyxElsa/ProSpect.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Taming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514 · cited 1×
“Recent personalization methods [Gal et al. 2023a; Huang et al. 2023c; Kumari et al. 2023a; Ruiz et al. 2023] learn the textual conditioning of a common concept from a set of images and then use text prompts to create new scenarios that incorporate the concept.”
MethodDreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2022) · arXiv · doi:10.48550/arxiv.2208.12242 · Cited by 2,442
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295 · cited 9×
“It can be seen that the diffusion model acts different optimizations to token embeddings 𝑝 𝑖 at different stages to reconstruct the given image.”
BackgroundMethodInfluentialAttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks
Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923
Instructpix2pix: Learning to follow image editing instructions
Brooks, T., Holynski, A., & Efros, A. A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01764 · Cited by 1,466 · cited 4×
“By replacing them with embeddings of other images, we can achieve attribute transfer, as shown in the 2 nd row of Fig.”
BackgroundMethodResultInfluentialNull-text Inversion for Editing Real Images using Guided Diffusion Models
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., & Cohen-Or, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00585 · Cited by 747 · cited 1×
“Compared to previous personalization approaches, ProSpect offers better transferability of diverse image visual attributes.”
BackgroundHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Multi-Concept Customization of Text-to-Image Diffusion
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., & Zhu, J.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00192 · Cited by 674 · cited 1×
“Recent personalization methods [Gal et al. 2023a; Huang et al. 2023c; Kumari et al. 2023a; Ruiz et al. 2023] learn the textual conditioning of a common concept from a set of images and then use text prompts to create new scenarios that incorporate the concept.”
Method
Cited by
Diffusion-Based Visual Art Creation: A Survey and New Perspectives
Wang, B., Chen, Q., & Wang, Z. (2025) · ACM Computing Surveys · doi:10.1145/3728459 · Cited by 40
Ma, X., & Huo, Y. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103755 · Cited by 30
Style-A-Video: Agile Diffusion for Arbitrary Text-based Video Style Transfer
Huang, N., Zhang, Y., & Dong, W. (2023) · arXiv · doi:10.48550/arxiv.2305.05464 · Cited by 15
Self-citationOne Model to Rig Them All: Diverse Skeleton Rigging with UniRig
Zhang, J.-P., Pu, C.-F., Guo, M.-H., Cao, Y.-P., & Hu, S.-M. (2025) · ACM Transactions on Graphics · doi:10.1145/3730930 · Cited by 12
“Tapmo [Zhang et al. 2023a], inspired by SfPT, employs a Mesh Handle Predictor and Motion Diffusion to generate motion sequences and retarget them to diverse characters.”
InfluentialSkin Tokens: A Learned Compact Representation for Unified Autoregressive Rigging
Zhang, J.-P., Pu, C.-F., Guo, M.-H., Cao, Y.-P., & Hu, S.-M. (2026) · arXiv · doi:10.48550/arxiv.2602.04805
AniGen: Unified S 3 Fields for Animatable 3D Asset Generation
Huang, Y.-H., Zou, Z.-X., He, Y., Chang, C., Pu, C.-F., Yang, Z., Guo, Y.-C., Cao, Y.-P., & Qi, X. (2026) · ACM Transactions on Graphics · doi:10.1145/3811297
“The rapid advancement of generative 3D models [Chen et al. 2025b; Li et al. 2025c; Wu et al. 2024a; Xiang et al. 2025b; Zhang et al. 2023, 2024b] has begun to democratize 3D content creation, enabling the synthesis of visually stunning geometry and appearance from simple text or image prompts.”
Metadata
- Authors
- Zhang, Y., Dong, W., Tang, F., Huang, N., Huang, H., Ma, C., Lee, T.-Y., Deussen, O., & Xu, C.
- Venue
- ACM Transactions on Graphics
- Published
- 2023-12-05
- Volume
- 42
- Issue
- 6
- Pages
- 1-14
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W4389334989
- Cited by
- 100
- References
- 12
- FWCI
- 9.09
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.