The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
Open accessAbstract
Large-scale text-to-image generative models have been a revolutionary breakthrough in the evolution of generative AI, synthesizing diverse images with highly complex visual concepts. However, a pivotal challenge in leveraging such models for real-world content creation is providing users with control over the generated content. In this paper, we present a new framework that takes text-to- image synthesis to the realm of image-to-image translation - given a guidance image and a target text prompt as input, our method harnesses the power of a pre-trained text-to-image diffusion model to generate a new image that complies with the target text, while preserving the semantic layout of the guidance image. Specifically, we observe and empirically demonstrate that fine-grained control over the generated structure can be achieved by manipulating spatial features and their self-attention inside the model. This results in a simple and effective approach, where features extracted from the guidance image are directly injected into the generation process of the translated image, requiring no training or fine-tuning. We demonstrate high-quality results on versatile text-guided image translation tasks, including translating sketches, rough drawings and animations into realistic images, changing the class and appearance of objects in a given image, and modifying global qualities such as lighting and color.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Diffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 4×
“Recently, text-to-image generative models [13,29,35,37, 41] have demonstrated unprecedented capabilities in generating high-quality and diverse images from text, capturing complex visual concepts (e.”
BackgroundInfluentialDenoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2022) · arXiv · doi:10.48550/arxiv.2208.12242 · Cited by 2,442
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295
Palette: Image-to-Image Diffusion Models
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., & Norouzi, M. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530757 · Cited by 1,717
Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., & Salesin, D. H. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383295 · Cited by 1,518
Contrastive Learning for Unpaired Image-to-Image Translation
Park, T., Efros, A. A., Zhang, R., & Zhu, J.-Y. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58545-7_19 · Cited by 1,442
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00823 · Cited by 1,250
Matching Local Self-Similarities across Images and Videos
Shechtman, E., & Irani, M. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.383198 · Cited by 1,076
Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation
Richardson, E., Alaluf, Y., Patashnik, O., Nitzan, Y., Azar, Y., Shapiro, S., & Cohen-Or, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00232 · Cited by 1,072
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., & Lischinski, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00209 · Cited by 1,023
Imagic: Text-Based Real Image Editing with Diffusion Models
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., & Irani, M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00582 · Cited by 767
Self-citationBlended Diffusion for Text-driven Editing of Natural Images
Avrahami, O., Lischinski, D., & Fried, O. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01767 · Cited by 764
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Designing an encoder for StyleGAN image manipulation
Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., & Cohen-Or, D. (2021) · ACM Transactions on Graphics · doi:10.1145/3450626.3459838 · Cited by 686
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., & Cohen-Or, D. (2022) · arXiv · doi:10.48550/arxiv.2208.01618 · Cited by 611
Sketch2Photo: internet image montage
Chen, T., Cheng, M.-M., Tan, P., Shamir, A., & Hu, S.-M. (2009) · ACM Transactions on Graphics · doi:10.1145/1618452.1618470 · Cited by 574
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation
Kim, G., Kwon, T., & Ye, J. C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00246 · Cited by 570
Cascaded Diffusion Models for High Fidelity Image Generation
Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., & Salimans, T. (2021) · arXiv · doi:10.48550/arxiv.2106.15282 · Cited by 514
StyleGAN-NADA: CLIP-guided domain adaptation of image generators
Gal, R., Patashnik, O., Maron, H., Bermano, A. H., Chechik, G., & Cohen-Or, D. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530164 · Cited by 500
Swapping Autoencoder for Deep Image ManipulationPreprint
Park, T., Zhu, J.-Y., Wang, O., Lu, J., Shechtman, E., Efros, A. A., & Zhang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00653 · Cited by 388
Prompt-to-Prompt Image Editing with Cross Attention Control
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., & Cohen‐Or, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.01626 · Cited by 360
Avrahami, O., Fried, O., & Lischinski, D. (2023) · ACM Transactions on Graphics · doi:10.1145/3592450 · Cited by 345 · cited 1×
“, a GAN [7, 15, 27, 32] or a diffusion model [2, 3, 22, 25].”
BackgroundMake-A-Scene: Scene-Based Text-to-Image Generation with Human Priors
Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., & Taigman, Y. (2022) · arXiv · doi:10.48550/arxiv.2203.13131 · Cited by 332
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors
Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., & Taigman, Y. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19784-0_6 · Cited by 323
ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement
Alaluf, Y., Patashnik, O., & Cohen-Or, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00664 · Cited by 302
Compositional Visual Generation with Composable Diffusion Models
Liu, N., Li, S., Du, Y., Torralba, A., & Tenenbaum, J. B. (2022) · arXiv · doi:10.48550/arxiv.2206.01714 · Cited by 295
Compositional Visual Generation with Composable Diffusion Models
Liu, N., Li, S., Du, Y., Torralba, A., & Tenenbaum, J. B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19790-1_26 · Cited by 282
Style Transfer by Relaxed Optimal Transport and Self-Similarity
Kolkin, N., Salavon, J., & Shakhnarovich, G. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01029 · Cited by 281
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance
Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, L., & Raff, E. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19836-6_6 · Cited by 279
Pretraining is All You Need for Image-to-Image TranslationPreprint
Wang, T., Zhang, T., Zhang, B., Ouyang, H., Chen, D., Chen, Q., & Wen, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.12952 · Cited by 260
Text2LIVE: Text-Driven Layered Image and Video Editing
Bar-Tal, O., Ofri-Amar, D., Fridman, R., Kasten, Y., & Dekel, T. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19784-0_41 · Cited by 222
Self-citationData-driven hallucination of different times of day from a single outdoor photo
Shih, Y., Paris, S., Durand, F., & Freeman, W. T. (2013) · ACM Transactions on Graphics · doi:10.1145/2508363.2508419 · Cited by 187
Splicing ViT Features for Semantic Appearance Transfer
Tumanyan, N., Bar-Tal, O., Bagon, S., & Dekel, T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01048 · Cited by 153
Self-citationDiffusion-based Image Translation using Disentangled Style and Content RepresentationPreprint
Kwon, G., & Ye, J. C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.15264 · Cited by 129
SDEdit: Image Synthesis and Editing with Stochastic Differential EquationsPreprint
Meng, C., Song, Y. S., Song, J., Wu, J., Zhu, J.-Y., & Ermon, S. (2021) · arXiv (Cornell University) · Cited by 103
FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space OptimizationPreprint
Liu, X., Gong, C., Wu, L., Zhang, S., Su, H., & Liu, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.01573 · Cited by 95
Sparse, Smart Contours to Represent and Edit Images
Dekel, T., Gan, C., Krishnan, D., Liu, C., & Freeman, W. T. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00370 · Cited by 84
Self-citationFlexIT: Towards Flexible Semantic Image Translation
Couairon, G., Grechka, A., Verbeek, J. J., Schwenk, H., & Cord, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01773 · Cited by 67
Detecting and sketching the common
Bagon, S., Brostovski, O., Galun, M., et al. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540233 · Cited by 42
Self-citationA Style-aware Discriminator for Controllable Image Translation
Kim, K. H., Park, S., Jeon, E., Kim, T., & Kim, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01770 · Cited by 33
Efros and Freeman Image Quilting Algorithm for Texture Synthesis
Raad, L., & Galerne, B. (2017) · Image Processing On Line · doi:10.5201/ipol.2017.171 · Cited by 22
Image Shape Manipulation from a Single Augmented Training Sample
Vinker, Y., Horwitz, E., Zabari, N., & Hoshen, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01351 · Cited by 17
Cited by
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547
Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation
Wu, J. Z., Ge, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., & Shou, M. Z. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00701 · Cited by 575
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing
Cao, M., Wang, X., Qi, Z., Shan, Y., Qie, X., & Zheng, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02062 · Cited by 413
OmniGen: Unified Image Generation
Xiao, S., Wang, Y., Zhou, J., Yuan, H., Xing, X., Yan, R., Li, C., Wang, S., Huang, T., & Liu, Z. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.01241 · Cited by 75
Training-Free Consistent Text-to-Image Generation
Tewel, Y., Kaduri, O., Gal, R., Kasten, Y., Wolf, L., Chechik, G., & Atzmon, Y. (2024) · ACM Transactions on Graphics · doi:10.1145/3658157 · Cited by 68
Metadata
- Authors
- Tumanyan, N., Geyer, M., Bagon, S., & Dekel, T.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 1921-1930
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386113271
- Cited by
- 609
- References
- 52
- FWCI
- 69.25
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.