The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Zero-shot Image-to-Image Translation
Open accessAbstract
Large-scale text-to-image generative models have shown their remarkable ability to synthesize diverse, high-quality images. However, directly applying these models for real image editing remains challenging for two reasons. First, it is hard for users to craft a perfect text prompt depicting every visual detail in the input image. Second, while existing models can introduce desirable changes in certain regions, they often dramatically alter the input content and introduce unexpected changes in unwanted regions. In this work, we introduce pix2pix-zero, an image-to-image translation method that can preserve the original image’s content without manual prompting. We first automatically discover editing directions that reflect desired edits in the text embedding space. To preserve the content structure, we propose cross-attention guidance, which aims to retain the cross-attention maps of the input image throughout the diffusion process. Finally, to enable interactive editing, we distill the diffusion model into a fast conditional GAN. We conduct extensive experiments and show that our method outperforms existing and concurrent works for both real and synthetic image editing. In addition, our method does not need additional training for these edits and can directly use the existing pre-trained text-to-image diffusion model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Self-citationImage-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · cited 1×
“One line of work is to train conditional GANs that translate an input image from one domain to a target domain [Baek et al. 2021; Choi et al. 2018; Huang et al. 2018; Isola et al. 2017; Lee et al. 2018; Liu et al. 2017; Park et al. 2020, 2019; Sangkloy et al. 2017; Wang et al. 2018; Zhu et al. 2017], which often requi…”
BackgroundSelf-citationHigh-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Self-citation2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017) · doi:10.1109/cvpr35066.2017 · Cited by 7,141
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00813 · Cited by 6,332
International Conference on Learning Representations (ICLR 2013)
松尾, 豊. (2013) · 人工知能学会誌 = Journal of Japanese Society for Artificial Intelligence · Cited by 6,286
Semantic Image Synthesis With Spatially-Adaptive Normalization
Park, T., Liu, M.-Y., Wang, T.-C., & Zhu, J.-Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00244 · Cited by 2,974
Self-citationZero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
Score-Based Generative Modeling through Stochastic Differential Equations
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020) · arXiv · doi:10.48550/arxiv.2011.13456 · Cited by 1,953 · cited 1×
“Other methods [Choi et al. 2021; Song et al. 2020] proposes to modify the diffusion process by incorporating conditioning user inputs but have been only applied to single-category models.”
MethodInstructpix2pix: Learning to follow image editing instructions
Brooks, T., Holynski, A., & Efros, A. A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01764 · Cited by 1,466 · cited 3×
“2022a], InstructPix2Pix [Brooks et al. 2022], PITI [Wang et al.”
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.595 · Cited by 1,232
Generative Visual Manipulation on the Natural Image Manifold
Zhu, J.-Y., Krähenbühl, P., Shechtman, E., & Efros, A. A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_36 · Cited by 1,171
Self-citationImage2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?
Abdal, R., Qin, Y., & Wonka, P. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00453 · Cited by 1,076
Null-text Inversion for Editing Real Images using Guided Diffusion Models
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., & Cohen-Or, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00585 · Cited by 747
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Image2StyleGAN++: How to Edit the Embedded Images?
Abdal, R., Qin, Y., & Wonka, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00832 · Cited by 501
CogView: Mastering Text-to-Image Generation via Transformers
Ding, M., Yang, Z., Hong, W., Zheng, W., Zhou, C., Yin, D., Lin, J., Zou, X., Shao, Z., Yang, H., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.13290 · Cited by 450
Probability and Random Processes for Electrical and Computer Engineers
Gubner, J. A. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511813610 · Cited by 438
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors
Gafni, O., Polyak, A., Ashual, O., Sheynin, S., Parikh, D., & Taigman, Y. (2022) · arXiv · doi:10.48550/arxiv.2203.13131 · Cited by 332 · cited 1×
“Recently, large-scale text-to-image models have dramatically improved the image quality and diversity by training on an internet-scale text-image datasets [Ding et al. 2021; Gafni et al. 2022; Ramesh et al. 2022, 2021; Saharia et al. 2022b; Yu et al. 2022].”
BackgroundSemantic photo manipulation with a generative image priorPreprint
Bau, D., Strobelt, H., Peebles, W., Wulff, J., Zhou, B., Zhu, J.-Y., & Torralba, A. B. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 291
Self-citationExploiting Deep Generative Prior for Versatile Image Restoration and Manipulation
Pan, X., Zhan, X., Dai, B., Lin, D., Loy, C. C., & Luo, P. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3115428 · Cited by 277
Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
Zhao, S., Cui, J., Sheng, Y., Dong, Y., Liang, X., Chang, E. I., & Xu, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.10428 · Cited by 224 · cited 2×
“Figure 7 shows the results obtained by distilling using Co-Mod-GAN [67].”
MethodSemantic photo manipulation with a generative image prior
Bau, D., Strobelt, H., Peebles, W., Wulff, J., Zhou, B., Zhu, J.-Y., & Torralba, A. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323023 · Cited by 207
Self-citationCLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions
Abdal, R., Zhu, P., Femiani, J. C., Mitra, N. J., & Wonka, P. (2022) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/3528233.3530747 · Cited by 116
Transforming and Projecting Images into Class-Conditional Generative Networks
Huh, M., Zhang, R., Zhu, J.-Y., Paris, S., & Hertzmann, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58536-5_2 · Cited by 114
Self-citationSpatially-Adaptive Multilayer Selection for GAN Inversion and Editing
Parmar, G., Li, Y., Lu, J., Zhang, R., Zhu, J.-Y., & Singh, K. K. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01111 · Cited by 58
Self-citation
Cited by
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547
MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing
Cao, M., Wang, X., Qi, Z., Shan, Y., Qie, X., & Zheng, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02062 · Cited by 413
DIRE for Diffusion-Generated Image Detection
Wang, Z., Bao, J., Zhou, W., Wang, W., Hu, H., Chen, H., & Li, H. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02051 · Cited by 354
Training-Free Consistent Text-to-Image Generation
Tewel, Y., Kaduri, O., Gal, R., Kasten, Y., Wolf, L., Chechik, G., & Atzmon, Y. (2024) · ACM Transactions on Graphics · doi:10.1145/3658157 · Cited by 68
Diffusion-Based Visual Art Creation: A Survey and New Perspectives
Wang, B., Chen, Q., & Wang, Z. (2025) · ACM Computing Surveys · doi:10.1145/3728459 · Cited by 40
Metadata
- Authors
- Parmar, G., Kumar Singh, K., Zhang, R., Li, Y., Lu, J., & Zhu, J.-Y.
- Venue
- Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings
- Publisher
- ACM
- Published
- 2023-07-23
- Pages
- 1-11
- OpenAlex
- W4385527149
- Cited by
- 361
- References
- 30
- FWCI
- 41.05
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.