The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
Blended Diffusion for Text-driven Editing of Natural Images
Open accessAbstract
Natural language offers a highly intuitive interface for image editing. In this paper, we introduce the first solution for performing local (region-based) edits in generic natural images, based on a natural language description along with an ROI mask. We achieve our goal by leveraging and combining a pretrained language-image model (CLIP), to steer the edit towards a user-provided text prompt, with a denoising diffusion probabilistic model (DDPM) to generate natural-looking results. To seamlessly fuse the edited region with the unchanged parts of the image, we spatially blend noised versions of the input image with the local text-guided diffusion latent at a progression of noise levels. In addition, we show that adding augmentations to the diffusion process mitigates adversarial results. We compare against several baselines and related methods, both qualitatively and quantitatively, and show that our method outperforms these solutions in terms of overall realism, ability to preserve the background and matching the text. Finally, we show several text-driven editing applications, including adding a new object to an image, removing/replacing/altering existing objects, background replacement, and image extrapolation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Research works printed but not linked: 14
- murdock, 0, The big sleep BigGANxCLIP
- mansimov, 2016, Generating images from captions with attention, CoRR, abs 1511 2793
- nichol, 0, Improved denoising diffusion probabilistic models, International Conference on Machine Learning, 8162
- li, 2019, Controllable text-to-image generation, ArXiv Preprint
- vaswani, 2017, Attention is all you need, Advances in neural information processing systems, 5998
- zhu, 0, Indomain GAN inversion for real image editing, European Conference on Computer Vision, 592
- xia, 2021, GAN inversion: A survey, ArXiv Preprint
- 10.1145/1531326.1531373 · doi:10.1145/1531326.1531373
- fried, 2020, Editing self-image, 63, 70
- goh, 2021, Distillation
- bau, 2021, Paint by word, ArXiv Preprint
- blake, 0, Trump is reportedly suggesting the ‘access hollywood’ tape was fake news
- crowson, 0, CLIP guided diffusion HQ 256x256
- crowson, 0, VQGAN+CLIP
Cited by
Adding Conditional Control to Text-to-Image Diffusion Models
Zhang, L., Rao, A., & Agrawala, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00355 · Cited by 4,547
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2022) · arXiv · doi:10.48550/arxiv.2208.12242 · Cited by 2,442
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295
Diffusion Models in Vision: A Survey
Croitoru, F.-A., Hondru, V., Ionescu, R. T., & Shah, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3261988 · Cited by 1,859
Diffusion Models: A Comprehensive Survey of Methods and Applications
Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., & Yang, M.-H. (2024) · ACM Computing Surveys · doi:10.1145/3626235 · Cited by 1,704
Instructpix2pix: Learning to follow image editing instructions
Brooks, T., Holynski, A., & Efros, A. A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01764 · Cited by 1,466
Imagic: Text-Based Real Image Editing with Diffusion Models
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., & Irani, M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00582 · Cited by 767
Null-text Inversion for Editing Real Images using Guided Diffusion Models
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., & Cohen-Or, D. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00585 · Cited by 747
DiffusionDet: Diffusion Model for Object Detection
Chen, S., Sun, P., Song, Y., & Luo, P. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01816 · Cited by 657
Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation
Tumanyan, N., Geyer, M., Bagon, S., & Dekel, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00191 · Cited by 609
Exploiting Diffusion Prior for Real-World Image Super-Resolution
Wang, J., Yue, Z., Zhou, S., Chan, K. C. K., & Loy, C. C. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02168-7 · Cited by 439
Paint by Example: Exemplar-based Image Editing with Diffusion Models
Yang, B., Gu, S., Zhang, B., Zhang, T., Chen, X., Sun, X., Chen, D., & Wen, F. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01763 · Cited by 356
Avrahami, O., Fried, O., & Lischinski, D. (2023) · ACM Transactions on Graphics · doi:10.1145/3592450 · Cited by 345
Self-citationDIRE for Diffusion-Generated Image Detection
Wang, Z., Bao, J., Zhou, W., Wang, W., Hu, H., Chen, H., & Li, H. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02051 · Cited by 354
Proceedings of the 32nd ACM International Conference on Multimedia
(2024) · doi:10.1145/3664647 · Cited by 288
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance
Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, L., & Raff, E. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19836-6_6 · Cited by 279
Generative Diffusion Prior for Unified Image Restoration and Enhancement
Fei, B., Lyu, Z., Pan, L., Zhang, J., Yang, W., Luo, T., Zhang, B., & Dai, B. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00958 · Cited by 279
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models
Höllein, L., Cao, A., Owens, A., Johnson, J., & Nießner, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00727 · Cited by 167
Diffusion Facial Forgery Detection
Cheng, H., Guo, Y., Wang, T., Nie, L., & Kankanhalli, M. (2024) · Proceedings of the 32nd ACM International Conference on Multimedia · doi:10.1145/3664647.3680797 · Cited by 47
Generative AI in depth: A survey of recent advances, model variants, and real-world applications
Yazdani, S., Singh, A., Saxena, N., Wang, Z., Palikhe, A., Pan, D., Pal, U., Yang, J., & Zhang, W. (2025) · Journal of Big Data · doi:10.1186/s40537-025-01247-x · Cited by 40
Rafner, J., Zana, B., Bang Hansen, I., Ceh, S., Sherson, J., Benedek, M., & Lebuda, I. (2025) · Creativity Research Journal · doi:10.1080/10400419.2025.2587803 · Cited by 38
The Chosen One: Consistent Characters in Text-to-Image Diffusion Models
Avrahami, O., Hertz, A., Vinker, Y., Arar, M., Fruchter, S., Fried, O., Cohen-Or, D., & Lischinski, D. (2024) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers · doi:10.1145/3641519.3657430 · Cited by 29
Self-citationThe Chosen One: Consistent Characters in Text-to-Image Diffusion Models
Avrahami, O., Hertz, A., Vinker, Y., Arar, M., Fruchter, S., Fried, O., Cohen-Or, D., & Lischinski, D. (2023) · arXiv · doi:10.48550/arxiv.2311.10093 · Cited by 29
Self-citationGlob-Diffusion: A Global Consistent Diffusion Model for Large-Scale Image Generation
Kang, Y., Shi, H., Liu, H., Xie, W., Fang, L., & Bruzzone, L. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3643743 · Cited by 1
Metadata
- Authors
- Avrahami, O., Lischinski, D., & Fried, O.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 18187-18197
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3216352822
- Cited by
- 764
- References
- 50
- FWCI
- 63.26
- Type
- conference paper
- Language
- en
- Open access
- green
- Funders
- Israel Science Foundation (2492/20,1574/21)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.