CoChat
Sign Up
Cited by 345Generative Adversarial Networks and Image SynthesisDigital Media Forensic DetectionMultimodal Machine Learning Applications

Blended Latent Diffusion

Open access
Avrahami, Omri, Fried, Ohad, Lischinski, Dani·2023·ACM Transactions on Graphics
DOI 10.1145/3592450
Open this paper in CoChat — free account

Abstract

The tremendous progress in neural image generation, coupled with the emergence of seemingly omnipotent vision-language models has finally enabled text-based interfaces for creating and editing images. Handling generic images requires a diverse underlying generative model, hence the latest works utilize diffusion models, which were shown to surpass GANs in terms of diversity. One major drawback of diffusion models, however, is their relatively slow inference time. In this paper, we present an accelerated solution to the task of local text-driven editing of generic images, where the desired edits are confined to a user-provided mask. Our solution leverages a text-to-image Latent Diffusion Model (LDM), which speeds up diffusion by operating in a lower-dimensional latent space and eliminating the need for resource-intensive CLIP gradient calculations at each diffusion step. We first enable LDM to perform local image edits by blending the latents at each step, similarly to Blended Diffusion. Next we propose an optimization-based solution for the inherent inability of LDM to accurately reconstruct images. Finally, we address the scenario of performing local edits using thin masks. We evaluate our method against the available baselines both qualitatively and quantitatively and demonstrate that in addition to being faster, it produces more precise results.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3592450

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Image inpainting approaches provide a reasonable baseline but their results lack temporal and spatial stability.

    “While image inpainting approaches [1; 28] provide a reasonable baseline, the results lack temporal and spatial stability.”

    From SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix (2024) · cites this paper as [1]

    Section 5.1 (p. 6) compares the method with image-editing baselines, and Section 4.2 (pp. 4–5) describes spatial defects such as lost background detail and visible seams. The paper does not assess temporal stability, so it does not support that part of the claim. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Blended Latent Diffusion demonstrates significantly improved results over the Stable Diffusion model in preserving foreground objects while translating the background.

    “Recent work proposed the Blended Latent Diffusion [1] which demonstrates significantly improved results over the Stable Diffusion model in preserving foreground objects while also translating the background.”

    From SafeSea: Synthetic Data Generation for Adverse & Low Probability Maritime Conditions (2023) · cites this paper as [1]

    Figure 1 (p. 1) shows background alteration, and Section 4.1 (p. 4) says blending keeps areas outside the edit mask unchanged, which can preserve foreground objects when the background is edited. The comparisons in Section 5.1 (p. 8) do not include Stable Diffusion, so the claimed significant improvement over that model is unsupported. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • BLD produces better image quality than SD Inpainting.

    “As expected the BLD produces better image quality than the SD Inpainting as also shown in [1]”

    From SafeSea: Synthetic Data Generation for Adverse & Low Probability Maritime Conditions (2023) · cites this paper as [1]

    Section 5.1 (pp. 6–8) reports better visual quality for BLD than several listed baselines, but SD Inpainting is not among them. Searches for “SD Inpainting” and “Stable Diffusion inpainting” found no comparison with that method in our copy. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • The image passing rate for BLD after applying the SafeSea filter is 71.85%.

    “| Image passing rate (in %) | 8.16% | 71.85% |”

    From SafeSea: Synthetic Data Generation for Adverse & Low Probability Maritime Conditions (2023) · cites this paper as [1]

    The quantitative comparison in §5.1 (p. 8) reports precision, diversity, and human-evaluation results, not an image passing rate after SafeSea filtering. Searches for SafeSea, “71.85%,” and filter passing rates found no passage addressing the claim. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

  • The diffusion model employed by BLD has constraints in generating realistic wave and whitecap patterns.

    “Furthermore, the diffusion model employed by BLD [1] has constraints in generating realistic wave and whitecap patterns.”

    From SafeSea: Synthetic Data Generation for Adverse & Low Probability Maritime Conditions (2023) · cites this paper as [1]

    The Limitations & Conclusions section (p. 9) discusses output images that may lack overall realism, but attributes this to CLIP-based ranking rather than difficulty generating waves or whitecaps. Searches for waves, whitecaps, and water found no passage in our copy addressing the claimed pattern-generation constraint. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 19 · Cited by 345

Research works printed but not linked: 63

  1. Rameen Abdal, Yipeng Qin, and Peter Wonka. 2019. Image2stylegan: How to embed images into the StyleGAN latent space?. In Proc. ICCV . 4432-4441.
  2. Rameen Abdal, Yipeng Qin, and Peter Wonka. 2020. Image2stylegan++: How to edit the embedded images?. In Proc. CVPR . 8296-8305.
  3. Oron Ashual, Shelly Sheynin, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. 2022. KNN-Diffusion: Image Generation via Large-Scale Retrieval. arXiv preprint arXiv:2204.02849 (2022). · arXiv
  4. Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. 2022a. SpaText: Spatio-Textual Representation for Controllable Image Generation. arXiv preprint arXiv:2211.14305 (2022). · arXiv
  5. Omri Avrahami, Dani Lischinski, and Ohad Fried. 2022b. Blended diffusion for textdriven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18208-18218.
  6. Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. 2022. Text2LIVE: Text-Driven Layered Image and Video Editing. arXiv preprint arXiv:2204.02491 (2022). · arXiv
  7. David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba. 2021. Paint by word. arXiv preprint arXiv:2103.10951 (2021). · arXiv
  8. David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. 2020. Semantic photo manipulation with a generative image prior. arXiv preprint arXiv:2005.07727 (2020). · arXiv
  9. Sam Bond-Taylor, Peter Hessey, Hiroshi Sasaki, Toby P Breckon, and Chris G Willcocks. 2021. Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes. arXiv preprint arXiv:2111.12701 (2021). · arXiv
  10. Gary Bradski and Adrian Kaehler. 2000. OpenCV. Dr. Dobb's journal of software tools 3 (2000), 2.
  11. Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018). · arXiv
  12. Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. 2021. ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . IEEE, 1434714356.
  13. Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. 2022. VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance. arXiv preprint arXiv:2204.08583 (2022). · arXiv
  14. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proc. CVPR . Ieee, 248-255.
  15. Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems 34 (2021).
  16. Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. 2021. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems 34 (2021), 19822-19835.
  17. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations .
  18. Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. 2021b. ImageBART: Bidirectional context with multinomial diffusion for autoregressive image synthesis. Advances in Neural Information Processing Systems 34 (2021).
  19. Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021a. Taming transformers for high-resolution image synthesis. In Proc. CVPR . 12873-12883.
  20. Ohad Fried, Jennifer Jacobs, Adam Finkelstein, and Maneesh Agrawala. 2020. Editing Self-Image. Commun. ACM 63, 3, 70-79. https://doi.org/10.1145/3326601 · DOI
  21. Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022. StyleGAN-NADA: CLIP-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG) 41, 4 (2022), 1-13.
  22. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems . 2672-2680.
  23. Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. 2021. Vector quantized diffusion model for text-to-image synthesis. arXiv preprint arXiv:2111.14822 (2021). · arXiv
  24. Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2022. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022). · arXiv
  25. Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. In Proc. NeurIPS .
  26. Minghui Hu, Yujie Wang, Tat-Jen Cham, Jianfei Yang, and PN Suganthan. 2021. Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generation. arXiv preprint arXiv:2112.01799 (2021). · arXiv
  27. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020. Analyzing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 8110-8119.
  28. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014). · arXiv
  29. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 (2013). · arXiv
  30. Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. 2021. More Control for Free! Image Synthesis with Semantic Diffusion Guidance. arXiv preprint arXiv:2112.05744 (2021). · arXiv
  31. Elman Mansimov, Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. 2016. Generating Images from Captions with Attention. CoRR abs/1511.02793 (2016). · arXiv
  32. Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. 2021. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations .
  33. Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021). · arXiv
  34. Alexander Quinn Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models. In Proc. ICML . 8162-8171.
  35. Roni Paiss, Hila Chefer, and Lior Wolf. 2022. No Token Left Behind: Explainability-Aided Image Classification and Generation. arXiv preprint arXiv:2204.04908 (2022). · arXiv
  36. Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. 2021. Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
  37. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019), 8026-8037.
  38. Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. 2021. StyleCLIP: Text-driven manipulation of StyleGAN imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 2085-2094.
  39. Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson image editing. In Proc. ACM SIGGRAPH 2003 . 313-318.
  40. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021). · arXiv
  41. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 (2022). · arXiv
  42. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092 (2021). · arXiv
  43. Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. 2019. Generating diverse highfidelity images with VQ-VAE-2. Advances in neural information processing systems 32 (2019).
  44. Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016. Generative adversarial text to image synthesis. In Proc. ICLR . 1060-1069.
  45. Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. 2021. Pivotal tuning for latent-based editing of real images. arXiv preprint arXiv:2106.05744 (2021). · arXiv
  46. Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. 2022a. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings . 1-10.
  47. Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. 2022b. Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. arXiv preprint arXiv:2205.11487 (2022). · arXiv
  48. Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 (2021). · arXiv
  49. Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In Proc. ICML . 2256-2265.
  50. Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising Diffusion Implicit Models. In International Conference on Learning Representations .
  51. Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning . PMLR, 6105-6114.
  52. Rotem Tzaban, Ron Mokady, Rinon Gal, Amit H Bermano, and Daniel Cohen-Or. 2022. Stitch it in Time: GAN-Based Facial Editing of Real Videos. arXiv preprint arXiv:2201.08361 (2022). · arXiv
  53. Arash Vahdat, Karsten Kreis, and Jan Kautz. 2021. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems 34 (2021), 1128711302.
  54. Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. Advances in neural information processing systems 30 (2017).
  55. Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural Discrete Representation Learning. In NIPS .
  56. Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi. 2022. CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP. arXiv preprint arXiv:2203.00386 (2022). · arXiv
  57. Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. 2021. GAN inversion: A survey. arXiv preprint arXiv:2101.05278 (2021). · arXiv
  58. Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. 2022. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. arXiv preprint arXiv:2206.10789 (2022). · arXiv
  59. Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. 2017. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proc. ICCV . 5907-5915.
  60. Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. 2018b. StackGAN++: Realistic image synthesis with stacked generative adversarial networks. IEEE transactions on pattern analysis and machine intelligence 41, 8 (2018), 1947-1962.
  61. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018a. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition . 586-595.
  62. Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. 2021. LAFITE: Towards Language-Free Training for Text-to-Image Generation. arXiv preprint arXiv:2111.13792 (2021). · arXiv
  63. Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. 2020. In-domain GAN inversion for real image editing. In Proc. ECCV . Springer, 592-608.

Authors
Avrahami, O., Fried, O., & Lischinski, D.
Affiliations
  • Hebrew University of Jerusalem (IL)
  • Reichman University (IL)
ORCID
Venue
ACM Transactions on Graphics
Publisher
Association for Computing Machinery (ACM)
Published
2023-07-26
Volume
42
Issue
4
Pages
1-11
ISSN
  • 0730-0301 (ISSN-L)
  • 1557-7368
DOI
10.1145/3592450
OpenAlex
W4385271281
Cited by
345
References
19
Citations by year
2026: 74 · 2025: 120 · 2024: 120 · 2023: 31
FWCI
31.36
Type
journal article
Language
en
Open access
Licence
Locations
  • ACM Transactions on Graphics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Israel Science Foundation (1574/21, 2492/20)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
latent diffusion models, text-to-image generation, inference acceleration, optimization-based reconstruction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review