The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Inversion-based Style Transfer with Diffusion Models
Open accessAbstract
The artistic style within a painting is the means of expression, which includes not only the painting material, colors, and brushstrokes, but also the high-level attributes, including semantic elements and object shapes. Previous arbitrary example-guided artistic image generation methods often fail to control shape changes or convey elements. Pre-trained text-to-image synthesis diffusion probabilistic models have achieved remarkable quality but often require extensive textual descriptions to accurately portray the attributes of a particular painting. The uniqueness of an artwork lies in the fact that it cannot be adequately explained with normal language. Our key idea is to learn the artistic style directly from a single painting and then guide the synthesis without providing complex textual descriptions. Specifically, we perceive style as a learnable textual description of a painting. We propose an inversion-based style transfer method (InST), which can efficiently and accurately learn the key information of an image, thus capturing and transferring the artistic style of a painting. We demonstrate the quality and efficiency of our method on numerous paintings of various artists and styles. Codes are available at https://github.com/zyxElsa/InST.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Denoising Diffusion Probabilistic Models
Ho, J., Jain, A., & Abbeel, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11239 · Cited by 8,712
Image Style Transfer Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.265 · Cited by 6,025
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization
Huang, X., & Belongie, S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.167 · Cited by 5,076
Diffusion Models Beat GANs on Image Synthesis
Dhariwal, P., & Nichol, A. (2021) · arXiv · doi:10.48550/arxiv.2105.05233 · Cited by 3,362
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528
Taming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2020) · arXiv · doi:10.48550/arxiv.2012.09841 · Cited by 2,514
Taming Transformers for High-Resolution Image Synthesis
Esser, P., Rombach, R., & Ommer, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01268 · Cited by 2,514
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 2×
“Ramesh [40] develop a textconditional image generator based on the diffusion models and the inverted CLIP.”
BackgroundMethodDenoising Diffusion Implicit Models
Song, J., Meng, C., & Ermon, S. (2020) · arXiv · doi:10.48550/arxiv.2010.02502 · Cited by 2,507
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2022) · arXiv · doi:10.48550/arxiv.2208.12242 · Cited by 2,442
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., & Aberman, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02155 · Cited by 2,295
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2021) · arXiv · doi:10.48550/arxiv.2112.10741 · Cited by 1,357
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., & Lischinski, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00209 · Cited by 1,023
Imagic: Text-Based Real Image Editing with Diffusion Models
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., & Irani, M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00582 · Cited by 767
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
Choi, J., Kim, S., Jeong, Y., Gwon, Y., & Yoon, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01410 · Cited by 634
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., & Cohen-Or, D. (2022) · arXiv · doi:10.48550/arxiv.2208.01618 · Cited by 611 · cited 10×
“Ours TexIn [15] SDM [42] MLP w/o drop Accuracy(%)↑ 78.”
BackgroundMethodInfluentialStyleGAN-NADA: CLIP-guided domain adaptation of image generators
Gal, R., Patashnik, O., Maron, H., Bermano, A. H., Chechik, G., & Cohen-Or, D. (2022) · ACM Transactions on Graphics · doi:10.1145/3528223.3530164 · Cited by 500
Visual attribute transfer through deep image analogy
Liao, J., Yao, Y., Yuan, L., Hua, G., & Kang, S. B. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073683 · Cited by 487
Controlling Perceptual Factors in Neural Style Transfer
Gatys, L. A., Ecker, A. S., Bethge, M., Hertzmann, A., & Shechtman, E. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.397 · Cited by 486
Arbitrary Style Transfer With Style-Attentional Networks
Park, D. Y., & Lee, K. H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00603 · Cited by 466
AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer
Liu, S., Lin, T., He, D., Li, F., Wang, M., Li, X., Sun, Z., Li, Q., & Ding, E. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00658 · Cited by 420
StyTr2: Image Style Transfer with Transformers
Deng, Y., Tang, F., Dong, W., Ma, C., Pan, X., Wang, L., & Xu, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01104 · Cited by 402
Self-citationScaling Autoregressive Models for Content-Rich Text-to-Image Generation
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.10789 · Cited by 371
Prompt-to-Prompt Image Editing with Cross Attention Control
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., & Cohen‐Or, D. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.01626 · Cited by 360
Universal Style Transfer via Feature Transforms
Li, Y., Chen, F., Yang, J., Wang, Z., Lü, X., & Yang, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08086 · Cited by 346
CLIPDraw: Exploring Text-To-Drawing Synthesis Through Language-Image Encoders
Frans, K., Soros, L. B., & Witkowski, O. (2022) · neural information processing systems · doi:10.52202/068431-0376 · Cited by 325
Learning Linear Transformations for Fast Image and Video Style Transfer
Li, X., Liu, S., Kautz, J., & Yang, M.-H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00393 · Cited by 295
Style Transfer by Relaxed Optimal Transport and Self-Similarity
Kolkin, N., Salavon, J., & Shakhnarovich, G. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01029 · Cited by 281
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance
Crowson, K., Biderman, S., Kornis, D., Stander, D., Hallahan, E., Castricato, L., & Raff, E. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19836-6_6 · Cited by 279
ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows
An, J., Huang, S., Song, Y., Dou, D., Liu, W., & Luo, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00092 · Cited by 262
CLIPstyler: Image Style Transfer with a Single Text Condition
Kwon, G., & Ye, J. C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01753 · Cited by 251
Exact Feature Distribution Matching for Arbitrary Style Transfer and Domain Generalization
Zhang, Y., Li, M., Li, R., Jia, K., & Zhang, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00787 · Cited by 250
Self-citationDomain Enhanced Arbitrary Image Style Transfer via Contrastive Learning
Zhang, Y., Tang, F., Dong, W., Huang, H., Ma, C., Lee, T.-Y., & Xu, C. (2022) · Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings · doi:10.1145/3528233.3530736 · Cited by 213
Self-citationArbitrary Style Transfer via Multi-Adaptation Network
Deng, Y., Tang, F., Dong, W., Sun, W., Huang, F., & Xu, C. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3414015 · Cited by 197
Self-citation(2021) · doi:10.52202/068431-1229 · Cited by 166
Arbitrary Video Style Transfer via Multi-Channel Correlation
Deng, Y., Tang, F., Dong, W., Huang, H., Ma, C., & Xu, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16208 · Cited by 159
Self-citationStyleFormer: Real-time Arbitrary Style Transfer via Parametric Style Composition
Xiao-lei, W., Hu, Z., Sheng, L., & Xu, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01435 · Cited by 134
DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization
Huang, N., Zhang, Y., Tang, F., Ma, C., Huang, H., Dong, W., & Xu, C. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3342645 · Cited by 112
Self-citationEfficient example-based painting and synthesis of 2D directional texture
Wang, B., Wang, W., Yang, H., & Sun, J. (2004) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2004.1272726 · Cited by 93
Two-Stage Peer-Regularized Feature Recombination for Arbitrary Image Style Transfer
Svoboda, J., Anoosheh, A., Osendorfer, C., & Masci, J. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01383 · Cited by 86
Language-Driven Artistic Style Transfer
Fu, T.-J., Wang, X. E., & Wang, W. Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_41 · Cited by 83
xCos: An Explainable Cosine Metric for Face Verification Task
Lin, Y., Liu, Z.-Y., Chen, Y., Wang, Y.-S., Chang, Y.-L., & Hsu, W. H. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3469288 · Cited by 78
StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation
Schaldenbrand, P., Liu, Z., & Oh, J. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/688 · Cited by 74
Draw Your Art Dream: Diverse Digital Art Synthesis with Multimodal Guided Diffusion
Huang, N., Tang, F., Dong, W., & Xu, C. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548282 · Cited by 68
Self-citationImage-Based CLIP-Guided Essence Transfer
Chefer, H., Benaim, S., Paiss, R., & Wolf, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19778-9_40 · Cited by 50
Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective
Kong, X., Deng, Y., Tang, F., Dong, W., Ma, C., Chen, Y., He, Z., & Xu, C. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3230084 · Cited by 42
Self-citationA Comparative Study of CNN- and Transformer-Based Visual Style Transfer
Wei, H., Deng, Y., Tang, F., Pan, X., & Dong, W. (2022) · Journal of Computer Science and Technology · doi:10.1007/s11390-022-2140-7 · Cited by 36
Self-citationStroke-GAN Painter: Learning to paint artworks using stroke-style generative adversarial networks
Wang, Q., Guo, C., Dai, H., et al. (2023) · Computational Visual Media · doi:10.1007/s41095-022-0287-3 · Cited by 29
Cited by
Diffusion-Based Visual Art Creation: A Survey and New Perspectives
Wang, B., Chen, Q., & Wang, Z. (2025) · ACM Computing Surveys · doi:10.1145/3728459 · Cited by 40
Metadata
- Authors
- Zhang, Y., Huang, N., Tang, F., Huang, H., Ma, C., Dong, W., & Xu, C.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 10146-10156
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386071613
- Cited by
- 353
- References
- 54
- FWCI
- 40.14
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.