Demystifying Neural Style Transfer
Open accessThe DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Neural Style Transfer has recently demonstrated very exciting results which catches eyes in both academia and industry. Despite the amazing results, the principle of neural style transfer, especially why the Gram matrices could represent style remains unclear. In this paper, we propose a novel interpretation of neural style transfer by treating it as a domain adaptation problem. Specifically, we theoretically show that matching the Gram matrices of feature maps is equivalent to minimize the Maximum Mean Discrepancy (MMD) with the second order polynomial kernel. Thus, we argue that the essence of neural style transfer is to match the feature distributions between the style images and the generated images. To further support our standpoint, we experiment with several other distribution alignment methods, and achieve appealing results. We believe this novel interpretation connects these two important research fields, and could enlighten future researches.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387 · cited 1×
“Domain Adaptation Domain adaptation belongs to the area of transfer learning [19].”
BackgroundPhoto-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
Image Style Transfer Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.265 · Cited by 6,025
Learning Transferable Features with Deep Adaptation Networks
Long, M., Cao, Y., Wang, J., & Jordan, M. I. (2015) · arXiv · doi:10.48550/arxiv.1502.02791 · Cited by 4,505 · cited 1×
“It is a popular choice in domain adaptation works [26, 17, 18].”
Background(2012) · MPG.PuRe (Max Planck Society) · doi:10.5555/2188385.2188410 · Cited by 3,603
Texture synthesis by non-parametric sampling
Efros, A. A., & Leung, T. K. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790383 · Cited by 3,023
Image quilting for texture synthesis and transfer
Efros, A. A., & Freeman, W. T. (2001) · Proceedings of the 28th annual conference on Computer graphics and interactive techniques · doi:10.1145/383259.383296 · Cited by 2,347
Deep domain confusion Maximizing for domain invariance
Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., & Darrell, T. (2014) · arXiv · doi:10.48550/arxiv.1412.3474 · Cited by 2,285
Cited by
A Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
“) reliably encode the style of an image [19, 36] while spatially varying features encode a specific instance.”
BackgroundImage Style Transfer Using Convolutional Neural Networks
Gatys, L. A., Ecker, A. S., & Bethge, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.265 · Cited by 6,025
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization
Huang, X., & Belongie, S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.167 · Cited by 5,076
Deep visual domain adaptation: A survey
Wang, M., & Deng, W. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.083 · Cited by 2,209
Deng, W., Zheng, L., Ye, Q., Kang, G., Yang, Y., & Jiao, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00110 · Cited by 1,011
Image Quality Assessment: Unifying Structure and Texture Similarity
Ding, K., Ma, K., Wang, S., & Simoncelli, E. P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3045810 · Cited by 952
Neural Style Transfer: A Review
Jing, Y., Yang, Y., Feng, Z., Ye, J., Yu, Y., & Song, M. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2921336 · Cited by 856
Review the state-of-the-art technologies of semantic segmentation based on deep learning
Mo, Y., Wu, Y., Yang, X., Liu, F., & Liao, Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.005 · Cited by 632
Arbitrary Style Transfer With Style-Attentional Networks
Park, D. Y., & Lee, K. H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00603 · Cited by 466
Style Transfer in Text: Exploration and Evaluation
Fu, Z., Tan, X., Peng, N., Zhao, D., & Yan, R. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11330 · Cited by 460
StyTr2: Image Style Transfer with Transformers
Deng, Y., Tang, F., Dong, W., Ma, C., Pan, X., Wang, L., & Xu, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01104 · Cited by 402
Convolutional Networks with Adaptive Inference Graphs
Veit, A., & Belongie, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_1 · Cited by 364
SRM: A Style-Based Recalibration Module for Convolutional Neural Networks
Lee, H., Kim, H.-E., & Nam, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00194 · Cited by 351
Avatar-Net: Multi-scale Zero-Shot Style Transfer by Feature Decoration
Sheng, L., Lin, Z., Shao, J., & Wang, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00860 · Cited by 313
Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
Huang, Z., & Wang, N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.01219 · Cited by 300
Self-citationLearning Linear Transformations for Fast Image and Video Style Transfer
Li, X., Liu, S., Kautz, J., & Yang, M.-H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00393 · Cited by 295
Domain Generalization Using a Mixture of Multiple Latent Domains
Matsuura, T., & Harada, T. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6846 · Cited by 289
Atapour-Abarghouei, A., & Breckon, T. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00296 · Cited by 276
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts
Liang, J., He, R., & Tan, T. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-024-02181-w · Cited by 271
Editing in Style: Uncovering the Local Semantics of GANs
Collins, E., Bala, R., Price, B., & Susstrunk, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00581 · Cited by 268
ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows
An, J., Huang, S., Song, Y., Dou, D., Liu, W., & Luo, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00092 · Cited by 262
Demystifying Brain Tumor Segmentation Networks: Interpretability and Uncertainty Analysis
Natekar, P., Kori, A., & Krishnamurthi, G. (2020) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2020.00006 · Cited by 103
“3 Style Regularizer In order to obtain visualizations which are similar in style to the set of possible input images, we implement a style regularizer inspired from the work of (Li et al., 2017).”
Multimodal Style Transfer via Graph Cuts
Zhang, Y., Fang, C., Wang, Y., Wang, Z., Lin, Z., Fu, Y., & Yang, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00604 · Cited by 98
Metadata
- Authors
- Li, Y., Wang, N., Liu, J., & Hou, X.
- Venue
- Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
- Published
- 2017-08
- Pages
- 2230-2236
- OpenAlex
- W2583638424
- Cited by
- 514
- References
- 26
- FWCI
- 13.27
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.