Real-time style transfer with efficient vision transformers
Abstract
Style Transfer aims at transferring the artistic style from a reference image to a content image. While Deep Learning (DL) has achieved state-of-the-art Style Transfer performance using Convolutional Neural Networks (CNN), its real-time application still requires powerful hardware such as GPU-accelerated systems. This paper leverages transformer-based models to accelerate real-time Style Transfer on mobile and embedded hardware platforms. We designed a Neural Architecture Search (NAS) algorithm dedicated to vision transformers to find the best set of architecture hyperparameters that maximizes the Style Transfer performance, expressed in Frame/seconds (FPS). Our approach has been evaluated and validated on the Xiaomi Redmi 7 mobile phone and the Raspberry Pi 3 platform. Experimental evaluation shows that our approach allows to achieve a 3.5x and 2.1x speedups compared to CNN-based Style Transfer models and Transformer-based models respectively1.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The searched network for Real-time Style Transfer is at least 2.1× faster than the baseline model for style transfer on Xiaomi Redmi 10 mobile and Raspberry Pi 3 embedded devices.
“The searched network is at least 2.1 × faster than the baseline model for style transfer on Xiaomi Redmi 10 mobile and Raspberry Pi 3 embedded devices.”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [200]
Cites
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283 · see citing works
Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization
Huang, X., & Belongie, S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.167 · Cited by 5,076 · see citing works
StyTr 2 : Image Style Transfer with Transformers
Deng, Y., Tang, F., Dong, W., Ma, C., Pan, X., Wang, L., & Xu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01104 · Cited by 442 · see citing works
Algorithms for Learning Kernels Based on Centered AlignmentPreprint
Cortes, C., Mohri, M., & Rostamizadeh, A. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1203.0550 · Cited by 392 · see citing works
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
Wang, H., Wu, Z., Liu, Z., Cai, H., Zhu, L., Gan, C., & Han, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.686 · Cited by 227 · see citing works
Attention-Aware Multi-Stroke Style Transfer
Yao, Y., Ren, J., Xie, X., Liu, W., Liu, Y., & Wang, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00156 · Cited by 197 · see citing works
Hardware-Aware Neural Architecture Search: Survey and Taxonomy
Benmeziane, H., Maghraoui, K. E., Ouarnoughi, H., Niar, S., Wistuba, M., & Wang, N. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/592 · Cited by 137 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
Neural Architecture Search for Transformers: A Survey
Chitty-Venkata, K. T., Emani, M., Vishwanath, V., & Somani, A. K. (2022) · IEEE Access · doi:10.1109/access.2022.3212767 · Cited by 118 · see citing works
Conditional image hiding network based on style transfer
Zhang, F., Feng, B., Xia, Z., Weng, J., Lu, W., & Chen, B. (2024) · Information Sciences · doi:10.1016/j.ins.2024.120225 · Cited by 9 · see citing works
Efficient Hardware-aware Neural Architecture Search for Edge Computing
Benmeziane, H. (2023) · theses.fr (ABES) · doi:10.70675/12a3f1afza163z4f13zb3cfz9a3425c3d45d · Cited by 1 · see citing works
Self-citationAdvances in Multi-Style and Real-Time Transfer
Kini, N., Kamath, V., Kelkar, T., Kanikar, P., Nanade, A., & Kolhe, A. (2024) · International Conference on Software, Systems and Information Technology (SSITCON) · doi:10.1109/ssitcon62437.2024.10796439
- Authors
- Benmeziane, H., Ouarnoughi, H., Maghraoui, K. E., & Niar, S.
- Affiliations
- Université Polytechnique Hauts-de-France (FR)
- IBM Research - Thomas J. Watson Research Center (US)
- Univ. Polytechnique Hauts-de-France, Valenciennes, France
- Venue
- Proceedings of the 5th International Workshop on Edge Systems, Analytics and Networking
- Publisher
- ACM
- Published
- 2022-04-05
- Pages
- 31-36
- OpenAlex
- W4220870460
- Cited by
- 5
- References
- 7
- Citations by year
- 2024: 2 · 2023: 2 · 2022: 1
- FWCI
- 0.45
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Workshop on Edge Systems, Analytics and Networking (EdgeSys) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- style transfer, real-time inference, vision transformer, neural architecture search, mobile devices, embedded hardware, inference acceleration, frame rate
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
