Anycost GANs for Interactive Image Synthesis and Editing
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Generative adversarial networks (GANs) have enabled photorealistic image synthesis and editing. However, due to the high computational cost of large-scale generators (e.g., StyleGAN2), it usually takes seconds to see the results of a single edit on edge devices, prohibiting interactive user experience. In this paper, inspired by quick preview features in modern rendering software, we propose Anycost GAN for interactive natural image editing. We train the Anycost GAN to support elastic resolutions and channels for faster image generation at versatile speeds. Running subsets of the full generator produce outputs that are perceptually similar to the full generator, making them a good proxy for quick preview. By using sampling-based multi-resolution training, adaptive-channel training, and a generator-conditioned discriminator, the anycost generator can be evaluated at various configurations while achieving better image quality compared to separately trained models. Furthermore, we develop new encoder training and latent code optimization techniques to encourage consistency between the different sub-generators during image projection. Anycost GAN can be executed at various cost budgets (up to 10× computation reduction) and adapt to a wide range of hardware and la tency requirements. When deployed on desktop CPUs and edge devices, our model can provide perceptually similar previews at 6-12× speedup, enabling interactive image editing. The ${\color{RubineRed}{code}}$ and ${\color{RubineRed}{demo}}$ are publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Self-citationA Style-Based Generator Architecture for Generative Adversarial Networks
Karras, T., Laine, S., & Aila, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00453 · Cited by 11,714
Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L.-C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., Pang, R., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00140 · Cited by 9,708
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00813 · Cited by 6,332
Analyzing and Improving the Image Quality of StyleGAN
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2019) · arXiv · doi:10.48550/arxiv.1912.04958 · Cited by 6,332
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · cited 1×
“Commonly used approaches include pruning [28, 49, 55, 23, 22, 70], quantization [22, 77], knowledge distillation [30, 56, 10], efficient architecture design [12, 36, 34, 64], and autoML-based methods [27, 33, 68, 81, 52, 9, 20, 48].”
MethodSelf-citationContext Encoders: Feature Learning by Inpainting
Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.278 · Cited by 5,691
High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs
Wang, T.-C., Liu, M.-Y., Zhu, J.-Y., Tao, A., Kautz, J., & Catanzaro, B. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00917 · Cited by 5,360
Self-citationStarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., & Choo, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00916 · Cited by 4,163
Neural Architecture Search with Reinforcement Learning
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717
Self-Attention Generative Adversarial Networks
Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2018) · arXiv · doi:10.48550/arxiv.1805.08318 · Cited by 3,353 · cited 1×
“As image quality and model capacity have increased [17, 61, 74, 41, 73, 43, 40], so have computational costs and inference time.”
BackgroundDARTS: Differentiable architecture search
Liu, H., Simonyan, K., & Yang, Y. (2018) · arXiv · doi:10.48550/arxiv.1806.09055 · Cited by 3,248
StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks
Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.629 · Cited by 3,021 · cited 1×
“As image quality and model capacity have increased [17, 61, 74, 41, 73, 43, 40], so have computational costs and inference time.”
BackgroundSemantic Image Synthesis With Spatially-Adaptive Normalization
Park, T., Liu, M.-Y., Wang, T.-C., & Zhu, J.-Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00244 · Cited by 2,974 · cited 1×
“Some recent examples include high-resolution face synthesis [42, 43], ImageNet-scale class-conditional generation [6], and semantic photo synthesis [69, 58].”
BackgroundSelf-citationLearning Efficient Convolutional Networks through Network Slimming
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.298 · Cited by 2,761
CyCADA: Cycle-Consistent Adversarial Domain Adaptation
Hoffman, J., Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A. A., & Darrell, T. (2017) · arXiv · doi:10.48550/arxiv.1711.03213 · Cited by 2,656 · cited 1×
“GANs [17] have enabled photorealistic synthesis for many vision and graphics applications [45, 11, 37, 80, 59, 31, 15].”
BackgroundSelf-citationChannel Pruning for Accelerating Very Deep Neural Networks
He, Y., Zhang, X., & Sun, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.155 · Cited by 2,647 · cited 3×
“Figure 5: Anycost GAN outperforms existing compression baselines [3, 28] at various computation budgets, despite only training a single, flexible generator across computation budgets.”
BackgroundMethodDiverse Image-to-Image Translation via Disentangled Representations
Lee, H.-Y., Tseng, H.-Y., Huang, J.-B., Singh, M., & Yang, M.-H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_3 · Cited by 1,541
AMC: AutoML for Model Compression and Acceleration on Mobile Devices
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J., & Han, S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_48 · Cited by 1,400
Self-citationInterpreting the Latent Space of GANs for Semantic Face Editing
Shen, Y., Gu, J., Tang, X., & Zhou, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00926 · Cited by 1,063
Once-for-All: Train One Network and Specialize it for Efficient Deployment
Cai, H., Gan, C., Wang, T., Zhang, Z., & Han, S. (2019) · arXiv · doi:10.48550/arxiv.1908.09791 · Cited by 845 · cited 1×
“We use an evolutionary search algorithm [62, 8, 20] to find an effective sub-generator architecture under diverse resource budgets (e.”
MethodSelf-citationNeural Photo Editing with Introspective Adversarial Networks
Brock, A., Lim, T., Ritchie, J. M., & Weston, N. (2016) · arXiv · doi:10.48550/arxiv.1609.07093 · Cited by 275
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
Cai, H., Zhu, L., & Han, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00332 · Cited by 267
Self-citationGenerative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
Liu, M.-Y., Huang, X., Yu, J., Wang, T.-C., & Mallya, A. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3049196 · Cited by 236 · cited 13×
“To achieve this, one can project the image into the image manifold of GANs by finding a latent code that reconstructs the image, and then modify the code to produce final outputs [79].”
MethodInfluentialGAN Compression: Efficient Architectures for Interactive Conditional GANs
Li, M., Lin, J., Ding, Y., Liu, Z., Zhu, J.-Y., & Han, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00533 · Cited by 216
Self-citationSemantic photo manipulation with a generative image prior
Bau, D., Strobelt, H., Peebles, W., Wulff, J., Zhou, B., Zhu, J.-Y., & Torralba, A. (2019) · ACM Transactions on Graphics · doi:10.1145/3306346.3323023 · Cited by 207 · cited 1×
“Several methods have been proposed, including choosing better or multiple layers to project and edit [1, 2, 19], fine-tuning network weights for each image [5], modeling image corruption and transformations [4, 35], and discovering meaningful latent directions [65, 16, 38, 24].”
MethodSelf-citation
Cited by
Bond-Taylor, S., Leach, A., Long, Y., & Willcocks, C. G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3116668 · Cited by 703
Scaling up GANs for Text-to-Image Synthesis
Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., & Park, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00976 · Cited by 405
Self-citationHyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing
Alaluf, Y., Tov, O., Mokady, R., Gal, R., & Bermano, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01796 · Cited by 259
GAN Compression: Efficient Architectures for Interactive Conditional GANs
Li, M., Lin, J., Ding, Y., Liu, Z., Zhu, J.-Y., & Han, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00533 · Cited by 216
Self-citationDeepfakes generation and detection: a short survey
Akhtar, Z. (2023) · Journal of Imaging · doi:10.3390/jimaging9010018 · Cited by 106
“…various approaches have been devised, e.g., coupled GANs [117], invertible convolution [118], U-Net [119], from speech to face GANs [120], high-resolution deep convolutional GANs [121], interactive anycost GANs [122], and structured disentanglement framework for face generation and editing [123].”
Metadata
- Authors
- Lin, J., Zhang, R., Ganz, F., Han, S., & Zhu, J.-Y.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 14981-14991
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3173562028
- Cited by
- 81
- References
- 32
- FWCI
- 5.63
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.