CLIPasso: semantically-aware object sketching
Open accessAbstract
Abstraction is at the heart of sketching due to the simple and minimal nature of line drawings. Abstraction entails identifying the essential visual properties of an object or scene, which requires semantic understanding and prior knowledge of high-level concepts. Abstract depictions are therefore challenging for artists, and even more so for machines. We present CLIPasso, an object sketching method that can achieve different levels of abstraction, guided by geometric and semantic simplifications. While sketch generation methods often rely on explicit sketch datasets for training, we utilize the remarkable ability of CLIP (Contrastive-Language-Image-Pretraining) to distill semantic concepts from sketches and images alike. We define a sketch as a set of Bézier curves and use a differentiable rasterizer to optimize the parameters of the curves directly with respect to a CLIP-based perceptual loss. The abstraction degree is controlled by varying the number of strokes. The generated sketches demonstrate multiple levels of abstraction while maintaining recognizability, underlying structure, and essential visual components of the subject drawn.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Human behaviours, emotions, events, actions, and humour are multimodal, so various human-centred multimodal learning tasks are widely studied.
“Furthermore, human behaviours, emotions, events, actions, and humour are multimodal, thus various human-centred MML tasks are widely studied, including multimodal emotion recognition [67], multimodal event representation [68], understanding multimodal humor [69], face-body-voice based video person-clustering [70], etc.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [69]
Cites
A Computational Approach to Edge Detection
Canny, J. (1986) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1986.4767851 · Cited by 30,004 · see citing works
Image-to-Image Translation with Conditional Adversarial Networks
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.632 · Cited by 22,930 · see citing works
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623 · see citing works
Contour Detection and Hierarchical Image Segmentation
Arbeláez, P., Maire, M., Fowlkes, C. C., & Malik, J. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.161 · Cited by 5,472 · see citing works
U2-Net: Going deeper with nested U-structure for salient object detection
Qin, X., Zhang, Z., Huang, C., Dehghan, M., Zai͏̈ane, O. R., & Jägersand, M. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107404 · Cited by 2,452 · see citing works
A lifespan database of adult facial stimuli
Minear, M. E., & Park, D. C. (2004) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03206543 · Cited by 1,227 · see citing works
Sangkloy, P., Burnell, N., Ham, C., & Hays, J. H. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925954 · Cited by 623 · see citing works
XDoG: An eXtended difference-of-Gaussians compendium including advanced image stylization
Winnemöller, H., Kyprianidis, J. E., & Olsen, S. C. (2012) · Computers & Graphics · doi:10.1016/j.cag.2012.03.004 · Cited by 248 · see citing works
A survey of stroke-based rendering
Hertzmann, A. (2003) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2003.1210867 · Cited by 242 · see citing works
Multimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., Carter, S., Petrov, M., Schubert, L., Radford, A., & Olah, C. (2021) · Distill · doi:10.23915/distill.00030 · Cited by 240 · see citing works
Differentiable vector graphics rasterization for editing and learning
Li, T., Lukáč, M., Gharbi, M., et al. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417871 · Cited by 182 · see citing works
SketchyCOCO: Image Generation From Freehand Scene Sketches
Gao, C., Liu, Q., Xu, Q., Wang, L., Liu, J., & Zou, C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00522 · Cited by 147 · see citing works
Example-based facial sketch generation with non-parametric sampling
Chen, H., Xu, Y., Shum, H., et al. (2002) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2001.937657 · Cited by 133 · see citing works
Style and abstraction in portrait sketching
Berger, I., Shamir, A., Mahler, M., Carter, E. J., & Hodgins, J. K. (2013) · ACM Transactions on Graphics · doi:10.1145/2461912.2461964 · Cited by 125 · see citing works
Self-citationLearning to Sketch with Shortcut Cycle Consistency
Song, J., Pang, K., Song, Y.-Z., Xiang, T., & Hospedales, T. M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00090 · Cited by 111 · see citing works
Gryaditskaya, Y., Sypesteyn, M., Hoftijzer, J. W., et al. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356533 · Cited by 96 · see citing works
The genesis of errors in drawing
Chamberlain, R., & Wagemans, J. P. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.04.002 · Cited by 44 · see citing works
Bhunia, A. K., Das, A., Muhammad, U. R., et al. (2020) · ACM Transactions on Graphics · doi:10.1145/3414685.3417840 · Cited by 44 · see citing works
Bénard, P., & Hertzmann, A. (2019) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000075 · Cited by 41 · see citing works
Pragmatic Inference and Visual Abstraction Enable Contextual Flexibility During Visual Communication
Fan, J. E., Hawkins, R. D., Wu, M., & Goodman, N. D. (2019) · Computational Brain & Behavior · doi:10.1007/s42113-019-00058-7 · Cited by 33 · see citing works
The Role of Edges in Line Drawing Perception
Hertzmann, A. (2021) · Perception · doi:10.1177/0301006621994407 · Cited by 15 · see citing works
Emergent Graphical Conventions in a Visual Communication GamePreprint
Qiu, S., Xie, S., Fan, L., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.14210 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
Wang, Y., Shen, S., & Lim, B. Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581402 · Cited by 120 · see citing works
- Authors
- Vinker, Y., Pajouheshgar, E., Bo, J. Y., Bachmann, R. C., Bermano, A. H., Cohen-Or, D., Zamir, A., & Shamir, A.
- Affiliations
- Tel Aviv University (IL)
- École Polytechnique Fédérale de Lausanne (CH)
- Tel Aviv University, Israel
- Reichman University (IL)
- ORCID
- Venue
- ACM Transactions on Graphics
- Published
- 2022-07
- Volume
- 41
- Issue
- 4
- Pages
- 1-11
- ISSN
- 0730-0301 (ISSN-L)
- 1557-7368
- OpenAlex
- W4286611322
- Cited by
- 217
- References
- 24
- Citations by year
- 2026: 49 · 2025: 71 · 2024: 68 · 2023: 26 · 2022: 3
- FWCI
- 13.75
- Type
- journal article
- Language
- en
- Open access
- green
- infoscience.epfl.ch
- Licence
- Locations
- ACM Transactions on Graphics (publishedVersion)
- Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
- Funders
- Deutsch Foundation and the Yandex Initiative in Machine Learning
- Israel Science Foundation (2492/20, 3441/21)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Computer Graphics and Visualization Techniques
- Keywords
- sketch generation, abstraction, CLIP, Bézier curves, differentiable rasterization, perceptual loss, geometric simplification, contrastive language-image pretraining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
