The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Image Segmentation Using Text and Image Prompts
Open accessAbstract
Image segmentation is usually addressed by training a model for a fixed set of object classes. Incorporating additional classes or more complex queries later is expensive as it requires re-training the model on a dataset that encompasses these expressions. Here we propose a system that can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text or an image. This approach enables us to create a unified model (trained once) for three common segmentation tasks, which come with distinct challenges: referring expression segmentation, zero-shot segmentation and one-shot segmentation. We build upon the CLIP model as a backbone which we extend with a transformer-based decoder that enables dense prediction. After training on an extended version of the PhraseCut dataset, our system generates a binary segmentation map for an image based on a free-text prompt or on an additional image expressing the query. We analyze different variants of the latter image-based prompts in detail. This novel hybrid input allows for dynamic adaptation not only to the three segmentation tasks mentioned above, but to any binary segmentation task where a text or image query can be formulated. Finally, we find our system to adapt well to generalized queries involving affordances or properties. Code is available at https://eckerlab.org/code/CLIPSeg
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
The Ecological Approach to Visual Perception
Bornstein, M. H., & Gibson, J. J. (1980) · The Journal of Aesthetics and Art Criticism · doi:10.2307/429816 · Cited by 22,935
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 4×
“1): • In generalized zero-shot segmentation, seen as well as unseen categories needs to be segmented by putting un-seen categories in relation to seen ones, e.g. through word embeddings [1] or WordNet [2].”
MethodInfluentialKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
(2017) · Cited by 11,625
A simple framework for contrastive learning of visual representations
Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020) · arXiv · doi:10.48550/arxiv.2002.05709 · Cited by 10,857
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J. M., & Luo, P. (2021) · arXiv · doi:10.48550/arxiv.2105.15203 · Cited by 5,090
TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A. L., & Zhou, Y. (2021) · arXiv · doi:10.48550/arxiv.2102.04306 · Cited by 4,206 · cited 1×
“Analogously to image classification, there have been efforts to make use of transformers for segmentation: TransUNet [11] and SETR [12] employ a hybrid architecture which combine a visual transformer for encoding with a CNN-based decoder.”
MethodRethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 1×
“Analogously to image classification, there have been efforts to make use of transformers for segmentation: TransUNet [11] and SETR [12] employ a hybrid architecture which combine a visual transformer for encoding with a CNN-based decoder.”
MethodExploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01549 · Cited by 3,725
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304 · cited 1×
“Zhou et al. [45] propose context optimization (CoOp) which automatically learns tokens that perform well for given downstream tasks.”
BackgroundThe Senses Considered as Perceptual Systems
Boring, E. G., & Gibson, J. J. (1967) · The American Journal of Psychology · doi:10.2307/1420561 · Cited by 3,157
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Segmenter: Transformer for Semantic Segmentation
Strudel, R., Garcia, R., Laptev, I., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00717 · Cited by 2,051
PANet: Few-Shot Image Semantic Segmentation With Prototype Alignment
Wang, K., Liew, J. H., Zou, Y., Zhou, D., & Feng, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00929 · Cited by 1,500
LVIS: A Dataset for Large Vocabulary Instance Segmentation
Gupta, A., Dollar, P., & Girshick, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00550 · Cited by 1,274
Generation and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.9 · Cited by 1,174
(2016) · doi:10.1007/978-3-319-46475-6_5 · Cited by 1,081
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., & Lischinski, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00209 · Cited by 1,023
One-Shot Learning for Semantic Segmentation
Shaban, A., Bansal, S., Liu, Z., Essa, I., & Boots, B. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.167 · Cited by 821
MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., & Carion, N. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00180 · Cited by 729
Prior Guided Feature Enrichment Network for Few-Shot Segmentation
Tian, Z., Zhao, H., Shu, M., Yang, Z., Li, R., & Jia, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3013717 · Cited by 679
Zhang, C., Lin, G., Liu, F., Yao, R., & Shen, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00536 · Cited by 670
Learning multiple visual domains with residual adapters
Rebuffi, S.-A., Bilen, H., & Vedaldi, A. (2017) · arXiv · doi:10.48550/arxiv.1705.08045 · Cited by 605 · cited 2×
“This multi-modal input format goes beyond existing multi-task benchmarks such as Visual Decathlon [3] where input is always provided in form of images.”
BackgroundMethodCross-Modal Self-Attention Network for Referring Image Segmentation
Ye, L., Rochan, M., Liu, Z., & Wang, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01075 · Cited by 532
Segmentation from Natural Language Expressions
Hu, R., Rohrbach, M., & Darrell, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_7 · Cited by 438
Prototype Mixture Models for Few-Shot Semantic Segmentation
Yang, B., Liu, C., Li, B., Jiao, J., & Ye, Q. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58598-3_45 · Cited by 412
Part-Aware Prototype Network for Few-Shot Semantic Segmentation
Liu, Y., Zhang, X., Zhang, S., & He, X. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58545-7_9 · Cited by 353
Pyramid Graph Networks With Connection Attentions for Region-Based One-Shot Semantic Segmentation
Zhang, C., Lin, G., Liu, F., Guo, J., Wu, Q., & Yao, R. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00968 · Cited by 346
Language-driven Semantic Segmentation
Li, B., Weinberger, K. Q., Belongie, S., Koltun, V., & Ranftl, R. (2022) · arXiv · doi:10.48550/arxiv.2201.03546 · Cited by 316
Referring Image Segmentation via Recurrent Refinement Networks
Li, R., Li, K., Kuo, Y.-C., Shu, M., Qi, X., Shen, X., & Jia, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00602 · Cited by 275
Recurrent Multimodal Interaction for Referring Image Segmentation
Liu, C., Lin, Z., Shen, X., Yang, J., Lu, X., & Yuille, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.143 · Cited by 272
Semantic Projection Network for Zero- and Few-Label Semantic Segmentation
Xian, Y., Choudhury, S., He, Y., Schiele, B., & Akata, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00845 · Cited by 240
CLIPort: What and Where Pathways for Robotic Manipulation
Shridhar, M., Manuelli, L., & Fox, D. (2021) · arXiv · doi:10.48550/arxiv.2109.12098 · Cited by 223
Few-Shot Segmentation Without Meta-Learning: A Good Transductive Inference Is All You Need?
Boudiaf, M., Kervadec, H., Masud, Z. I., Piantanida, P., Ayed, I. B., & Dolz, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01376 · Cited by 188
PhraseCut: Language-Based Image Segmentation in the Wild
Wu, C., Lin, Z., Cohen, S. D., Bui, T., & Maji, S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01023 · Cited by 183
Key-Word-Aware Network for Referring Expression Image Segmentation
Shi, H., Li, H., Meng, F., & Wu, Q. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_3 · Cited by 178
Hypercorrelation Squeeze for Few-Shot SegmentationPreprint
Min, J., Kang, D., & Cho, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01538 · Cited by 175
Context-aware Feature Generation For Zero-shot Semantic Segmentation
Gu, Z., Zhou, S., Niu, L., Zhao, Z., & Zhang, L. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413593 · Cited by 165
Dumoulin, V., Perez, E., Schucher, N., Strub, F., Vries, H., Courville, A., & Bengio, Y. (2018) · Distill · doi:10.23915/distill.00011 · Cited by 160
Few-Shot Segmentation Propagation with Guided NetworksPreprint
Rakelly, K., Shelhamer, E., Darrell, T., Efros, A. A., & Levine, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07373 · Cited by 150
One-Shot Instance SegmentationPreprint
Michaelis, C., Ustyuzhaninov, I., Bethge, M., & Ecker, A. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.11507 · Cited by 102
Self-citationExploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation
Baek, D., Oh, Y., & Ham, B. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00940 · Cited by 101
Prototypical Matching and Open Set Rejection for Zero-Shot Semantic Segmentation
Zhang, H., & Ding, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00689 · Cited by 83
Uncertainty-Aware Learning for Zero-Shot Semantic Segmentation
Hu, P., Sclaroff, S., & Saenko, K. (2020) · neural information processing systems · Cited by 72
Consistent Structural Relation Learning for Zero-Shot Segmentation.
Li, P. P., Wei, Y., & Yang, Y. (2020) · UTS ePRESS (University of Technology Sydney) · Cited by 51
Zero-Shot Detection via Vision and Language Knowledge Distillation.Preprint
Gu, X., Lin, T.-Y., Kuo, W., & Cui, Y. (2021) · arXiv (Cornell University) · Cited by 51
Zero-Shot Open Set Detection by Extending CLIPPreprint
Esmaeilpour, S., Liu, B., Robertson, E., & Shu, L. (2021) · arXiv (Cornell University) · Cited by 6
Cited by
Vision-Language Models for Vision Tasks: A Survey
Zhang, J., Huang, J., Jin, S., & Lu, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3369699 · Cited by 1,051
MaPLe: Multi-modal Prompt Learning
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., & Khan, F. S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01832 · Cited by 925
LERF: Language Embedded Radiance Fields
Kerr, J., Kim, C. M., Goldberg, K., Kanazawa, A., & Tancik, M. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01807 · Cited by 370
CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection
Liu, J., Zhang, Y., Chen, J.-N., Xiao, J., Lu, Y., Landman, B. A., Yuan, Y., Yuille, A., Tang, Y., & Zhou, Z. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01934 · Cited by 333
A Survey on Multimodal Large Language Models for Autonomous Driving
Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.-D., et al. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw60836.2024.00106 · Cited by 369
Metadata
- Authors
- Luddecke, T., & Ecker, A.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 7076-7086
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4312420092
- Cited by
- 562
- References
- 58
- FWCI
- 41.91
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.