Foundation Models Defining a New Era in Vision: A Survey and Outlook
The DOI matches its Crossref record; 227 of the 227 references it cites were found in a registry or an index.
Abstract
Vision systems that see and reason about the compositional nature of visual scenes are fundamental to understanding our world. The complex relations between objects and their locations, ambiguities, and variations in the real-world environment can be better described in human language, naturally governed by grammatical rules and other modalities such as audio and depth. The models learned to bridge the gap between such modalities and large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time. These models are referred to as foundation models. The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions. In this survey, we provide a comprehensive review of such emerging foundation models, including typical architecture designs to combine different modalities (vision, text, audio, etc.), training objectives (contrastive, generative), pre-training datasets, fine-tuning mechanisms, and the common prompting patterns; textual, visual, and heterogeneous. We discuss the open challenges and research directions for foundation models in computer vision, including difficulties in their evaluations and benchmarking, gaps in their real-world understanding, limitations of contextual understanding, biases, vulnerability to adversarial attacks, and interpretability issues. We review recent developments in this field, covering a wide range of applications of foundation models systematically and comprehensively.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Vision Foundation Models (VFMs)-large-scale models pre-trained on massive collections of unlabeled images to learn robust and transferable visual representations [5,6]”
From Trustworthy Celestial Eye: Calibrated and Robust Planetary Classification via Self-Supervised Vision Transformers (2026) · cites this paper as [5]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“This model is based on transformer-based architecture [89], pre-trained to predict the next word token using public and private datasets.”
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“Image and text are masked randomly following Devlin et al. [86] and He et al. [65], and the model is trained on joint conditional reconstruction task as well as Image-text Contrastive (ITC) [42], [43] and Image Text Matching (TIM) [49] task.”
InfluentialMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Prompt Engineering: Prompt engineering has primarily been used with Large Language Models (LLMs) to make them do certain tasks [3], [53].”
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 10×
“To achieve better generalization, this model is trained on multiple uni and multi-modal losses, including a global contrastive loss similar to CLIP [42] for cross-modal alignment, masked multi-modal masking and image-text matching, masked-image modeling, masked-languagemodeling, etc.…”
InfluentialMomentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953 · cited 8×
“Adapting SAM for other Applications: A visual foundation model like SAM [11] is trained on large-scale datasets that contain more than one billion masks and 11 million images.”
InfluentialExploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 2×
“MetaLM shows the casual language models’ excellent zero and few-shot capabilities and better transferability of non-casual encoders [86], [138].”
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 1×
“TheproposedVision-Language ( VL-x ) employs a pre-trained encoder-decoder language model, such as BART or T5 [138], [145].”
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · cited 2×
“Image and text are masked randomly following Devlin et al. [86] and He et al. [65], and the model is trained on joint conditional reconstruction task as well as Image-text Contrastive (ITC) [42], [43] and Image Text Matching (TIM) [49] task.”
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · cited 1×
“These attacks involve carefully crafted inputs that can cause the model to generate incorrect or harmful outputs [201], [202], [203].”
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602 · cited 1×
“These attacks involve carefully crafted inputs that can cause the model to generate incorrect or harmful outputs [201], [202], [203].”
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 1×
“…[86], Chen et al. [49] propose Universal Image-TExt Representation ( UNITER ), a method to leverage conventional image-text datasets (COCO [142], Visual Genome [143], Conceptual Captions [54], SBU Captions [144]) to train foundation models that can be used for heterogeneous vision-language tasks.”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · cited 2×
“Second, a large multi-modal framework based on a large pre-trained language model (LLaMA [114]) and CLIP vision encoder (ViT) is presented.”
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Tarvainen, A., & Valpola, H. (2017) · arXiv · doi:10.48550/arxiv.1703.01780 · Cited by 4,275
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595
Self-citationTransformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2021) · arXiv · doi:10.48550/arxiv.2101.01169 · Cited by 3,594
Self-citationTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1×
“Using contextual learning, a model can be quickly adapted to various tasks with only a few prompts and examples in NLP [169].”
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 1×
“Restrictions apply. approaches include Low-rank Adaptation (LoRA) [194] and its variants such as QLoRA [195] and GLoRA [196], Prefix tuning [197], [198], Adapters [164], [166], Prompt tuning [199], [200].”
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092 · cited 1×
“Restrictions apply. approaches include Low-rank Adaptation (LoRA) [194] and its variants such as QLoRA [195] and GLoRA [196], Prefix tuning [197], [198], Adapters [164], [166], Prompt tuning [199], [200].”
Segment anything in medical images
Ma, J., He, Y., Li, F., Han, L., You, C., & Wang, B. (2024) · Nature Communications · doi:10.1038/s41467-024-44824-z · Cited by 2,913
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · cited 2×
“Foundational Models and Scope of the Survey: The term “foundational model” was introduced by [1] at Stanford Institute for Human-Centered AI.”
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
YOLACT: Real-Time Instance Segmentation
Bolya, D., Zhou, C., Xiao, F., & Lee, Y. J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00925 · Cited by 2,536
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 1×
“Restrictions apply. approaches include Low-rank Adaptation (LoRA) [194] and its variants such as QLoRA [195] and GLoRA [196], Prefix tuning [197], [198], Adapters [164], [166], Prompt tuning [199], [200].”
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1516 · Cited by 2,428
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · cited 3×
“Unification of tasks: Inspired by the generalizability of BERT [69] in Natural Langauge Processing (NLP) tasks [86], Chen et al. [49] propose Universal Image-TExt Representation ( UNITER ), a method to leverage conventional image-text datasets (COCO [142], Visual Genome [143], Conceptual Captions…”
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · cited 3×
“Zhao et al. [2] review recent advances in LLMs, distinguishing different aspects of LLMs such as pre-training, adaptation tuning, LLM utilization, and evaluation.”
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090 · cited 3×
“Zhao et al. [2] review recent advances in LLMs, distinguishing different aspects of LLMs such as pre-training, adaptation tuning, LLM utilization, and evaluation.”
Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72970-6_3 · Cited by 2,074
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · cited 3×
“Image and text are masked randomly following Devlin et al. [86] and He et al. [65], and the model is trained on joint conditional reconstruction task as well as Image-text Contrastive (ITC) [42], [43] and Image Text Matching (TIM) [49] task.”
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · cited 4×
“To avoid these steps, Jia et al. in ALIGN [43] gather one billion noisy image-caption pairs from Conceptual Captions Dataset [54].”
InfluentialJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · cited 1×
“Vicuna-Instruction-80 introduces this LLM-as-a-judge approach [128], [171] benchmark for LLM that comprises of 9 instruction categories: generic, knowledge, math, counterfactual, Fermi, coding, writing, roleplay, common-sense .”
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · cited 3×
“…[86], Chen et al. [49] propose Universal Image-TExt Representation ( UNITER ), a method to leverage conventional image-text datasets (COCO [142], Visual Genome [143], Conceptual Captions [54], SBU Captions [144]) to train foundation models that can be used for heterogeneous vision-language tasks.”
Diffusion Models in Vision: A Survey
Croitoru, F.-A., Hondru, V., Ionescu, R. T., & Shah, M. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3261988 · Cited by 1,859
Self-citationLi, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741 · cited 1×
“Most foundation models rely on the large-scale nature of noisy image-text datasets, which Li et al. [122] argue is a suboptimal strategy.”
Diffusion Models: A Comprehensive Survey of Methods and Applications
Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., & Yang, M.-H. (2024) · ACM Computing Surveys · doi:10.1145/3626235 · Cited by 1,704 · cited 1×
“Note that we do not cover image generative models aimed at model data distribution such as GANs, VAEs, and Diffusion models owing to dedicated surveys already existing in this area [38], [39], [40], [41] and because the former model class can cover a broader range of downstream applications.”
Self-citationCarpenter, P. A., Just, M. A., & Shell, P. (1990) · Psychological Review · doi:10.1037/0033-295x.97.3.404 · Cited by 1,648
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
Flamingo: a Visual Language Model for Few-Shot Learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.14198 · Cited by 1,555 · cited 2×
“To bridge this gap, multimodal open-source foundation models such as BLIP2 [125], GIT [170], and Flamingo [104] can be extended with the instruction-following and human intent alignment to obtain ChatGPT-like in the multimodal space.”
YFCC100M: the new data in multimedia research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., & Li, L.-J. (2016) · Communications of the ACM · doi:10.1145/2812802 · Cited by 1,541
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 2×
“However, the use of an external GPT4 model as a gold standard is still debatable, and new efforts in LLM for benchmarking and identifying the corner cases have been reported to address the limitations of existing evaluation measures, e.g., [36], [173], [174].”
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · cited 1×
“Prompt Engineering: Prompt engineering has primarily been used with Large Language Models (LLMs) to make them do certain tasks [3], [53].”
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., Chen, D., & Guo, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01181 · Cited by 1,358 · cited 1×
“Similarly, a METER [56] head is used for vision language representation, and CSwin [57] is used for video-based understanding.”
COCO-Stuff: Thing and Stuff Classes in Context
Caesar, H., Uijlings, J., & Ferrari, V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00132 · Cited by 1,277
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · cited 1×
“It not only enables rapid model development and provides better performance for both in-domain and out-domain scenarios but also leads to the “emergent properties” of intelligence from large-scale foundational models trained on massive datasets [35], [36].”
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00823 · Cited by 1,250
Grounded Language-Image Pre-training
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01069 · Cited by 1,086
(2016) · doi:10.1007/978-3-319-46475-6_5 · Cited by 1,081
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078 · cited 1×
“Similarly, Image-Text Matching ( ITM ) loss [44] aims to correctly predict whether a pair of images and text is positively or negatively matched.”
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · cited 1×
“The work of [26] reviews transformer techniques for multimodal data with a survey of vanilla transformers, vision transformers, and multimodal transformers from a geometrically topological perspective.”
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 1×
“GPT4 is then fine-tuned with Reinforcement Learning from Human Feedback (RLHF) [162].”
Vision-Language Models for Vision Tasks: A Survey
Zhang, J., Huang, J., Jin, S., & Lu, S. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3369699 · Cited by 1,051 · cited 1×
“The work of [28], [29] summarizes different vision-language pre-training network architectures, objectives, and downstream tasks and catego-rizes vision-language pre-training frameworks.”
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., & Lischinski, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00209 · Cited by 1,023 · cited 1×
“Similarly, GANs and diffusion models employ CLIP and its variants for text-driven image manipulation and editing [220], [221].”
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
Li, J., Li, D., Savarese, S., & Hoi, S. (2023) · arXiv · doi:10.48550/arxiv.2301.12597 · Cited by 908 · cited 3×
“Like BLIP-2 [125], the architecture consists of a visual encoder, a Q-Former, and an LLM.”
Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · cited 1×
“The PaLI architecture combines a text encoder-decoder transformer (mT5) [121] with a ViT [90] for visual tokens.”
A Comprehensive Survey of Deep Learning for Image Captioning
Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H. (2019) · ACM Computing Surveys · doi:10.1145/3295748 · Cited by 879 · cited 1×
“StandardCaptioning( Cap )loss[45] also aims to predict the next token given previous tokens and images.”
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · cited 4×
“Approaches like ImageBind [17] demonstrate viable ways to achieve alignment between several modalities.”
InfluentialHallucination of Multimodal Large Language Models: A SurveyPreprint
Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z., & Shou, Z. (2024) · arXiv (Cornell University) · doi:10.13140/rg.2.2.27797.13280 · Cited by 836
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834 · cited 1×
“Similarly, the recent 540-billion parameter Pathways Language Model (PaLM) has demonstrated state-of-the-art capabilities on numerous challenging problems ranging from language understanding and generation to reasoning and code-related tasks [4], [5].”
Faster Segment Anything: Towards Lightweight SAM for Mobile ApplicationsPreprint
Zhang, C., Han, D., Qiao, Y., Kim, J. U., Bae, S., Lee, S., & Hong, C. S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.14289 · Cited by 808 · cited 1×
“Similarly, visual prompt-based models like Segment Anything [11] would benefit from a real-time speed to ensure intractability [150], [192].”
Do ImageNet Classifiers Generalize to ImageNet?
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019) · arXiv · doi:10.48550/arxiv.1902.10811 · Cited by 790
Fast Segment AnythingPreprint
Zhao, X. F., Ding, W., An, Y., Du, Y., Yu, C., Li, M., Tang, M., & Wang, J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.12156 · Cited by 740 · cited 1×
“Similarly, visual prompt-based models like Segment Anything [11] would benefit from a real-time speed to ensure intractability [150], [192].”
Medical SAM adapter: Adapting segment anything model for medical image segmentation
Wu, J., Wang, Z., Hong, M., Ji, W., Fu, H., Xu, Y., Xu, M., & Jin, Y. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103547 · Cited by 677
Reproducible Scaling Laws for Contrastive Language-Image Learning
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., & Jitsev, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00276 · Cited by 673 · cited 1×
“Cherti et al. [74] supplement this open-source effort by studying the scaling laws of CLIP.”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023) · arXiv · doi:10.48550/arxiv.2304.13712 · Cited by 644
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630
RegionCLIP: Region-based Language-Image Pretraining
Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01629 · Cited by 595
PaLM-E: An Embodied Multimodal Language Model
Driess, D., Xia, F., Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.03378 · Cited by 591 · cited 1×
“Notably, Palm-E [156] integrates continuous sensor input data into a language model, facilitating grounded inference.”
DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation
Kim, G., Kwon, T., & Ye, J. C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00246 · Cited by 570 · cited 1×
“Similarly, GANs and diffusion models employ CLIP and its variants for text-driven image manipulation and editing [220], [221].”
Image Segmentation Using Text and Image Prompts
Luddecke, T., & Ecker, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00695 · Cited by 562 · cited 1×
“CLIPSeg [147] exploits CLIP’s [42] generalization capabilities to achieve zero-shot and one-shot segmentation.”
Evaluating Object Hallucination in Large Vision-Language Models
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.20 · Cited by 559
A comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02443-6 · Cited by 557
FLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01519 · Cited by 554 · cited 2×
“Aimingtodevelopafoundationmodelperformingwellacross vision,language,andvision-languagetasks,Singhetal.propose FLAVA [50] consisting of image and text encoders and multi-modal encoders, vision-task heads, language task heads, and multi-modal task heads.”
Extract Free Dense Labels from CLIP
Zhou, C., Loy, C. C., & Dai, B. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19815-1_40 · Cited by 513
RemoteCLIP: A Vision Language Foundation Model for Remote Sensing
Liu, F., Chen, D., Guan, Z., Zhou, X., Zhu, J., Ye, Q., Fu, L., & Zhou, J. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3390838 · Cited by 507 · cited 1×
“4) and contrastive methods from different perspectives, such as remote sensing [77], visual document understanding [78] and health-care [79].”
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., & Cao, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01855 · Cited by 506
CoCa: Contrastive Captioners are Image-Text Foundation Models
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., & Wu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.01917 · Cited by 503 · cited 1×
“Yu et al. [119] propose a unified encoder-decoder-based model called Con-trastive Captioner ( CoCa ) that has the capabilities of the single encoder, dual encoder, and encoder-decoder models.”
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487
Diversity is All You Need: Learning Skills without a Reward Function
Eysenbach, B., Gupta, A., Ibarz, J., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1802.06070 · Cited by 480
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1657 · Cited by 475
GroupViT: Semantic Segmentation Emerges from Text Supervision
Xu, J., De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J., & Wang, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01760 · Cited by 456
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., & Rombach, R. (2023) · arXiv · doi:10.48550/arxiv.2307.01952 · Cited by 455 · cited 1×
“Role of Visual Language FM Models in Generative Modeling: State-of-the-art (SOTA) generative models extensively rely on visual-language (VL) models for text-guided generation; e.g., open-source Latent Diffusion Models (LDMs) often integrate fixed CLIP [218], [219].”
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2022) · arXiv · doi:10.48550/arxiv.2212.10403 · Cited by 428
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models
Maaz, M., Rasheed, H., Khan, S., & Khan, F. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.679 · Cited by 424
Self-citationBig Self-Supervised Models are Strong Semi-Supervised Learners
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., & Hinton, G. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.10029 · Cited by 421
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., & Hoi, S. C. H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.06500 · Cited by 407 · cited 3×
“…under three primary settings: to improve a model’s performance on a specific task (e.g., open-world object detection), to improve a model for a particular capability (e.g., visual grounding), and to instruction-tune a model to make it solve different downstream vision tasks (e.g., InstructBLIP [9]).”
Opt: Open pre-trained transformer language models
Zhang, S. (2025) · TIB Data Manager · doi:10.57702/dn0pmcbs · Cited by 405
CRIS: CLIP-Driven Referring Image Segmentation
Wang, Z., Lu, Y., Li, Q., Tao, X., Guo, Y., Gong, M., & Liu, T. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01139 · Cited by 404
Florence: A New Foundation Model for Computer Vision
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.11432 · Cited by 389 · cited 2×
“Computer vision has recently witnessed significant progress fueled by foundational models [11], [37] with an extensive body of literature encompassing discriminative and generative models.”
Chen, K., Liu, C., Chen, H., Zhang, H., Li, W., Zou, Z., & Shi, Z. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3356074 · Cited by 383 · cited 1×
“Further, such a generic visual prompt-based segmentation model can be adapted for specific downstream tasks such as medical image segmentation [12], [13], video object segmentation [14], robotics [15], and remote sensing [16].”
DaViT: Dual Attention Vision Transformers
Ding, M., Xiao, B., Codella, N., Luo, P., Wang, J., & Yuan, L. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20053-3_5 · Cited by 382
Learn to Explain: Multimodal Reasoning Via Thought Chains for Science Question Answering
Lu, P., Mishra, S. R., Xia, T., Qiu, L., Chang, K.-W., Zhu, S. C., Tafjord, O., Clark, P., & Kalyan, A. (2022) · neural information processing systems · doi:10.52202/068431-0182 · Cited by 362
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358
Multimodal Few-Shot Learning with Frozen Language Models
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., & Hill, F. (2021) · arXiv · doi:10.48550/arxiv.2106.13884 · Cited by 355 · cited 1×
“Tsimpoukelli et al. [102] propose Frozen , an efficient approach to add visual modality in the LMs without updating their weights.”
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Dou, Z.-Y., Xu, Y., Gan, Z., Wang, J., Wang, S., Wang, L., Zhu, C., Zhang, P., Yuan, L., Peng, N., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01763 · Cited by 348 · cited 1×
“Similarly, a METER [56] head is used for vision language representation, and CSwin [57] is used for video-based understanding.”
Simple Open-Vocabulary Object Detection
Minderer, M., Gritsenko, A., Stone, A., Neumann, M., Weissenborn, D., Dosovitskiy, A., Mahendran, A., Arnab, A., Dehghani, M., Shen, Z., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20080-9_42 · Cited by 348
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
Ghiasi, G., Gu, X., Cui, Y., & Lin, T.-Y. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20059-5_31 · Cited by 339
Audioclip: Extending Clip to Image, Text and Audio
Guzhov, A., Raue, F., Hees, J., & Dengel, A. (2022) · ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp43922.2022.9747631 · Cited by 330
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 327 · cited 1×
“Wang et al. [111] pro-pose a minimalist pre-training framework for vision-language models.”
ImageNet: Constructing a large-scale image database
Fei-Fei, L., Deng, J., & Li, K. (2010) · Journal of Vision · doi:10.1167/9.8.1037 · Cited by 286
Valley: Video Assistant with Large Language Model Enhanced Ability
Luo, R., Zhao, Z., Min, Y., Yang, Z., Qiu, M., Wei, Z., Wang, Y., & Chen, C. (2026) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3796716 · Cited by 280
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
Gu, X., Lin, T.-Y., Kuo, W., & Cui, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13921 · Cited by 273
ViperGPT: Visual Inference via Python Execution for Reasoning
Surís, D., Menon, S., & Vondrick, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01092 · Cited by 273
Yang, J., Li, C., Dai, X., Yuan, L., & Gao, J. (2022) · arXiv · doi:10.48550/arxiv.2203.11926 · Cited by 265
EVA-02: A visual representation for neon genesis
Fang, Y., Sun, Q., Wang, X., Huang, T., Wang, X., & Cao, Y. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105171 · Cited by 264
SLIP: Self-supervision Meets Language-Image Pre-training
Mu, N., Kirillov, A., Wagner, D., & Xie, S. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19809-0_30 · Cited by 258
Classification is a Strong Baseline for Deep Metric LearningPreprint
Zhai, A., & Wu, H. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12649 · Cited by 254
Evaluation and Analysis of Hallucination in Large Vision-Language ModelsPreprint
Wang, J., Zhou, Y., Xu, G., Shi, P., Zhao, C., Xu, H., Ye, Q., Yan, M., Zhang, J., Zhu, J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.15126 · Cited by 248 · cited 1×
“:1)detection[183],[184],[185],[186]byquantifying hallucinations using various methods such as VQAs, classification of objects etc; 2) construction of fine-tuning data aimed at reducing object hallucinations [184], [187], and algorithmic approaches to minimize hallucinations post-training [188],…”
A Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.00234 · Cited by 248
Scaling Language-Image Pre-Training via Masking
Li, Y., Fan, H., Hu, R., Feichtenhofer, C., & He, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02240 · Cited by 247 · cited 2×
“Specifically, Sun et al. [70] present a method to improve training stability and reduce computational costs, including improved initialization, better optimizer, and random masking of images [66].”
High-Performance Large-Scale Image Recognition Without Normalization
Brock, A., De, S., Smith, S. L., & Simonyan, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06171 · Cited by 245
Segment Everything Everywhere All at Once
Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Wang, J., Yang, J., Gao, J., & Lee, Y. J. (2023) · neural information processing systems · doi:10.52202/075280-0868 · Cited by 235
Visual Programming: Compositional visual reasoning without training
Gupta, T., & Kembhavi, A. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01436 · Cited by 230 · cited 1×
“Recently, Visual Programming uses the inter-pretable neuro-symbolic representation to break down a complex task into more straightforward steps that explain the rationale of a particular output from GPT-3 [217].”
Voyager: An Open-Ended Embodied Agent with Large Language Models
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023) · arXiv · doi:10.48550/arxiv.2305.16291 · Cited by 222 · cited 1×
“Furthermore, Mine-Dojo [158] gathers and trains agents on Minecraft data, while VOYAGER [159] is developed for lifelong learning based on LLM.”
Self-Supervised Multimodal Learning: A Survey
Zong, Y., Aodha, O. M., & Hospedales, T. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3429301 · Cited by 221 · cited 1×
“In the context of multimodal learning, the recent review [27] focuses on self-supervised multimodal learning techniques to effectively utilize supervision from raw multimodal data.”
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
Srinivasan, K., Raman, K., Chen, J., Bendersky, M., & Najork, M. (2021) · Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3404835.3463257 · Cited by 214
A Comprehensive Survey on Segment Anything Model for Vision and BeyondPreprint
Zhang, C., Liu, L., Cui, Y., Huang, G., Lin, W., Yang, Y., & Hu, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.08196 · Cited by 210 · cited 1×
“Recently, the work of [32] reviews the visually prompted foundation segmentation model, segmenting anything, and discusses its potential downstream tasks.”
GIT: A Generative Image-to-text Transformer for Vision and Language
Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., & Wang, L. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.14100 · Cited by 205 · cited 1×
“To bridge this gap, multimodal open-source foundation models such as BLIP2 [125], GIT [170], and Flamingo [104] can be extended with the instruction-following and human intent alignment to obtain ChatGPT-like in the multimodal space.”
A Unified Sequence Interface for Vision Tasks
Chen, T., Saxena, S., Li, L., Lin, T.-Y., Fleet, D. J., & Hinton, G. E. (2022) · neural information processing systems · doi:10.52202/068431-2272 · Cited by 204
FILIP: Fine-grained Interactive Language-Image Pre-Training
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., & Xu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07783 · Cited by 202 · cited 1×
“To address this, Yao et al. [64] introduce FILIP (Fine-grained Interactive Language Image Pre-training) with cross-modal late interaction to capture token-wise semantic alignment.”
Are Emergent Abilities of Large Language Models a Mirage?
Schaeffer, R., Miranda, B., & Koyejo, S. (2023) · arXiv · doi:10.48550/arxiv.2304.15004 · Cited by 200 · cited 1×
“However, the use of an external GPT4 model as a gold standard is still debatable, and new efforts in LLM for benchmarking and identifying the corner cases have been reported to address the limitations of existing evaluation measures, e.g., [36], [173], [174].”
(2023) · doi:10.52202/075280-2425 · Cited by 200
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., & Bing, L. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01316 · Cited by 199 · cited 1×
“…hallucinations using various methods such as VQAs, classification of objects etc; 2) construction of fine-tuning data aimed at reducing object hallucinations [184], [187], and algorithmic approaches to minimize hallucinations post-training [188], [189], [190].”
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
Leng, S., Zhang, H., Chen, G., Li, X., Lu, S., Miao, C., & Bing, L. (2023) · arXiv · doi:10.48550/arxiv.2311.16922 · Cited by 199 · cited 1×
“…hallucinations using various methods such as VQAs, classification of objects etc; 2) construction of fine-tuning data aimed at reducing object hallucinations [184], [187], and algorithmic approaches to minimize hallucinations post-training [188], [189], [190].”
Images Speak in Images: A Generalist Painter for In-Context Visual Learning
Wang, X., Wang, W., Cao, Y., Shen, C., & Huang, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00660 · Cited by 197
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., & Duan, N. (2023) · arXiv · doi:10.48550/arxiv.2303.04671 · Cited by 197
Yang, J., Li, C., Dai, X., & Gao, J. (2022) · neural information processing systems · doi:10.52202/068431-0304 · Cited by 193
Generalized Decoding for Pixel, Image, and Language
Zou, X., Dou, Z.-Y., Yang, J., Gan, Z., Li, L., Li, C., Dai, X., Behl, H., Wang, J., Yuan, L., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01451 · Cited by 191
Language Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · neural information processing systems · doi:10.52202/075280-3155 · Cited by 191
A Survey of Vision-Language Pre-Trained Models
Du, Y., Liu, Z., Li, J., & Zhao, W. X. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/762 · Cited by 190
PaLI: A Jointly-Scaled Multilingual Language-Image Model
Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.06794 · Cited by 188 · cited 1×
“Chen et al. [120] propose PaLI and analyze the scaling effects of large image-text models through a jointly scaled architecture and a vast multilingual image-text dataset.”
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182
Unifying Vision-and-Language Tasks via Text Generation
Cho, J., Lei, J., Tan, H. X., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.02779 · Cited by 179 · cited 2×
“Likewise, it shows improved performance compared with established baselines [49], [117] for zero-shot generalization, in-context learning, and fine-tuning.”
LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action
Shah, D., Osinski, B., Ichter, B., & Levine, S. (2022) · arXiv · doi:10.48550/arxiv.2207.04429 · Cited by 179 · cited 1×
“Finally, LM-Nav [160] utilizes VLMs to devise actionable plans in an environment.”
Unified Hallucination Detection for Multimodal Large Language Models
Chen, X., Wang, C., Xue, Y., Zhang, N., Yang, X., Li, Q., Shen, Y., Liang, L., Gu, J., & Chen, H. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.178 · Cited by 174
MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining
Dong, X., Bao, J., Zheng, Y., Zhang, T., Chen, D., Yang, H., Zeng, M., Zhang, W., Yuan, L., Chen, D., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01058 · Cited by 172 · cited 1×
“Similarly, albeit with a different motivation, Dong et al. [67] argue that the language description of an image cannot express complete information as an image is a continuous and fine-grained signal.”
One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningPreprint
Chavan, A., Liu, Z., Gupta, D. K., Xing, E. P., & Shen, Z. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.07967 · Cited by 170
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention
Zhang, R., Han, J., Liu, C., Gao, P., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., & Qiao, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.16199 · Cited by 166
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality
Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.14178 · Cited by 165 · cited 2×
“[10]present mPLUG-OWL , a modular vision-languages model trained on language modeling objectives.”
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al. (2023) · neural information processing systems · doi:10.52202/075280-2688 · Cited by 162
Detecting and Preventing Hallucinations in Large Vision Language Models
Gunjal, A., Yin, J., & Bas, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29771 · Cited by 161
Detecting Everything in the Open World: Towards Universal Object Detection
Wang, Z., Li, Y., Chen, X., Lim, S.-N., Torralba, A. B., Zhao, H., & Wang, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01100 · Cited by 160
Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., & Wang, L. (2023) · arXiv · doi:10.48550/arxiv.2306.14565 · Cited by 149 · cited 2×
“…hallucinations using various methods such as VQAs, classification of objects etc; 2) construction of fine-tuning data aimed at reducing object hallucinations [184], [187], and algorithmic approaches to minimize hallucinations post-training [188], [189], [190].”
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
Wang, W., Bao, H., Li, D., Björck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.10442 · Cited by 145
Retentive Network: A Successor to Transformer for Large Language Models
Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., & Wei, F. (2023) · arXiv · doi:10.48550/arxiv.2307.08621 · Cited by 142 · cited 1×
“Efforts, such as retentive networks [193], can be integrated within VLMs for high throughput processing.”
Adversarial Demonstration Attacks on Large Language ModelsPreprint
Wang, J., Liu, Z., Park, K. H., Jiang, Z., Zheng, Z., Wu, Z., Chen, M., & Xiao, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.14950 · Cited by 139 · cited 1×
“The exemplars (input-output pairs) provided for In-Context Learning of VLMs can also be changed to fool the model [210], [211].”
Language Models are General-Purpose InterfacesPreprint
Hao, Y., Song, H., Li, D., Huang, S., Chi, Z., Wang, W., Ma, S., & Wei, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2206.06336 · Cited by 136 · cited 2×
“Hao et al. [107] develop MetaLM .”
Monkey: Image Resolution and Text Label are Important Things for Large Multi-Modal Models
Li, Z., Yang, B., Liu, Q., Ma, Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., & Bai, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02527 · Cited by 136
Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models
Lovenia, H., Dai, W., Cahyawijaya, S., Ji, Z., & Fung, P. (2024) · Workshop on Advances in Language and Vision Research (ALVR) · doi:10.18653/v1/2024.alvr-1.4 · Cited by 135
What does CLIP know about a red circle? Visual prompt engineering for VLMs
Shtedritski, A., Rupprecht, C., & Vedaldi, A. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01101 · Cited by 134 · cited 1×
“For example, the recent work by Shtedriski et al. [214] shows the sensitivity of the CLIP model towards red circles, where simply drawing a red circle around a person’s face increases its chances of being misclassified as a murderer, suspect, or a missing person as data from new media consists of a…”
BridgeTower: Building Bridges between Encoders in Vision-Language Representation Learning
Xiao, X., Wu, C., Rosenman, S., Lal, V., Che, W., & Duan, N. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26263 · Cited by 129
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2020) · arXiv · doi:10.48550/arxiv.2012.15723 · Cited by 124 · cited 1×
“Prompt Engineering: Prompt engineering has primarily been used with Large Language Models (LLMs) to make them do certain tasks [3], [53].”
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S. C., & Gao, J. (2023) · neural information processing systems · doi:10.52202/075280-1882 · Cited by 120
Image Captioners Are Scalable Vision Learners Too
Tschannen, M., (ed), M. K., Steiner, A., Zhai, X., Houlsby, N., & Beyer, L. (2023) · neural information processing systems · doi:10.52202/075280-2029 · Cited by 114
Ignore Previous Prompt: Attack Techniques For Language Models
Perez, F., & Ribeiro, I. (2022) · arXiv · doi:10.48550/arxiv.2211.09527 · Cited by 110 · cited 1×
“For example, models based on conversational LLMs have been shown vulnerable to adversari-ally prompt injection, which needs a direct interaction between the adversary and the LLM-based conversational agent [204], [205], [206].”
Robot-Enabled Construction Assembly with Automated Sequence Planning based on ChatGPT: RoboGPT
You, H., Ye, Y., Zhou, T., Zhu, Q., & Du, J. (2023) · arXiv · doi:10.48550/arxiv.2304.11018 · Cited by 100
On Robustness and Transferability of Convolutional Neural Networks
Djolonga, J., Yung, J., Tschannen, M., Romijnders, R., Beyer, L., Kolesnikov, A., Puigcerver, J., Minderer, M., D'Amour, A., Moldovan, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.08558 · Cited by 98
Multimodal Chain-of-Thought Reasoning in Language Models
Zhang, Z., Zhang, A., Li, M., Zhao, H. Y., Karypis, G., & Smola, A. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.00923 · Cited by 97 · cited 1×
“Strategies tomitigatehallucinationincludeprovidingexplicitinstructions (orso-called system commands ), the chain of thought prompting [175], [176], [177], self-consistency (voting) [178], [179] and use of knowledge bases for retrieval augmented generation [180], [181], [182].”
ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Application
Wake, N., Kanehira, A., Sasabuchi, K., Takamatsu, J., & Ikeuchi, K. (2023) · IEEE Access · doi:10.1109/access.2023.3310935 · Cited by 96
VIMA: General Robot Manipulation with Multimodal Prompts
Jiang, Y., Gupta, A., Zhang, Z., Wang, G., Dou, Y., Chen, Y., Fei-Fei, L., Anandkumar, A., Zhu, Y., & Fan, L. (2022) · arXiv · doi:10.48550/arxiv.2210.03094 · Cited by 95 · cited 1×
“Similarly, ViMA [157] demonstrates the efficacy of multimodal prompting, interweaving textual and visual tokens, thereby enabling effective learning of robot manipulation through multimodal prompts.”
Deng, A., Chen, Z., & Hooi, B. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.15300 · Cited by 95
Lin, Y.-T., & Chen, Y.-N. (2023) · Proceedings of the 5th Workshop on NLP for Conversational AI (NLP4ConvAI 2023) · doi:10.18653/v1/2023.nlp4convai-1.5 · Cited by 94
Coarse-to-fine vision-language pre-training with fusion in the backbone
Dou, Z.-Y., Kamath, A., Gan, Z., Zhang, P., Wang, J., Li, L., Liu, Z., Liu, C., Lecun, Y., Peng, N., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2387 · Cited by 93
Vision-and-Language Pretrained Models: A Survey
Long, S., Cao, F., Han, S. C., & Yang, H. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/773 · Cited by 87
Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT modelsPreprint
Ranjit, M., Ganapathy, G., Manuel, R. F., & Ganu, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03660 · Cited by 87 · cited 1×
“Strategies tomitigatehallucinationincludeprovidingexplicitinstructions (orso-called system commands ), the chain of thought prompting [175], [176], [177], self-consistency (voting) [178], [179] and use of knowledge bases for retrieval augmented generation [180], [181], [182].”
Multi-Task Self-Training for Learning General Representations
Ghiasi, G., Zoph, B., Cubuk, E. D., Le, Q. V., & Lin, T.-Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00873 · Cited by 84
Masked Vision and Language Modeling for Multi-modal Representation LearningPreprint
Kwon, G., Cai, Z., Ravichandran, A., Bas, E., Bhotika, R., & Soatto, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.02131 · Cited by 83 · cited 1×
“Kwon et al. [140] develop a joint masked reconstruction language model where the masked input of one is reconstructed conditioned on other unmasked input.”
A Closer Look at the Robustness of Contrastive Language-Image Pre-Training (CLIP)
Tu, W., Deng, W., & Gedeon, T. (2023) · neural information processing systems · doi:10.52202/075280-0603 · Cited by 80
WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
Huo, Y., Zhang, M., Liu, G., Lu, H., Gao, Y., Yang, G., Wen, J., Zhang, H., Xu, B., Zheng, W., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06561 · Cited by 79 · cited 1×
“Huo et al. [62] propose WenLan to address this issue by using two-tower architecture and cross-modal contrastive learning based on MoCo [63], which can leverage more negative samples in limited GPU resources.”
EVA-CLIP: Improved Training Techniques for CLIP at Scale
Sun, Q., Fang, Y., Wu, L., Wang, X., & Cao, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.15389 · Cited by 79 · cited 2×
“Specifically, Sun et al. [70] present a method to improve training stability and reduce computational costs, including improved initialization, better optimizer, and random masking of images [66].”
An Inverse Scaling Law for CLIP Training
Li, X., Wang, Z., & Xie, C. (2023) · neural information processing systems · doi:10.52202/075280-2132 · Cited by 72
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., & Anandkumar, A. (2022) · arXiv · doi:10.48550/arxiv.2206.08853 · Cited by 72 · cited 1×
“Furthermore, Mine-Dojo [158] gathers and trains agents on Minecraft data, while VOYAGER [159] is developed for lifelong learning based on LLM.”
SegGPT: Segmenting Everything In ContextPreprint
Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., & Huang, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.03284 · Cited by 70 · cited 2×
“2): textually prompted models (contrastive, generative, hybrid, and conversational), visually prompted models (e.g., SegGPT [33], SAM [11]) and heterogeneous modalities-based models (e.g., ImageBind [17], Valley [34]).”
VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution
Hall, S. M., Abrantes, F. G., Zhu, H., Sodunke, G., Shtedritski, A., & Kirk, H. R. (2023) · neural information processing systems · doi:10.52202/075280-2782 · Cited by 70
OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S. Y., Sagawa, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01390 · Cited by 69 · cited 1×
“OpenFlamingo [106] is an open-source version of the Flamingo model based on the same architecture but with a new Multimodal C4 dataset and 10M samples from LAION-2B.”
YFCC100M: The New Data in Multimedia Research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv · doi:10.48550/arxiv.1503.01817 · Cited by 69
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners
Ren, A. Z., Dixit, A., Bodrova, A., Singh, S., Tu, S., Brown, N., Xu, P., Takayama, L., Xia, F., Varley, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.01928 · Cited by 69
Visual Instruction Tuning towards General-Purpose Multimodal Large Language Model: A Survey
Huang, J., Zhang, J., Jiang, K. Y., Qiu, H., Zhang, X., Shao, L., Lu, S., & Tao, D. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02572-7 · Cited by 63
A Survey on Generative Diffusion Model
Cao, H., Tan, C., Gao, Z., Xu, Y., Chen, G., Heng, P., & Li, S. Z. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.02646 · Cited by 62 · cited 1×
“Note that we do not cover image generative models aimed at model data distribution such as GANs, VAEs, and Diffusion models owing to dedicated surveys already existing in this area [38], [39], [40], [41] and because the former model class can cover a broader range of downstream applications.”
Chat with the Environment: Interactive Multimodal Perception Using Large Language Models
Zhao, X., Li, M., Weber, C., Hafez, M. B., & Wermter, S. (2023) · 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342363 · Cited by 62
Mass-Producing Failures of Multimodal Systems with Language Models
Tong, S., Jones, E., & Steinhardt, J. (2023) · neural information processing systems · doi:10.52202/075280-1275 · Cited by 60
Research works printed but not linked: 10
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1
- Schuhmann, LAION-5B: An open large-scale dataset for training next generation image-text models, 25278
- 10.5555/3524938.3525087 · doi:10.5555/3524938.3525087
- Liu, 2019, RoBERTa: A robustly optimized bert pretraining approach, ArXiv
- Zhang, GLIPv2: Unifying localization and vision-language understanding, 36067
- Kwon, Masked vision and language modeling for multi-modal representation learning, 4
- Ordonez, Im2text: Describing images using 1 million captioned photographs, 3
- 2023, GPT-4 technical report, PREPRINT
- 10.18653/v1/2023.emnlp-main.20 · DOI · Paper page
- Hu, LoRA: Low-rank adaptation of large language models, 3
Cited by
Segment anything model for medical image segmentation: Current applications and future directions
Zhang, Y., Shen, Z., & Jiao, R. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108238 · Cited by 332
Foundation Models for Time Series Analysis: A Tutorial and Survey
Liang, Y., Wen, H., Nie, Y., Jiang, Y., Jin, M., Song, D., Pan, S., & Wen, Q. (2024) · Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3637528.3671451 · Cited by 225
GLaMM: Pixel Grounding Large Multimodal Model
Rasheed, H., Maaz, M., Shaji, S., Shaker, A., Khan, S., Cholakkal, H., Anwer, R. M., Xing, E., Yang, M.-H., & Khan, F. S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01236 · Cited by 189
Self-citationHadi, M. U., tashi, Q. A., Qureshi, R., Shah, A., muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Wu, J., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 103
Self-citationVSCode: General Visual Salient and Camouflaged Object Detection with 2D Prompt Learning
Luo, Z., Liu, N., Zhao, W., Yang, X., Zhang, D., Fan, D.-P., Khan, F. S., & Han, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01625 · Cited by 237
Self-citation
Metadata
- Authors
- Awais, M., Naseer, M., Khan, S., Anwer, R. M., Cholakkal, H., Shah, M., Yang, M.-H., & Khan, F. S.
- Affiliations
- MBZ University of AI, Abu Dhabi, UAE
- Australian National University (AU)
- Mohamed bin Zayed University of Artificial Intelligence (AE)
- University of Central Florida (US)
- University of California, Merced (US)
- Linköping University (SE)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2025-04
- Volume
- 47
- Issue
- 4
- Pages
- 2245-2264
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4406208390
- Cited by
- 304
- References
- 198
- Citations by year
- 2026: 184 · 2025: 118
- FWCI
- 105.44
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- vision foundation models, multimodal models, vision-language models, prompting, pre-training dataset, fine-tuning, contrastive learning, generative models, benchmarking, adversarial attacks, interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.