Flamingo: a visual language model for few-shot learning

Alayrac, Jean-Baptiste; Donahue, Jeff; Luc, Pauline; Miech, Antoine; Barr, Iain; Hasson, Yana; Lenc, Karel; Mensch, Arthur; Millican, Katherine; Reynolds, Malcolm; Ring, Roman; Rutherford, Eliza; Cabi, Serkan; Han, Tengda; Gong, Zhitao; Samangooei, Sina; Monteiro, Marianne; Menick, Jacob L; Borgeaud, Sebastian; Brock, Andy; Nematzadeh, Aida; Sharifzadeh, Sahand; Binkowski, Mikolaj; Barreira, Ricardo; Vinyals, Oriol; Zisserman, Andrew; Simonyan, Karén

doi:10.52202/068431-1723

Cited by 966Domain Adaptation and Few-Shot LearningMultimodal Machine Learning ApplicationsAdvanced Neural Network Applications

Flamingo: a visual language model for few-shot learning

Alayrac, Jean-Baptiste, Donahue, Jeff, Luc, Pauline, Miech, Antoine, Barr, Iain, Hasson, Yana, Lenc, Karel, Mensch, Arthur +19 more·2022·Advances in Neural Information Processing Systems 35

DOI 10.52202/068431-1723

Open this paper in CoChat — free account

Abstract

Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer, captioning tasks, which evaluate the ability to describe a scene or an event, and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.

Integrity signals

Registry

Crossref · 10.52202/068431-1723

Cites

154 of 154 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Vision-language models combine a language model with an image encoder and are trained on text-image pairs, enabling prediction of word continuations in the context of an image.
“vision-language models (VLMs) such as VilBERT 19 and Flamingo 2 combine a language model with an image encoder and are trained on a multi-modal corpus of text-image pairs.This enables them to predict how a given sequence of words will continue in the context of a given image.”
From Talking about Large Language Models (2024) · cites this paper as [2]
The latest generation of language models are multimodal, allowing interleaved image and text input.
“the latest generation of language models are multimodal, allowing interleaved image and text input (OpenAI, 2023a; Alayrac et al., 2022; Team et al., 2023; Li et al., 2023b)”
From Cognitive Architectures for Language Agents (2023) · cites this paper as Alayrac et al., 2022
Switch Transformer was one of the first MoE-based Transformers, activating experts selectively in each layer, reducing computational overhead while outperforming dense Transformers in NLP tasks.
“The Switch Transformer [201] was one of the first MoE-based Transformers, activating experts selectively in each layer, reducing computational overhead while outperforming dense Transformers in NLP tasks.”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [201]
The dominant architectural paradigm connects specialized, external multimodal encoders-such as CLIP for vision or CLAP for audio-to the LLM backbone via alignment modules like Q-Former or simple MLPs, a modular design that allows for independent encoder updates without retraining the entire model.
“The dominant architectural paradigm connects specialized, external multimodal encoders-such as CLIP for vision or CLAP for audio-to the LLM backbone via alignment modules like Q-Former or simple MLPs [19, 86, 615, 1139], a modular design that allows for independent encoder updates without retraining the entire model [624].”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [19]

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3× · see citing works
“Language modelling has recently made substantial progress following the introduction of Transformers [115].”
BackgroundMethodInfluential
Long Short-Term Memory
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2× · see citing works
“To ensure that at initialization, the conditioned model yields the same results as the original language model, we use a tanh-gating mechanism [41].”
Method
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 4× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundMethodInfluential
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 2× · see citing works
“Im ag e ImageNet-1k [94] 3 Object classification Val Top-1 acc.”
Background
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 12× · see citing works
“We evaluate the ability of our models to rapidly adapt to new tasks using in-context learning , analogously to GPT-3 [11], by interleaving support example pairs in the form of ( 𝑖𝑚𝑎𝑔𝑒, 𝑡𝑒𝑥𝑡 ) or ( 𝑣𝑖𝑑𝑒𝑜, 𝑡𝑒𝑥𝑡 ) , followed by the query visual input, to build a prompt (details in Method FT Shot OKVQA ( I )”
BackgroundMethodResultInfluential
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · cited 2× · see citing works
“[13] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.”
BackgroundMethod
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 12× · see citing works
“[85] we use “prompt-ensembling” in which we embed multiple texts using templates such as “A photo of a {class_name}” and average the resulting embedding.”
BackgroundMethodInfluential
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 3× · see citing works
“To measure the importance of text pretraining, we compare the performance of using a frozen decoder-only Transformer either pretrained on MassiveText (our main model) or pretrained on the C4 dataset [87] (row (iv) of Table 10).”
BackgroundMethod
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512 · cited 2× · see citing works
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
BackgroundMethod
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 2× · see citing works
“[116] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.”
BackgroundResultSelf-citation
Prototypical Networks for Few-shot LearningPreprint
Snell, J., Swersky, K., & Zemel, R. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.05175 · Cited by 5,863 · cited 2× · see citing works
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
BackgroundMethod
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 2× · see citing works
“The paradigm of first pretraining on a vast amount of data followed by an adaptation on a downstream task has become standard [11, 23, 32, 44, 52, 75, 87, 108].”
Background
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 2× · see citing works
“Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
BackgroundResultSelf-citation
Matching Networks for One Shot LearningPreprint
Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., & Wierstra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04080 · Cited by 4,874 · cited 2× · see citing works
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
BackgroundMethodSelf-citation
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2× · see citing works
“The paradigm of first pretraining on a vast amount of data followed by an adaptation on a downstream task has become standard [11, 23, 32, 44, 52, 75, 87, 108].”
Background
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770 · cited 2× · see citing works
“One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
Background
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684 · cited 1× · see citing works
“[105] Emma Strubell, Ananya Ganesh, and Andrew McCallum.”
Background
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · cited 1× · see citing works
“Note that the frozen LM was trained with the GeLU activation [39], while the remaining trainable transformer layers use the Squared ReLU activation [104], which we found to outperform GeLU.”
Method
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 2× · see citing works
“[56] Brian Lester, Rami Al-Rfou, and Noah Constant.”
Background
One-shot learning of object categories
Fei-Fei, L., Fergus, R., & Perona, P. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.79 · Cited by 3,029 · see citing works
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · cited 1× · see citing works
“The paradigm of first pretraining on a vast amount of data followed by an adaptation on a downstream task has become standard [11, 23, 32, 44, 52, 75, 87, 108].”
Background
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 3× · see citing works
“Note that as explained in our Model Card [77] in Appendix E, this model was developed for research purposes only and should not be used in specific applications before proper risk analyses are conducted and mitigation strategies are explored.”
BackgroundMethodInfluential
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 2× · see citing works
“To achieve this, Flamingo takes inspiration from recent work on large language models (LMs) which are good few-shot learners [11, 18, 42, 86].”
Background
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 2× · see citing works
“– Dialogue 1: Available from DALL·E 2 [89].”
Background
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 1× · see citing works
“These include adding small adapter modules [43], fine-tuning a small part of the LM [141], showing in-context examples in the prompt [11], or optimizing the prompt [56, 60] through gradient descent.”
Background
Datasheets for datasets
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 3× · see citing works
“[30] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford.”
BackgroundInfluential
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · cited 3× · see citing works
“[139] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.”
Background
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · cited 1× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Parameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · cited 4× · see citing works
“In contrast, gradient-based few-shot learning approaches [43] have constant complexity with respect to the number of shots during inference.”
BackgroundInfluential
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,177 · cited 1× · see citing works
“In some cases our results likely trail the state of the art due in part to the fact that we simply optimise log-likelihood and do not make use of common task-specific metric optimisation tricks, such as CIDEr optimisation [64, 90] for COCO captioning, and fine-tuning on dense annotations for VisDial [79].”
Method
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · cited 2× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundMethod
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · cited 1× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · cited 7× · see citing works
“For our image and text pairs we first leverage the ALIGN [50] dataset, composed of 1.”
BackgroundMethodInfluential
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · cited 2× · see citing works
“[98] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.”
Background
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 2× · see citing works
“In the language space, an important line of work has focused on establishing scaling laws for language models [42, 53].”
Background
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation
Li, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741 · cited 1× · see citing works
“Concurrent works [17, 58, 119, 124, 154] also propose to formulate numerous vision tasks as text generation problems.”
Background
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X. P., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,736 · cited 1× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · cited 1× · see citing works
“Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
Background
Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition
Bridle, J. S. (1990) · Springer eBooks · doi:10.1007/978-3-642-76153-9_28 · Cited by 1,278 · cited 1× · see citing works
“We use a learnable temperature parameter in the final log-softmax layer [9].”
Method
VideoBERT: A Joint Model for Video and Language Representation Learning
Sun, C., Myers, A., Vondrick, C., Murphy, K. P., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,212 · cited 1× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Exploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y.-H. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02410 · Cited by 1,202 · cited 2× · see citing works
“The paradigm of first pretraining on a vast amount of data followed by an adaptation on a downstream task has become standard [11, 23, 32, 44, 52, 75, 87, 108].”
BackgroundSelf-citation
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 2× · see citing works
“[125] Jason Wei, Maarten Bosma, Vincent Y.”
Background
Attention on Attention for Image Captioning
Huang, L., Wang, W., Chen, J., & Wei, X.-Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00473 · Cited by 1,092 · see citing works
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078 · cited 1× · see citing works
“Another family of vision-language models is based on contrastive learning [2, 5, 49, 50, 57, 74, 82, 85, 138, 140, 146].”
Background
Categorization and Naming in Children: Problems of Induction
Markman, E. M. (1989) · Choice Reviews Online · doi:10.5860/choice.27-2985 · Cited by 992 · cited 1× · see citing works
“One key aspect of intelligence is the ability to quickly learn to perform a new task given a short instruction [33, 70].”
Background
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00272 · Cited by 986 · see citing works
Self-citation
VinVL: Revisiting Visual Representations in Vision-Language Models
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., & Gao, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00553 · Cited by 965 · see citing works
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · see citing works
Rethinking Few-Shot Image Classification: A Good Embedding is All You Need?
Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, J. B., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58568-6_16 · Cited by 952 · cited 1× · see citing works
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
Method
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950 · cited 2× · see citing works
“[106] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.”
Background
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · cited 13× · see citing works
“• The Flamingo-9B model builds on top of a 7B frozen language model from [42].”
BackgroundMethodInfluentialSelf-citation
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · cited 2× · see citing works
“[99] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.”
BackgroundMethod
Scaling Vision Transformers
Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891 · see citing works
Unified Vision-Language Pre-Training for Image Captioning and VQA
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864 · see citing works
Low-Shot Visual Recognition by Shrinking and Hallucinating Features
Hariharan, B., & Girshick, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.328 · Cited by 824 · see citing works
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 4× · see citing works
“[111], adequately conditioned or fine-tuned Flamingo models could be used for filtering purposes of toxic or harmful samples in the training data.”
BackgroundResultInfluential
Towards VQA Models That Can Read
Singh, A., Natarajan, V., Shah, M., Jiang, Y. L., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00851 · Cited by 777 · see citing works
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.1 · Cited by 765 · cited 2× · see citing works
“[141] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg.”
Background
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · cited 4× · see citing works
“[76] show that the mapping from input to output generally has limited impact on few-shot performance, as opposed to specifying the overall format of the examples.”
BackgroundInfluential
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00331 · Cited by 749 · cited 5× · see citing works
“Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
BackgroundInfluential
Towards Automatic Learning of Procedures From Web Instructional Videos
Zhou, L., Xu, C., & Corso, J. J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12342 · Cited by 663 · see citing works
Meta-learning with differentiable closed-form solversPreprint
Bertinetto, L., Henriques, J. F., Torr, P. H. S., & Vedaldi, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08136 · Cited by 656 · cited 1× · see citing works
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
Method
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 626 · cited 1× · see citing works
“To alleviate this lack of data, numerous works [14, 50, 98, 110] automatically scrape readily available paired vision-text data.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos
Miech, A., Alayrac, J.-B., Smaira, L., Laptev, I., Šivic, J., & Zisserman, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00990 · Cited by 622 · cited 2× · see citing works
“[74] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.”
BackgroundSelf-citation
VizWiz Grand Challenge: Answering Visual Questions from Blind People
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., & Bigham, J. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00380 · Cited by 621 · see citing works
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · cited 2× · see citing works
“In this direction, recent work has shown that non-causal masked language modeling adaptation [120] followed by multitask fine-tuning [95, 125, 131] can efficiently improve the zero-shot performance of causal decoder-only language models.”
Background
ClipCap: CLIP Prefix for Image CaptioningPreprint
Mokady, R., Hertz, A., & Bermano, A. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09734 · Cited by 605 · cited 2× · see citing works
“One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
Background
FLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01519 · Cited by 597 · cited 2× · see citing works
“[101] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.”
Background
VQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · cited 7× · see citing works
“Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
BackgroundMethodInfluential
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563 · see citing works
SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionPreprint
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 517 · cited 4× · see citing works
“1 Restricted SotA† [140] [140] [124] [153] - - [51] [79] [79] [132] [137] [84] [62] 81.”
BackgroundInfluential
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · cited 1× · see citing works
“However, in-context learning is known to be highly sensitive to various aspects of the demonstrations [80, 148], and its inference compute cost and absolute performance scale poorly with the number of shots beyond this low-data regime.”
Background
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
Wang, X., Wu, J., Chen, J., Li, L., Wang, Y., & Wang, W. Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00468 · Cited by 506 · cited 1× · see citing works
“Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
Background
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492 · see citing works
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477 · cited 4× · see citing works
“As such Flamingo is exposed to the same risks of large language models: it can output potentially offensive language, propagate social biases and stereotypes, as well as leaking private information [126].”
BackgroundInfluential
Visual Dialog
Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.121 · Cited by 445 · see citing works
Gender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · cited 2× · see citing works
“7) in terms of gender bias on the Winogender dataset [93] which demonstrate that even though this model is less biased towards gender than previous models [86], gender biases are still present.”
Background
Video Question Answering via Gradually Refined Attention over Appearance and Motion
Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., & Zhuang, Y. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123427 · Cited by 438 · cited 5× · see citing works
“[130] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.”
BackgroundInfluential
Perceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414 · cited 10× · see citing works
“Perceiver Resampler: from varying-size large feature maps to few visual tokens.”
BackgroundMethodInfluentialSelf-citation
ActBERT: Learning Global-Local Video-Text Representations
Zhu, L., & Yang, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00877 · Cited by 414 · cited 1× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
Li, L., Chen, Y.-C., Cheng, Y., Gan, Z., Yu, L., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.161 · Cited by 408 · cited 1× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Women Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404 · cited 1× · see citing works
“Its ability to additionally handle visual inputs poses specific risks such as gender and racial biases relating to the contents of the input images, similar to a number of visual recognition systems [12, 21, 37, 97, 147].”
Background
LiT: Zero-Shot Transfer with Locked-image text Tuning
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01759 · Cited by 402 · cited 3× · see citing works
“Finally, state-of-the-art zero-shot models on ImageNet such as BASIC [82] and LiT [146] are particularly optimized on classification tasks as they are trained on JFT-3B [145], a dataset with images and labels.”
BackgroundMethod
Learning feed-forward one-shot learnersPreprint
Bertinetto, L., Henriques, J. F., Valmadre, J., Torr, P. H. S., & Vedaldi, A. (2016) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.1606.05233 · Cited by 393 · cited 1× · see citing works
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
Method
Florence: A New Foundation Model for Computer Vision
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.11432 · Cited by 389 · cited 5× · see citing works
“[140] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang.”
BackgroundMethodInfluential
FILIP: Fine-grained Interactive Language-Image Pre-TrainingPreprint
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., & Xu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07783 · Cited by 388 · cited 2× · see citing works
“[138] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.”
Background
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · cited 3× · see citing works
“[81] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving.”
BackgroundResultSelf-citation
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358 · cited 1× · see citing works
“We ablate replacing our image-text pairs (ITP) by the publicly available LAION-400M dataset [96], which leads to a slight degradation in performance.”
Method
High-Performance Large-Scale Image Recognition Without NormalizationPreprint
Brock, A., De, S., Smith, S. L., & Simonyan, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06171 · Cited by 356 · cited 2× · see citing works
“Our vision encoder is a pretrained and frozen NormalizerFree ResNet (NFNet) [10] – we use the F6 model.”
MethodSelf-citation
Multimodal Few-Shot Learning with Frozen Language Models
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., & Hill, F. (2021) · arXiv · doi:10.48550/arxiv.2106.13884 · Cited by 355 · cited 3× · see citing works
“One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
BackgroundSelf-citation
The hateful memes challenge: Detecting hate speech in multimodal memes
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D. (2020) · arXiv · doi:10.48550/arxiv.2005.04790 · Cited by 352 · cited 5× · see citing works
“VisDial [20], a visual dialogue benchmark; HatefulMemes [54] a vision and text classification benchmark; NextQA [129] which specially focuses on causality and temporal relation; STAR [128], a multiple-choice question answering task; and RareAct [73], a benchmark measuring compositionality in action recognition.”
BackgroundMethodInfluential
VirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01101 · Cited by 350 · cited 1× · see citing works
“In our ablation studies (Section 3.3), we compare the proposed GATED XATTN - DENSE layers against recent alternatives [22, 68] and explore the effect of how frequently these additional layers are inserted to trade off between efficiency and expressivity.”
MethodInfluential
Large-Scale Adversarial Training for Vision-and-Language Representation LearningPreprint
Gan, Z., Chen, Y.-C., Li, L., Zhu, C., Cheng, Y., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06195 · Cited by 343 · cited 1× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
VLMo: Unified Vision-Language Pre-Training with Mixture-Of-Modality-Experts
Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O. K., Aggarwal, K., Som, S., Piao, S., & Wei, F. (2022) · neural information processing systems · doi:10.52202/068431-2384 · Cited by 336 · cited 1× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA
Yang, Z., Gan, Z., Wang, J., Hu, X., Lu, Y., Liu, Z., & Wang, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20215 · Cited by 334 · cited 1× · see citing works
“One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
Background
A Short Note on the Kinetics-700-2020 Human Action Dataset
Smaira, L., Carreira, J., Noland, E., Clancy, E., Wu, A., & Zisserman, A. (2020) · arXiv · doi:10.48550/arxiv.2010.10864 · Cited by 334 · see citing works
Self-citation
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Sung, Y.-L., Cho, J., & Bansal, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00516 · Cited by 329 · see citing works
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions
Xiao, J., Shang, X., Yao, A., & Chua, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00965 · Cited by 315 · cited 1× · see citing works
“Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
Background
Multiview Transformers for Video Recognition
Yan, S., Xiong, X., Arnab, A., Lu, Z., Zhang, M., Sun, C., & Schmid, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00333 · Cited by 312 · cited 1× · see citing works
“[134] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid.”
Background
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.05482 · Cited by 307 · cited 1× · see citing works
“[127] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael GontijoLopes, Ari S.”
Background
Meta-Learning Probabilistic Inference for Prediction
Gordon, J., Bronskill, J. F., Bauer, M., et al. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1805.09921 · Cited by 291 · cited 2×
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
BackgroundMethod
Unifying Vision-and-Language Tasks via Text GenerationPreprint
Cho, J., Lei, J., Tan, H. X., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.02779 · Cited by 268 · cited 4× · see citing works
“We show in row (ii) the importance of our gradient accumulation strategy compared to using round-robin updates [17].”
BackgroundMethodResultInfluential
CrossTransformers: spatially-aware few-shot transferPreprint
Doersch, C., Gupta, A., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.11498 · Cited by 255 · cited 2× · see citing works
“[24] Carl Doersch, Ankush Gupta, and Andrew Zisserman.”
BackgroundMethodSelf-citation
Self-Supervised MultiModal Versatile NetworksPreprint
Alayrac, J.-B., Recasens, A., Schneider, R. G., Arandjelović, R., Ramapuram, J., Fauw, J. D., Smaira, L., Dieleman, S., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16228 · Cited by 247 · cited 1× · see citing works
“Another family of vision-language models is based on contrastive learning [2, 5, 49, 50, 57, 74, 82, 85, 138, 140, 146].”
BackgroundSelf-citation
Scaling Up Vision-Language Pretraining for Image Captioning
Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01745 · Cited by 241 · cited 1× · see citing works
“Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
Result
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · cited 8× · see citing works
“In the short term, effective approaches include relying on prompting to mitigate any biases and harmful outputs [86].”
BackgroundMethodInfluentialSelf-citation
True Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226 · cited 1× · see citing works
“However, in-context learning is known to be highly sensitive to various aspects of the demonstrations [80, 148], and its inference compute cost and absolute performance scale poorly with the number of shots beyond this low-data regime.”
Background
MERLOT: Multimodal Neural Script Knowledge ModelsPreprint
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02636 · Cited by 224 · cited 2× · see citing works
“[142] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.”
Background
Just Ask: Learning to Answer Questions from Millions of Narrated Videos
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00171 · Cited by 220 · see citing works
Self-citation
Transformer Language Models without Positional Encodings Still Learn Positional Information
Haviv, A., Ram, O., Press, O., Izsak, P., & Levy, O. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2203.16634 · Cited by 210 · cited 2×
“[36] Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy.”
Background
Fixing the train-test resolution discrepancyPreprint
Touvron, H., Vedaldi, A., Douze, M., & Jeǵou, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06423 · Cited by 209 · cited 2× · see citing works
“[113] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou.”
Background
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.00598 · Cited by 205 · cited 2× · see citing works
“One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
Background
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
Chen, J., Guo, H., Yi, K., Li, B., & Elhoseiny, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01750 · Cited by 205 · see citing works
UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and GenerationPreprint
Luo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Li, J., Bharti, T., & Zhou, M. Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.06353 · Cited by 187 · cited 1× · see citing works
“Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
Result
ReZero is All You Need: Fast Convergence at Large DepthPreprint
Bachlechner, T. C., Majumder, B. P., Mao, H. H., Cottrell, G. W., & McAuley, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04887 · Cited by 167 · cited 2× · see citing works
“This multiplies the output of a newly added layer by tanh(α) before adding it to the input representation from the residual connection, where α is a layer-specific learnable scalar initialized to 0 [4].”
Background
All in One: Exploring Unified Video-Language Pre-Training
Wang, J., Ge, Y., Yan, R., Ge, Y., Lin, K. Q., Tsutsui, S., Lin, X., Cai, G., Wu, J., Shan, Y., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00638 · Cited by 164 · see citing works
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound
Zellers, R., Lu, J., Lu, X., Yu, Y., Zhao, Y., Salehi, M., Kusupati, A., Hessel, J., Farhadi, A., & Choi, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01589 · Cited by 163 · cited 2× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundMethod
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Press, O., Smith, N. A., & Lewis, M. (2021) · arXiv · doi:10.48550/arxiv.2108.12409 · Cited by 161 · see citing works
Pix2seq: A Language Modeling Framework for Object DetectionPreprint
Chen, T., Saxena, S., Li, L., Fleet, D. J., & Hinton, G. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10852 · Cited by 159 · see citing works
Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive ProcessesPreprint
Requeima, J., Gordon, J. A. R., Bronskill, J. F., Nowozin, S., & Turner, R. E. (2019) · arXiv (Cornell University) · doi:10.17863/cam.47574 · Cited by 134 · cited 2× · see citing works
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
BackgroundMethod
Does Object Recognition Work for Everyone?Preprint
DeVries, T., Misra, I., Wang, C., & van der Maaten, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02659 · Cited by 132 · cited 3× · see citing works
“However, current automatic vision system performance has been reported to vary with race, gender or when applied across different demographics and geographic regions [12, 21, 97].”
Background
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token ModelingPreprint
Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12681 · Cited by 130 · cited 2× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
KAT: A Knowledge Augmented Transformer for Vision-and-Language
Gui, L., Wang, B., Huang, Q., Hauptmann, A., Bisk, Y., & Gao, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.70 · Cited by 130 · cited 1× · see citing works
“[34] Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao.”
Background
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
Yang, Z., Lu, Y., Wang, J., Yin, X. P., Florencio, D. A. F., Wang, L., Zhang, C., Zhang, L., & Luo, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00864 · Cited by 130 · see citing works
Doing more with less: meta-reasoning and meta-learning in humans and machines
Griffiths, T. L., Callaway, F., Chang, M. B., Grant, E., Krueger, P. M., & Lieder, F. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2019.01.005 · Cited by 128 · cited 1× · see citing works
“One key aspect of intelligence is the ability to quickly learn to perform a new task given a short instruction [33, 70].”
Background
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Dai, W., Hou, L., Shang, L.-F., et al. (2022) · Findings · doi:10.48550/arxiv.2203.06386 · Cited by 115 · cited 2×
“Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
BackgroundResult
Large-Scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline
Murahari, V., Batra, D., Parikh, D., & Das, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_20 · Cited by 107 · see citing works
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B., & Nematzadeh, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00385 · Cited by 100 · cited 2× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundSelf-citation
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
Xu, H., Ghosh, G., Huang, P.-Y., Arora, P., Aminzadeh, M., Feichtenhofer, C., Metze, F., & Zettlemoyer, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.370 · Cited by 94 · cited 1× · see citing works
“Others have explored visually-conditioned language generation [17, 114, 119, 124, 132] but have not yet shown good performance in low-data regimes.”
Background
A Multimodal Framework for the Detection of Hateful MemesPreprint
Lippe, P., Holla, N., Chandra, S., Rajamanickam, S., Antoniou, G., Shutova, E., & Yannakoudakis, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.12871 · Cited by 92 · cited 2× · see citing works
“1 Restricted SotA† [140] [140] [124] [153] - - [51] [79] [79] [132] [137] [84] [62] 81.”
Background
VD-BERT: A Unified Vision and Dialog Transformer with BERT
Wang, Y., Joty, S., Lyu, M. R.-T., King, I., Xiong, C., & Hoi, S. C. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.269 · Cited by 91 · cited 3× · see citing works
“[123] Yue Wang, Shafiq Joty, Michael Lyu, Irwin King, Caiming Xiong, and Steven Hoi.”
Background
Diagnosing Gender Bias in Image Recognition Systems
Schwemmer, C., Knight, C. R., Bello-Pardo, E. D., Oklobdzija, S. N., Schoonvelde, M., & Lockhart, J. W. (2020) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023120967171 · Cited by 91 · cited 3× · see citing works
“[97] Carsten Schwemmer, Carly Knight, Emily D.”
Background
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
Zhu, X., Zhu, J., Li, H., Wu, X., Li, H., Wang, X., & Dai, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01630 · Cited by 85 · cited 1× · see citing works
“Concurrent works [17, 58, 119, 124, 154] also propose to formulate numerous vision tasks as text generation problems.”
Background
Understanding and Evaluating Racial Biases in Image Captioning
Zhao, D., Wang, A., & Russakovsky, O. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01456 · Cited by 84 · cited 1× · see citing works
“Its ability to additionally handle visual inputs poses specific risks such as gender and racial biases relating to the contents of the input images, similar to a number of visual recognition systems [12, 21, 37, 97, 147].”
Background
Combined scaling for zero-shot transfer learning
Pham, H., Dai, Z., Ghiasi, G., Kawaguchi, K., Liu, H., Yu, A. W., Yu, J., Chen, Y.-T., Luong, M.-T., Wu, Y., et al. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126658 · Cited by 78 · cited 5× · see citing works
“Second, the classification performance of Flamingo lags behind that of state-of-the-art contrastive models [82, 85].”
BackgroundMethodInfluential
Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning SolutionPreprint
Zhu, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08290 · Cited by 78 · see citing works
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Bain, M., Nagrani, A., Varol, G., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00175 · Cited by 76 · cited 2× · see citing works
“Another family of vision-language models is based on contrastive learning [2, 5, 49, 50, 57, 74, 82, 85, 138, 140, 146].”
BackgroundSelf-citation
Teaching language models to support answers with verified quotes
Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.11147 · Cited by 53 · cited 3× · see citing works
“[72] Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nat McAleese.”
BackgroundResultSelf-citation
MAGMA – Multimodal Augmentation of Generative Models through Adapter-based Finetuning
Eichenberg, C., Black, S., Weinbach, S., Parcalabescu, L., & Frank, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.179 · Cited by 49 · cited 2× · see citing works
“[26] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank.”
Background
Primer: Searching for Efficient Transformers for Language ModelingPreprint
So, D. R., Mańke, W., Liu, H., Dai, Z., Shazeer, N., & Le, Q. V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08668 · Cited by 48 · cited 2× · see citing works
“[104] David R So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V.”
BackgroundMethod
STAR: A Benchmark for Situated Reasoning in Real-World VideosPreprint
Wu, B., Yu, S., Chen, Z., Tenenbaum, J. B., & Gan, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.09711 · Cited by 44 · see citing works
CM3: A Causal Masked Multimodal Model of the Internet
Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.07520 · Cited by 42 · cited 3× · see citing works
“We emphasize few-shot learning and vision tasks while CM3 [1] primarily evaluates on language-only benchmarks in a zero-shot or fine-tuned setup.”
BackgroundInfluential
ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization
Xu, H., Chen, Y., Du, Y., Shao, N., Yanggang, W., Li, H., & Yang, Z. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.312 · Cited by 39 · cited 2× · see citing works
“[131] Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang.”
Background
Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs
Islam, M. A., Kowal, M. D., Jia, S., Derpanis, K. G., & Bruce, N. D. B. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00083 · Cited by 34 · see citing works
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition
Massiceti, D., Zintgraf, L., Bronskill, J., Theodorou, L., Harris, M. T., Cutrell, E., Morrison, C., Hofmann, K., & Stumpf, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01064 · Cited by 33 · see citing works
UFO: A UniFied TransfOrmer for Vision-Language Representation LearningPreprint
Wang, J., Hu, X., Gan, Z., Yang, Z., Dai, X., Liu, Z., Lu, Y., & Wang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10023 · Cited by 31 · cited 2× · see citing works
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundMethod
Achieving Human Parity on Visual Question Answering
Yan, M., Xu, H., Li, C., Tian, J., Bi, B., Wang, W., Xu, X., Zhang, J., Huang, S., Huang, F., et al. (2022) · ACM Transactions on Information Systems · doi:10.1145/3572833 · Cited by 26 · cited 5× · see citing works
“6 Unrestricted SotA [133] [133] [119] [153] [65] [65] - - [123] - - - [152]”
BackgroundInfluential
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?Preprint
Wang, T., Roberts, A., Hesslow, D., Scao, T. L., Chung, H. W., Beltagy, I., Launay, J., & Raffel, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.05832 · Cited by 23 · cited 2× · see citing works
“[120] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel.”
Background
MURAL: Multimodal, Multitask Retrieval Across LanguagesPreprint
Jain, A., Guo, M., Srinivasan, K., Chen, T., Kudugunta, S., Jia, C., Yang, Y., & Baldridge, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05125 · Cited by 18 · cited 2× · see citing works
“[49] Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge.”
Background
RareAct: A video dataset of unusual interactionsPreprint
Miech, A., Alayrac, J.-B., Ivan Laptev, A., Šivic, J., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01018 · Cited by 14 · cited 4× · see citing works
“Note: We omit RareAct, our 16th benchmark, as it is a zero-shot benchmark with no available fine-tuned results to compare to.”
BackgroundMethodInfluentialSelf-citation
Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model
Qiao, Y., Chen, H., Wang, J., Zhao, S.-S., Chen, Y., Ye, X., Li, Z., Qi, X., Gao, P., & Xie, G.-T. (2021) · arXiv.org · doi:10.48550/arxiv.2106.15332 · Cited by 9 · cited 3×
“1 Restricted SotA† [140] [140] [124] [153] - - [51] [79] [79] [132] [137] [84] [62] 81.”
Background

Flagged first, then by citations

Sorted by citations

Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028 · see citing works
Scientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823 · see citing works
A Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.64 · Cited by 649 · see citing works
Talking about Large Language Models
Shanahan, M. (2024) · Communications of the ACM · doi:10.1145/3624724 · Cited by 381 · see citing works
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15818 · Cited by 276 · see citing works
“…[1, 2, 3] but emergent problem-solving [4, 5, 6] and creative generation of prose [7, 2] and code [8], while vision-language models enable open-vocabulary visual recognition [9, 10, 11] and can even make complex inferences about object-agent interactions in images [12, 13, 14, 15, 16, 17, 18].”
BackgroundInfluential
A Cookbook of Self-Supervised Learning
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12210 · Cited by 182 · see citing works
“These modern SSL models are capable of representing both vision and text, and can be used in a number of applications that are multimodal, from visual-question answering to multimodal generation [Alayrac et al., 2022, Li et al., 2022c, Nichol et al., 2022, Rao et al., 2022].”
Background
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding
Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye, T., Zhang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.16449 · Cited by 176 · see citing works
OCRBench: on the hidden mystery of OCR in large multimodal models
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., & Bai, X. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4235-6 · Cited by 167 · see citing works
“Flamingo [13] and OpenFlamingo [19] enhance a frozen pretrained LLM by incorporating novel gated cross-attention-dense layers, enabling conditioning on visual inputs.”
Language Is Not All You Need: Aligning Perception with Language Models
Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.14045 · Cited by 235 · see citing works
Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155 · see citing works
Eight Things to Know about Large Language Models
Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00612 · Cited by 137 · see citing works
Multimodal Foundation Models: From Specialists to General-Purpose Assistants
Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L., & Gao, J. (2024) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000110 · Cited by 225 · see citing works
Improving Factuality and Reasoning in Language Models through Multiagent Debate
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., & Mordatch, I. (2023) · arXiv · doi:10.48550/arxiv.2305.14325 · Cited by 96 · see citing works
“Our work is also related to existing works that focus on text generation by combining different models (Du et al., 2020; Liu et al., 2022b; Zeng et al., 2022; Alayrac et al., 2022; Du et al., 2023).”
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
Yang, J., Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., & Xie, S. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.00994 · Cited by 120 · see citing works
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98 · see citing works
“This approach has been proposed in Flamingo (Alayrac et al., 2022) with the integration of dense cross-attention blocks among the existing pre-trained layers of the LLM.”
Influential
Cognitive Architectures for Language Agents
Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2023) · arXiv · doi:10.48550/arxiv.2309.02427 · Cited by 72 · see citing works
“Finally, recent advances in vision-language models (VLMs; Alayrac et al., 2022) can simplify grounding by providing a straightforward translation of perceptual data into text (Zeng et al.”
BackgroundMethod
Artificial intelligence for science in quantum, atomistic, and continuum systems
Zhang, X., Wang, L., Helwig, J., Luo, Y., Fu, C., Xie, Y., Liu, M., Lin, Y., Zhao, X., Yan, K., et al. (2025) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000115 · Cited by 123 · see citing works
PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering
Zhang, X., Wu, C., Zhao, Z., Lin, W., Zhang, Y., Wang, Y., & Xie, W. (2023) · arXiv · doi:10.48550/arxiv.2305.10415 · Cited by 66 · see citing works
“Modality Source Images QA pairs VQA-RAD [18] Radiology MedPix ® database 0.3k 3.5k PathVQA [12] Pathology PEIR Digital Library [14] 5k 32.8k SLAKE [23] Radiology MSD [3], ChestX-ray8 [36], CHAOS [15] 0.7k 14k VQA-Med-2021 [5] Radiology MedPix ® database 5k 5k PMC-VQA Mixture * PubMed Central ® 149k 227k Figure 1: The…”
Background
VisualWebArena: evaluating multimodal agents on realistic visual web tasks
Koh, J. Y., Lo, R., Jang, L., Duvvur, V., Lim, M., Huang, P.-Y., Neubig, G., Zhou, S., Salakhutdinov, R., & Fried, D. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.50 · Cited by 50 · see citing works
A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings
Zambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-58344-x · Cited by 45 · see citing works
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“As a result, there is a pressing need for a universal and cross-platform computational framework that can address these challenges.”
<b>JARVIS</b> -1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models
Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., et al. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3511593 · Cited by 36 · see citing works
LLaVA-3D: A Simple Yet Effective Pathway to Empowering LMMs with 3D Capabilities
Zhu, C., Wang, T., Zhang, W., Pang, J., & Liu, X. (2025) · 2025 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00409 · Cited by 42 · see citing works
“Recent advancements in Large Multimodal Models (LMMs) [2, 5, 27, 44] have significantly enhanced their ability to understand and reason over visual and language inputs, leading to remarkable performance in 2D visual tasks.”
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., & Zhou, J. (2023) · arXiv · doi:10.48550/arxiv.2311.07919 · Cited by 61 · see citing works
“For visual modality, GPT4 (OpenAI, 2023), Flamingo (Alayrac et al., 2022), Kosmos (Peng et al., 2023), BLIP (Li et al., 2022), Shikra (Chen et al., 2023), Emu (Sun et al., 2023) and Qwen-VL (Bai et al., 2023) have proposed different integration method to enable image understanding or generation…”
Method
Data-Centric Foundation Models in Computational Healthcare: A Survey
Zhang, Y., Gao, J. G., Tan, Z., Zhou, L., Ding, K., Zhou, M., Zhang, S., & Wang, D. (2026) · ACM Computing Surveys · doi:10.1145/3800677 · Cited by 42 · see citing works
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2023) · arXiv · doi:10.48550/arxiv.2310.13289 · Cited by 67 · see citing works
“Recent studies focus on connecting LLMs with either the encoder of one additional type of input, such as image (Li et al., 2023a; Alayrac et al., 2022; Dai et al., 2023), silent video (Maaz et al., 2023; Chen et al., 2023b; Zhao et al., 2022), audio events (Gong et al., 2023b; Lyu et al., 2023) or…”
Background
GraphLLM: Boosting Graph Reasoning Ability of Large Language Model
Chai, Z., Zhang, T., Wu, L., Han, K., Hu, X., Huang, X., & Yang, Y. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3627488 · Cited by 77 · see citing works
Humanoid Locomotion and Manipulation: Current Progress and Challenges in Control, Planning, and Learning
Gu, Z., Li, J., Shen, W., Yu, W., Xie, Z., McCrory, S., Cheng, X., Shamsah, A., Griffin, R. J., Liu, C. K., et al. (2026) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2025.3579247 · Cited by 55 · see citing works
“Potential approaches include leveraging vision-language models to extract information from videos [263], [264], [265], employing video generation models [266] or physics-based simulation [180], [192] to generate synthetic data, and adopting hybrid strategies that effectively combine these synthetic…”
Orion: A Holistic End-To-End Autonomous Driving Framework by Vision-Language Instructed Action Generation
Fu, H., Zhang, D. H., Zhao, Z., Cui, J., Liang, D., Zhang, C., Zhang, D., Xie, H., Wang, B., & Bai, X. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02302 · Cited by 37 · see citing works
“To alleviate this bur-2 den, QwenVL [4] and Flamingo [2] reduce token redundancy using cross-attention, while Qwen2VL [57] enhances efficiency with dynamic resolution and multimodal rotary position embedding (M-RoPE) for simultaneously processing diverse modalities.”
TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document
Liu, Y., Yang, B., Liu, Q., Li, Z., Ma, Z., Zhang, S., & Bai, X. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3653415 · Cited by 103 · see citing works
“Unlike Flamingo’s [33] Resampler, our Token Resampler employs a similarity-based ﬁltering mechanism to retain the most representative tokens and suppress redundant ones.”
Semantic Gaussians: Open-Vocabulary Scene Understanding With 3D Gaussian Splatting
Guo, J., Ma, X., Fan, Y., Liu, H., & Li, Q. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3675320 · Cited by 40 · see citing works
“…are the most straightforward representation of 3D scenes, but allowing open-vocabulary understanding usually involves 2D vision-language models [6], which could struggle with consistency across different views, likely due to a lack of visual geometric knowledge; Point clouds are popular and…”
Learning visual language models for video understanding
Yang, A. (2023) · theses.fr (ABES) · doi:10.70675/802872a7z8092z4556za1fdz2c7af45f0e72 · Cited by 15 · see citing works
A Survey of Context Engineering for Large Language Models
Mei, L., Yao, J., Ge, Y., Wang, Y., Bi, B., Cai, Y., Liu, J., Li, M., Li, Z.-Z., Zhang, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2507.13334 · Cited by 13 · see citing works
Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Models
Zhang, H., Edelman, B. L., Francati, D., Venturi, D., Ateniese, G., & Barak, B. (2023) · arXiv · doi:10.48550/arxiv.2311.04378 · Cited by 12 · see citing works
Sparks of Large Audio Models: A Survey and Outlook
Latif, S., Shoukat, M., Shamshad, F., Usama, M., Ren, Y., Cuayáhuitl, H., Wang, W., Zhang, X., Togneri, R., Cambria, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12792 · Cited by 17 · see citing works
“Unlike earlier learning-based models that were tailored for specific tasks, these large models boast versatility in addressing diverse tasks [21, 22].”
Background
$π_0$: A Vision-Language-Action Flow Model for General Robot Control
Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.24164 · Cited by 13 · see citing works
“Following the standard late fusion VLM recipe [3, 11, 30], image encoders embed the robot’s image observations into the same embedding space as language tokens.”
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145 · see citing works
“This is accomplished through tool usage [41, 38], multiple input modalities [47, 2], and a wide range of prompting techniques [30], like few-shot learning [5], self-consistency [53], skeleton-of-thought [33], and tree-of-thought [56].”
AgroBench: Vision-Language Model Benchmark in Agriculture
Shinoda, R., Inoue, N., Kataoka, H., Onishi, M., & Ushiku, Y. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00716 · Cited by 14 · see citing works
“CLIP played a key role in introducing more sophisticated visual representation into language explanations e.g., Flamingo [2], BLIP [14, 15], Qwen [5], PaLI [8], LLaVA [17, 18], CogVLM [50], and Emu [44, 45, 51].”
MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
Daxberger, E., Wenzel, N., Griffiths, D., Gang, H., Lazarow, J., Kohavi, G., Kang, K., Eichner, M. L., Yang, Y., Dehghan, A., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00694 · Cited by 16 · see citing works
“MLLMs [49, 50, 64, 86, 103] have attracted signiﬁcant research focus, tracing back to Frozen [106] and Flamingo [6, 8], with more recent works such as LLaVA [80] and MiniGPT-4 [140] introducing visual instruction tuning.”
RoboCache: A Distributed Key–Value Store for Petabyte–Scale Multimodal Robot Learning Datasets
Patel, T., Shivam, S., Padhy, A. K., et al. (2025) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca66035.2025.11430805 · Cited by 7 · see citing works
“To maximize client-side GPU utilization, the API layer includes a pipelined reader that prefetches and batches encoded tensors following the pattern of high-throughput data loaders in vision-language systems [9], [10].”
LLM4AD: Large Language Models for Autonomous Driving—Concept, Review, Benchmark, Experiments, and Future Trends
Cui, C., Ma, Y., Park, S., Yang, Z., Zhou, Y., Liu, P., Lu, J., Peng, J., Zhang, J., Zhang, R., et al. (2026) · Proceedings of the IEEE · doi:10.1109/jproc.2026.3678829 · Cited by 20 · see citing works
“In addition, as these technologies move toward real-world deployment, it will be essential to establish rigorous safety and performance standards to ensure their reliable and responsible implementation.”
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., & Sadigh, D. (2024) · arXiv · doi:10.48550/arxiv.2402.07865 · Cited by 7
“Of primary concern is the generality of our model architecture; while the three component architecture we define in §2 is reflective of the majority of existing VLMs, there are other architecture innovations and optimization procedures that our study does not currently capture; as a notable example, we do not study ar…”
Method
ICL CHARACTERIZATION OF MULTI-MODAL GEO-FOUNDATION MODELS: WHEN CAN VISION-LANGUAGE TRANSFORMERS LEARN GEOSPATIAL TASKS?
Hawarey, M. (2026) · doi:10.65737/airmcs2026446 · Cited by 6 · see citing works
Are large pre-trained vision language models effective construction safety inspectors
Chen, X., & Zou, Z. (2026) · Data-Centric Engineering · doi:10.1017/dce.2026.10044 · Cited by 13 · see citing works
“Therefore, we input five captions without corresponding images for in-context learning, inspired by Alayrac et al. (2022).”
NuPlanQA: A Large-Scale Dataset and Benchmark for Multi-View Driving Scene Understanding in Multi-Modal Large Language Models
Park, S., Cui, C., Ma, Y., Moradipari, A., Gupta, R., Han, K., & Wang, Z. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00756 · Cited by 6 · see citing works
Advancing site-specific disease and pest management in precision agriculture: From reasoning-driven foundation models to adaptive, feedback-based learning
Rai, N., Choi, D. D., Boyd, N. S., et al. (2026) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2026.111859 · Cited by 6 · see citing works
Mdp 3 : a Training-Free Approach for List-Wise Frame Selection in Video-Llms
Sun, H., Lu, S., Wang, H., Chen, Q.-G., Xu, Z., Luo, W., Zhang, K., & Li, M. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02233 · Cited by 9 · see citing works
“With the success of VLMs [1, 44, 71], frames and queries can be mapped into a unified latent embedding space.”
The potential of large language models for sustainable building performance: A review of applications, evaluation practices, and reliability
Ibrahim, A., Alsukkar, M., Xu, Y., Hermaputi, R. L., Yin, S., & Xiao, Y. (2026) · Energy Sustainable Development/Energy for sustainable development · doi:10.1016/j.esd.2026.102104 · Cited by 10 · see citing works
MG-LLaVA: Toward Multi-Granularity Visual Instruction Tuning
Zhao, X., Li, X., Duan, H., Huang, H., Li, Y., Chen, K., & Yang, H. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3643469 · Cited by 6 · see citing works
“We substitute the gated-fusion model with a resampler in [79].”
Method
GR00T N1: An Open Foundation Model for Generalist Humanoid Robots
NVIDIA, :, Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L. "., Fang, Y., Fox, D., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.14734 · Cited by 10 · see citing works
Pseudo-Label Refinement for Multimodal Unsupervised Domain Adaptation
Shi, K., Lu, J., Fang, Z., et al. (2026) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2026.3660284 · Cited by 4 · see citing works
“In recent years, VLMs like CLIP [18] and Flamingo [17] have demonstrated their ability to learn joint representations of images and text through extensive pretraining on large-scale image–text datasets.”
ManuRAG: multi-modal retrieval augmented generation for manufacturing question answering
Li, Y., Dong, Z., Ameri, F., & Zhang, J. (2026) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-026-02800-y · Cited by 4 · see citing works
VLLM-LAD: Visual Large Language Model for Zero-Shot Logical Anomaly Detection
Peng, Y., Lin, X., Ma, N., Liu, C., & Chen, Q. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3675873 · Cited by 4 · see citing works
“Models such as Flamingo [18], LLaVA [19], and BLIP-2 [20] have demonstrated the ability to process both visual and textual inputs, enabling tasks like image captioning, visual question answering, and multimodal reasoning.”
A review of Carnot battery and its role towards smart integrated energy systems: Opportunities and challenges
Guo, W., Jin, X., Jiang, H., Xia, Y., Zhu, J., Qu, B., Zhong, W., Jiang, L., & Lin, X. (2026) · Renewable and Sustainable Energy Reviews · doi:10.1016/j.rser.2026.116928 · Cited by 5 · see citing works
Accelerating Multimodal Large Language Model Inference through Visual Feature Compression and Decoupled Architectures
Li, X., Zhu, Z., Feng, L., et al. (2025) · doi:10.1109/aibdf67964.2025.11440744 · Cited by 4 · see citing works
“Reference [12] proposed a multi-stage visual token filtering mechanism that enables sparse cross-modal attention computation.”
Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion
Lim, D., Kim, Y., Kim, D.-H., Yang, D.-H., & Chang, J.-H. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434748 · Cited by 4 · see citing works
“1(b), inspired by the Flamingo [24].”
Multimodal large language models for food safety detection within deep learning frameworks: a review
Ding, H., Chen, C., Song, X., Dong, G., Chen, H., Hou, H., Cui, X., Yu, W., & Wilsone, D. I. (2026) · Food Chemistry · doi:10.1016/j.foodchem.2026.149208 · Cited by 4 · see citing works
Decoding the multimodal maze: A systematic review on the adoption of explainability in multimodal attention-based models
Kibria, M. R., Lafond, S., & Arslan, J. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104405 · Cited by 11 · see citing works
ImageGen-CoT: Enhancing Text-to-Image in-context Learning with Chain-of-Thought Reasoning
Liao, J., Yang, Z., Li, L., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01599 · Cited by 5 · see citing works
“This ability allows LLMs to adapt to new tasks by observing a few illustrative examples provided as context, without requiring parameter updates.”
MISSRAG: Addressing the Missing Modality Challenge in Multimodal Large Language Models
Pipoli, V., Saporita, A., Bolelli, F., Cornia, M., Baraldi, L., Grana, C., Cucchiara, R., & Ficarra, E. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00308 · Cited by 4 · see citing works
“By leveraging these complementary modalities, multimodal models have achieved remarkable success in tasks like image captioning [2, 5, 16, 29], audio-video question answering [11, 17, 28], and cross-modal retrieval [1, 9, 35, 40].”
FE-CLIP: Frequency Enhanced CLIP Model for Zero-Shot Anomaly Detection and Segmentation
Gong, T., Chu, Q., Liu, B., Zhou, W., & Yu, N. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01971 · Cited by 6 · see citing works
“Among recent successes of large pre-trained vision-language models (VLM) [2, 20, 33], CLIP [33] is the ﬁrst to perform pre-training on web-scale image-text data, showing unprecedented generality: e.g., its language-driven zero-shot inference, improved both effective robustness [40] and perceptual…”
Large language models for transportation research: Methodologies, state of the art, and future opportunities
Yan, Y.-M., Zou, B., Liao, Y., Xu, G.-H., Yao, R., Fan, H., Sun, J., Wang, X., Sprinkle, J., An, Z.-Y., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104546 · Cited by 6 · see citing works
Privacy-Preserving Multimodal Reasoning for Internet of Things: A Retrieval-Augmented Large Language and Vision Assistant Framework
Ni, T., Yuan, X., Li, S., & Ni, W. (2026) · IEEE Internet of Things Magazine · doi:10.1109/miot.2026.3650753 · Cited by 3 · see citing works
“Compared with caption-centric LLM pipelines and non-private MLLM ﬁ netuning, our focus is a privacy-ﬁ rst integration suitable for IoT edge settings [4], [8].”
Influential
AquaticCLIP: A Vision-Language Foundation Model and Dataset for Underwater Scene Analysis
Alawode, B., Ganapathi, I. I., Javed, S., Bennamoun, M., & Mahmood, A. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2026.3657138 · Cited by 8 · see citing works
Mixture of experts for radiology report generation
Song, X., Liu, Z., Hou, X., Li, X., & Zhang, Y. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.114437 · Cited by 3 · see citing works
Short-Video Propagation Influence Rating: A New Real-World Dataset and a New Large Graph Model
Xue, D., Qian, S., Hu, C., & Xu, C. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3674132 · Cited by 3 · see citing works
“The extension of LLMs to multimodal domains has also gained traction, with models such as GPT-4 [29], Flamingo [37], and Qwen2-VL [17] supporting both textual and visual inputs.”
Vision–language foundation model driven agentic AI systems for healthcare
Chen, L., Xu, X., Qin, Y., Li, H., & Jiang, N. (2026) · The Visual Computer · doi:10.1007/s00371-026-04437-7 · Cited by 3 · see citing works
LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Multilingual Text-Centric VQA
Huang, J., Tan, Z., Gong, S., Zeng, F., Zhou, J. T., Miao, C., Tan, H., Yao, W., & Li, J. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792793 · Cited by 3 · see citing works
Low-light image enhancement: a multi-stage hybrid approach via Retinex and vision transformers
Sen, S. Y., Türkan, M., & Alankuş, G. (2026) · The Visual Computer · doi:10.1007/s00371-026-04485-z · Cited by 3 · see citing works
“…processing and deep learning with the advent of vision language models (VLMs) and MLLMs. VLMs (e.g., CLIP [63], ALIGN [64]) intend to align the visual and textual features for retrieval tasks, while MLLMs (e.g., Flamingo [65], Kosmos-1 [66]) fortify advanced multimodal generation and reasoning.”
Spatial Preference Rewarding for MLLMs Spatial Understanding
Qiu, H., Gao, P., Lu, L., Zhang, X.-Q., Shao, L., & Lu, S. (2025) · IEEE International Conference on Computer Vision · doi:10.1109/iccv51701.2025.00075 · Cited by 5 · see citing works
“Recently, the success of large language models (LLMs) [14, 49, 51] has been extended into the multimodal domain, resulting in models that demonstrate impressive performance in integrating vision and language [1, 15, 20, 32, 70].”
EgoM2P: Egocentric Multimodal Multitask Pretraining
Li, G. K., Chen, Y., Wu, Y., Zhao, K., Pollefeys, M., & Tang, S. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01008 · Cited by 3 · see citing works
“CLIP [81] recognition [42, 100, 128], hand motion and object interaction estimation [9, 24, 118], egocentric video prediction and generation [55, 57, 107, 114], egocentric camera localization [85, 93], among others.”
Token Activation Map to Visually Explain Multimodal LLMs
Li, Y., Wang, H., Ding, X., Wang, H., & Li, X. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00012 · Cited by 8 · see citing works
“Although extensive efforts have been devoted to enhancing the performance of MLLMs [4, 16, 35, 47, 51], research on MLLM explainabil-ity remains less explored.”
Kestrel: 3D Multimodal LLM for Part-Aware Grounded Description
Ahmed, M., Fei, J., Ding, J., Bakr, E. M., & Elhoseiny, M. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00839 · Cited by 3 · see citing works
“Key developments in this area include Flamingo [1], BLIP [29, 31], MiniGPT-4 [68], InstructBLIP [16], and LLaVA [35, 36], which advance AI’s ability to interpret and interact using vision and language.”
Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Models
Zamini, M., & Shukla, D. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00356 · Cited by 3 · see citing works
“These systems typically rely on a projector to map visual features into the language embedding space, enabling joint reasoning over text and images [2, 14, 21].”
Context-Infused Trajectories: Enhancing Context and Frame Consistency in Reasoning Video Object Segmentation
Zhuge, Y., Gong, S., Zhang, L., Xu, Q., Zhao, W., Zhan, J., & Lu, H. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3689427 · Cited by 4 · see citing works
Health system learning enables generalist neuroimaging models
Kondepudi, A., Rao, A. T., Zhao, C., Lyu, Y., Harake, E. S., Banerjee, S., Ogle, J. D., Joshi, R. S., Meißner, A.-K., Hou, X., et al. (2026) · Nature Medicine · doi:10.1038/s41591-026-04497-1 · Cited by 8 · see citing works
“To address this, our connector module first employs a Perceiver-style architecture that operates sequencewise (for example, on T1, T2 and FLAIR independently) 45,46 .”
Toward Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering
Wang, Z., Wan, W., Lao, Q., Chen, R., Lang, M., Wang, X., Gao, F., Wang, K., & Lin, L. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3651096 · Cited by 13 · see citing works
RAC3: Retrieval-Augmented Corner Case Comprehension for Autonomous Driving With Vision-Language Models
Wang, Y., Liu, Q., Fan, J., Hong, J., Chu, H., Tian, M., Gao, B., & Chen, H. (2026) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2026.3673470 · Cited by 6 · see citing works
“Such models have been effectively applied to critical areas, including autonomous driving, where precise scenario comprehension, especially comer case comprehension, is paramount for ensuring safety and functionality in complex environments [3], [4], [5], [6], [7], [8], [9].”
Pi-GPS: Enhancing Geometry Problem Solving by Unleashing the Power of Diagrammatic Information
Zhao, J., Zhang, T., Sun, J., Tian, M., & Hua, H. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00150 · Cited by 4 · see citing works
“Subsequent works [2, 7, 20, 34] expanded MLLMs by incorporating novel visual perception modules and hybrid vision encoders, enhancing their ability to address increasingly complex tasks.”
LeapVAD: A Leap in Autonomous Driving via Cognitive Perception and Dual-Process Thinking
Ma, Y., Wei, T., Zhong, N., Mei, J., Hu, T., Wen, L., Yang, X., Shi, B., & Liu, Y. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3626711 · Cited by 12 · see citing works
“Built on large language models (LLMs) such as LLaMA [11] and Vicuna [12], a wide range of VLMs [13], [14], [15], [16], [17], [18] has emerged, extending their capabilities to multimodal understanding.”
Rethinking Hallucinations: A Cognitive-Inspired Taxonomy and Comprehensive Survey in Large Language Models, Large Vision-Language Models, and Multimodal Large Language ModelsPreprint
Xia, S.-J., Zhang, H., Jiang, Y., Chen, X., Chen, Y., Li, Z., Wan, Z., & Sun, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5731427 · Cited by 3 · see citing works
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
Shukor, M., Aubakirova, D., Capuano, F., Kooijmans, P., Palma, S., Zouitine, A., Aractingi, M., Pascal, C., Russi, M., Marafioti, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2506.01844 · Cited by 3 · see citing works
“The success of text-based models has thus been extended to other modalities, sparking interest towards multi-modal vision-language (VLMs) (Alayrac et al., 2022; Chen et al., 2023; Huang et al., 2023; Liu et al., 2023b; Chen et al., 2024; Shukor et al., 2023b) and audio-language models (ALMs)…”
Chain-of-Detection: Enhancing Cross-Granularity Robotic Perception for Object Manipulation
Xu, T., Gao, H., Chen, C., et al. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3626567 · Cited by 2 · see citing works
“While vision–language models (VLMs) [10], [11], [12], [13], [14] have introduced a new paradigm in robotic detection and reasoning, and these methods predominantly focus on coarse-grained object detection, which is often insu ﬃ cient for applications demanding precise and detailed understanding of…”
Mitigating Inherent Bias of Answer Heuristic Based Frameworks in Knowledge-Based Visual Question Answering
Lu, J., Jiang, M., Kong, J., Zhuang, D., & Lu, M. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3645641 · Cited by 2 · see citing works
AVQACL++: Toward a Robust Framework and Benchmark for Audio-Visual Question Answering Continual Learning
Wu, K., Li, X., Li, X., et al. (2026) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2026.3662386 · Cited by 2 · see citing works
“With the emergence of powerful multimodal large language models (MLLMs) such as Flamingo [62], GPT-4V [63], and LLaVA [64], a natural question is whether our continual learning framework can be extended to work with such models.”
Contrastive context-aware emotion recognition: A multimodal framework for learning contextualized emotional representations
Tran, N., Le, T., Ta, P., Nguyen, H., & Nguyen, H. D. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133237 · Cited by 2 · see citing works
OpenUrban3D: Label-Free Open-Vocabulary Semantic Segmentation of Large-Scale Urban Point Clouds
Wang, C., Jing, K., ZHU, J., & Wang, D. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3670962 · Cited by 2 · see citing works
Fine-Tuning Vision-Language Models for Multimodal Polymer Property Prediction
Vuong, A., Van, M.-H., Verma, P., Zhao, C., & Wu, X. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402116 · Cited by 2 · see citing works
“2-Vision family [33] and the Flamingo/OpenFlamingo family [11], [12].”
Domain textual knowledge-enhanced few-shot utility tunnel video anomaly detection with multimodal large language models
Yin, B., Wang, S., Zhou, X.-L., & Wang, H. (2026) · Advanced Engineering Informatics · doi:10.1016/j.aei.2026.104594 · Cited by 3 · see citing works
Generative AI for multimodal content: a survey with empirical and experimental evaluations
Taha, K. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11525-6 · Cited by 4 · see citing works
Generative IoT: advancing IoT with generative AI and large language models
Firouzi, F., Ray, A., Farahani, B., Daneshmand, M., Song, J., Wu, S., & Chakrabarty, K. (2026) · Digital Communications and Networks · doi:10.1016/j.dcan.2026.03.010 · Cited by 3 · see citing works
Attention re-alignment in multimodal large language models via intermediate-layer guidance
Chen, Y., Wang, P., Qin, G., et al. (2026) · Scientific Reports · doi:10.1038/s41598-026-44935-1 · Cited by 2 · see citing works
Towards generalizable pathology reports via a multimodal LLM with the multicenter in-context learning
Li, Y., Lin, Z.-H., Zhang, Q.-X., Ding, X., Yang, H., Pi, L.-J., Yuan, W., Wen, Y.-Q., Guo, L., Zhang, Q.-L., et al. (2026) · Medical Image Anal. · doi:10.1016/j.media.2026.104060 · Cited by 2 · see citing works
Embodiment in multimodal large language models
Kadambi, A., Aziz‐Zadeh, L., Damásio, A. R., Iacoboni, M., & Narayanan, S. (2026) · Neuron · doi:10.1016/j.neuron.2026.03.004 · Cited by 2 · see citing works
“This limits their ability to dynamically integrate information across modalities to adapt to new contexts, unlike humans [1, 84].”
Influential
Generative Artificial Intelligence Models: A Survey
Almuammar, M., Alsultan, K., & Altukhaim, Y. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11546-1 · Cited by 6 · see citing works
“Finally, progress in multimodal integration and sustainable model design will shape future Transformer research (Alayrac et al. 2022 ).”
Generative adversarial fault-tolerance for robust multimodal sensor fusion in industrial systems
Altinses, D., & Schwung, A. (2026) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2026.114225 · Cited by 3 · see citing works
GroundCap: A visually grounded image captioning dataset with object and action identification
Oliveira, D. A. P., Teodoro, L., & de Matos, D. M. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113703 · Cited by 2 · see citing works
Three-dimensional primitive prediction with building embedded Swin Transformer for urban turbulence
Park, J., Kim, H., Panagiotis Tsiotras, B., & Lee, S. (2026) · Physics of Fluids · doi:10.1063/5.0324405 · Cited by 2 · see citing works
Medical multimodal large language models: A survey
Xiao, H., Hui, N., Xu, Y., Li, Z., & Peng, J. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104386 · Cited by 2 · see citing works
INNFusion: A Diffusion-Based Blind Image Super Resolution Scheme Using Reversible Degradation Process With Invertible Neural Networks
Poudineh, M., Esmaeilzehi, A., & Ahmad, M. O. (2026) · IEEE Transactions on Image Processing · doi:10.1109/tip.2026.3685457 · Cited by 2 · see citing works

Authors: Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., & Simonyan, K.
ORCID: Alayrac, Jean-Baptiste
Miech, Antoine
Lenc, Karel
Vinyals, Oriol
Zisserman, Andrew
Venue: Advances in Neural Information Processing Systems 35
Publisher: Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published: 2022
Pages: 23716-23736
ISSN: 1049-5258 (ISSN-L)
DOI: 10.52202/068431-1723
OpenAlex: W7133196460
Cited by: 966
References: 154
Citations by year: 2026: 847 · 2025: 107 · 2024: 3 · 2023: 7 · 2022: 2
FWCI: 15.48
Type: conference paper
Language: en
Open access: closed
Locations: neural information processing systems (publishedVersion)
Oxford University Research Archive (ORA) (University of Oxford) (publishedVersion)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords: vision-language models, few-shot learning, image-text models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Flamingo: a visual language model for few-shot learning

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Record

About these checks