Cited by 966Domain Adaptation and Few-Shot LearningMultimodal Machine Learning ApplicationsAdvanced Neural Network Applications

The DOI matches its Crossref record; 154 of the 154 references it cites were found in a registry or an index.

Flamingo: A Visual Language Model for Few-Shot Learning

Alayrac, Jean-Baptiste, Donahue, Jeff, Luc, Pauline, Miech, Antoine, Barr, Iain, Hasson, Yana, Lenc, Karel, Mensch, Arthur, Millican, Katherine, Reynolds, Malcolm, Ring, Roman, Rutherford, Eliza, Cabi, Serkan, Han, Tengda, Gong, Zhitao, Samangooei, Sina, Monteiro, Marianne, Menick, Jacob L, Borgeaud, Sebastian, Brock, Andy, Nematzadeh, Aida, Sharifzadeh, Sahand, Binkowski, Mikolaj, Barreira, Ricardo, Vinyals, Oriol, Zisserman, Andrew, Simonyan, Karén·2022·Advances in Neural Information Processing Systems 35

DOI 10.52202/068431-1723

Open this paper in CoChat — free account

Abstract

Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer, captioning tasks, which evaluate the ability to describe a scene or an event, and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.

Integrity signals

Registry

Crossref · 10.52202/068431-1723

Cites

154 of 154 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“Language modelling has recently made substantial progress following the introduction of Transformers [115].”
BackgroundMethodInfluential
Long Short-Term Memory
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“To ensure that at initialization, the conditioned model yields the same results as the original language model, we use a tanh-gating mechanism [41].”
Method
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundMethodInfluential
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 2×
“Im ag e ImageNet-1k [94] 3 Object classification Val Top-1 acc.”
Background
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 12×
“We evaluate the ability of our models to rapidly adapt to new tasks using in-context learning , analogously to GPT-3 [11], by interleaving support example pairs in the form of ( 𝑖𝑚𝑎𝑔𝑒, 𝑡𝑒𝑥𝑡 ) or ( 𝑣𝑖𝑑𝑒𝑜, 𝑡𝑒𝑥𝑡 ) , followed by the query visual input, to build a prompt (details in Method FT Shot OKVQA ( I )”
BackgroundMethodResultInfluential
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · cited 2×
“[13] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.”
BackgroundMethod
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 12×
“[85] we use “prompt-ensembling” in which we embed multiple texts using templates such as “A photo of a {class_name}” and average the resulting embedding.”
BackgroundMethodInfluential
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · cited 3×
“To measure the importance of text pretraining, we compare the performance of using a frozen decoder-only Transformer either pretrained on MassiveText (our main model) or pretrained on the C4 dataset [87] (row (iv) of Table 10).”
BackgroundMethod
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 2×
“In the language space, an important line of work has focused on establishing scaling laws for language models [42, 53].”
Background
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512 · cited 2×
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
BackgroundMethod
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 2×
“[116] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.”
BackgroundResultSelf-citation
Prototypical Networks for Few-shot LearningPreprint
Snell, J., Swersky, K., & Zemel, R. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.05175 · Cited by 6,245 · cited 2×
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
BackgroundMethod
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · cited 2×
“The paradigm of first pretraining on a vast amount of data followed by an adaptation on a downstream task has become standard [11, 23, 32, 44, 52, 75, 87, 108].”
Background
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 2×
“Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
BackgroundResultSelf-citation
Matching Networks for One Shot LearningPreprint
Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., & Wierstra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.04080 · Cited by 4,975 · cited 2×
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
BackgroundMethodSelf-citation
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 2×
“The paradigm of first pretraining on a vast amount of data followed by an adaptation on a downstream task has become standard [11, 23, 32, 44, 52, 75, 87, 108].”
Background
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770 · cited 2×
“One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
Background
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684 · cited 1×
“[105] Emma Strubell, Ananya Ganesh, and Andrew McCallum.”
Background
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · cited 1×
“Note that the frozen LM was trained with the GeLU activation [39], while the remaining trainable transformer layers use the Squared ReLU activation [104], which we found to outperform GeLU.”
Method
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · cited 2×
“[56] Brian Lester, Rami Al-Rfou, and Noah Constant.”
Background
One-shot learning of object categories
Fei-Fei, L., Fergus, R., & Perona, P. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.79 · Cited by 3,029
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · cited 1×
“The paradigm of first pretraining on a vast amount of data followed by an adaptation on a downstream task has become standard [11, 23, 32, 44, 52, 75, 87, 108].”
Background
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 3×
“Note that as explained in our Model Card [77] in Appendix E, this model was developed for research purposes only and should not be used in specific applications before proper risk analyses are conducted and mitigation strategies are explored.”
BackgroundMethodInfluential
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 2×
“To achieve this, Flamingo takes inspiration from recent work on large language models (LMs) which are good few-shot learners [11, 18, 42, 86].”
Background
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 2×
“– Dialogue 1: Available from DALL·E 2 [89].”
Background
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · cited 1×
“These include adding small adapter modules [43], fine-tuning a small part of the LM [141], showing in-context examples in the prompt [11], or optimizing the prompt [56, 60] through gradient descent.”
Background
Datasheets for datasets
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · cited 3×
“[30] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford.”
BackgroundInfluential
From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · cited 3×
“[139] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.”
Background
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · cited 1×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Parameter-efficient transfer learning for NLP
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Gelly, S. (2019) · arXiv · doi:10.48550/arxiv.1902.00751 · Cited by 2,270 · cited 4×
“In contrast, gradient-based few-shot learning approaches [43] have constant complexity with respect to the number of shots during inference.”
BackgroundInfluential
Self-Critical Sequence Training for Image Captioning
Rennie, S. J., Marcheret, E., Mroueh, Y., Ross, J., & Goel, V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.131 · Cited by 2,231 · cited 1×
“In some cases our results likely trail the state of the art due in part to the fact that we simply optimise log-likelihood and do not make use of common task-specific metric optimisation tricks, such as CIDEr optimisation [64, 90] for COCO captioning, and fine-tuning on dense annotations for VisDial [79].”
Method
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109 · cited 2×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundMethod
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · cited 1×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · cited 7×
“For our image and text pairs we first leverage the ALIGN [50] dataset, composed of 1.”
BackgroundMethodInfluential
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879 · cited 2×
“[98] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.”
Background
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X. P., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Li, D., Wei, F., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,791 · cited 1×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation
Li, J., Li, D., Xiong, C., & Hoi, S. (2022) · arXiv · doi:10.48550/arxiv.2201.12086 · Cited by 1,741 · cited 1×
“Concurrent works [17, 58, 119, 124, 154] also propose to formulate numerous vision tasks as text generation problems.”
Background
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · cited 1×
“Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
Background
Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition
Bridle, J. S. (1990) · Springer eBooks · doi:10.1007/978-3-642-76153-9_28 · Cited by 1,278 · cited 1×
“We use a learnable temperature parameter in the final log-softmax layer [9].”
Method
VideoBERT: A Joint Model for Video and Language Representation Learning
Sun, C., Myers, A., Vondrick, C., Murphy, K. P., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,255 · cited 1×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Exploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02410 · Cited by 1,238 · cited 2×
“The paradigm of first pretraining on a vast amount of data followed by an adaptation on a downstream task has become standard [11, 23, 32, 44, 52, 75, 87, 108].”
BackgroundSelf-citation
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 2×
“[125] Jason Wei, Maarten Bosma, Vincent Y.”
Background
Attention on Attention for Image Captioning
Huang, L., Wang, W., Chen, J., & Wei, X.-Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00473 · Cited by 1,092
Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty, S., Xiong, C., & Hoi, S. (2021) · arXiv · doi:10.48550/arxiv.2107.07651 · Cited by 1,078 · cited 1×
“Another family of vision-language models is based on contrastive learning [2, 5, 49, 50, 57, 74, 82, 85, 138, 140, 146].”
Background
VinVL: Revisiting Visual Representations in Vision-Language Models
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., & Gao, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00553 · Cited by 1,015
Categorization and Naming in Children: Problems of Induction
Markman, E. M. (1989) · Choice Reviews Online · doi:10.5860/choice.27-2985 · Cited by 992 · cited 1×
“One key aspect of intelligence is the ability to quickly learn to perform a new task given a short instruction [33, 70].”
Background
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00272 · Cited by 986
Self-citation
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952
Rethinking Few-Shot Image Classification: A Good Embedding is All You Need?
Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, J. B., & Isola, P. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58568-6_16 · Cited by 952 · cited 1×
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
Method
Towards VQA Models That Can Read
Singh, A., Natarajan, V., Shah, M., Jiang, Y. L., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00851 · Cited by 951
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950 · cited 2×
“[106] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.”
Background
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · cited 13×
“• The Flamingo-9B model builds on top of a 7B frozen language model from [42].”
BackgroundMethodInfluentialSelf-citation
ClipCap: CLIP Prefix for Image CaptioningPreprint
Mokady, R., Hertz, A., & Bermano, A. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09734 · Cited by 929 · cited 2×
“One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
Background
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · cited 2×
“[99] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.”
BackgroundMethod
Scaling Vision Transformers
Zhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891
Unified Vision-Language Pre-Training for Image Captioning and VQA
Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J., & Gao, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7005 · Cited by 864
Meta-learning with differentiable closed-form solversPreprint
Bertinetto, L., Henriques, J. F., Torr, P. H. S., & Vedaldi, A. (2018) · Oxford University Research Archive (ORA) (University of Oxford) · doi:10.48550/arxiv.1805.08136 · Cited by 857 · cited 1×
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
Method
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00331 · Cited by 845 · cited 5×
“Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
BackgroundInfluential
Low-Shot Visual Recognition by Shrinking and Hallucinating Features
Hariharan, B., & Girshick, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.328 · Cited by 843
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 4×
“[111], adequately conditioned or fine-tuned Flamingo models could be used for filtering purposes of toxic or harmful samples in the training data.”
BackgroundResultInfluential
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
Ben Zaken, E., Goldberg, Y., & Ravfogel, S. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.1 · Cited by 765 · cited 2×
“[141] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg.”
Background
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · cited 4×
“[76] show that the mapping from input to output generally has limited impact on few-shot performance, as opposed to specifying the overall format of the examples.”
BackgroundInfluential
Towards Automatic Learning of Procedures From Web Instructional Videos
Zhou, L., Xu, C., & Corso, J. J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12342 · Cited by 722
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P. K., Ding, N., & Soricut, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00356 · Cited by 701 · cited 1×
“To alleviate this lack of data, numerous works [14, 50, 98, 110] automatically scrape readily available paired vision-text data.”
SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionPreprint
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., & Cao, Y. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.10904 · Cited by 701 · cited 4×
“1 Restricted SotA† [140] [140] [124] [153] - - [51] [79] [79] [132] [137] [84] [62] 81.”
BackgroundInfluential
FLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01519 · Cited by 678 · cited 2×
“[101] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.”
Background
End-to-End Learning of Visual Representations From Uncurated Instructional Videos
Miech, A., Alayrac, J.-B., Smaira, L., Laptev, I., Šivic, J., & Zisserman, A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00990 · Cited by 646 · cited 2×
“[74] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman.”
BackgroundSelf-citation
VizWiz Grand Challenge: Answering Visual Questions from Blind People
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., & Bigham, J. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00380 · Cited by 621
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · cited 2×
“In this direction, recent work has shown that non-causal masked language modeling adaptation [120] followed by multitask fine-tuning [95, 125, 131] can efficiently improve the zero-shot performance of causal decoder-only language models.”
Background
FILIP: Fine-grained Interactive Language-Image Pre-TrainingPreprint
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., & Xu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07783 · Cited by 595 · cited 2×
“[138] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.”
Background
VQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570 · cited 7×
“Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
BackgroundMethodInfluential
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
Wang, X., Wu, J., Chen, J., Li, L., Wang, Y., & Wang, W. Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00468 · Cited by 536 · cited 1×
“Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
Background
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · cited 1×
“However, in-context learning is known to be highly sensitive to various aspects of the demonstrations [80, 148], and its inference compute cost and absolute performance scale poorly with the number of shots beyond this low-data regime.”
Background
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492
Ethical and social risks of harm from Language Models
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04359 · Cited by 477 · cited 4×
“As such Flamingo is exposed to the same risks of large language models: it can output potentially offensive language, propagate social biases and stereotypes, as well as leaking private information [126].”
BackgroundInfluential
Video Question Answering via Gradually Refined Attention over Appearance and Motion
Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., & Zhuang, Y. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123427 · Cited by 471 · cited 5×
“[130] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang.”
BackgroundInfluential
Visual Dialog
Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.121 · Cited by 445
Gender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · cited 2×
“7) in terms of gender bias on the Winogender dataset [93] which demonstrate that even though this model is less biased towards gender than previous models [86], gender biases are still present.”
Background
ActBERT: Learning Global-Local Video-Text Representations
Zhu, L., & Yang, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00877 · Cited by 425 · cited 1×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
VLMo: Unified Vision-Language Pre-Training with Mixture-Of-Modality-Experts
Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O. K., Aggarwal, K., Som, S., Piao, S., & Wei, F. (2022) · neural information processing systems · doi:10.52202/068431-2384 · Cited by 420 · cited 1×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Perceiver: General Perception with Iterative Attention
Jaegle, A., Gimeno, F., Brock, A., Zisserman, A., Vinyals, O., & Carreira, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03206 · Cited by 414 · cited 10×
“Perceiver Resampler: from varying-size large feature maps to few visual tokens.”
BackgroundMethodInfluentialSelf-citation
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
Li, L., Chen, Y.-C., Cheng, Y., Gan, Z., Yu, L., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.161 · Cited by 408 · cited 1×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Women Also Snowboard: Overcoming Bias in Captioning Models
Hendricks, L. A., Burns, K., Saenko, K., Darrell, T., & Rohrbach, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_47 · Cited by 404 · cited 1×
“Its ability to additionally handle visual inputs poses specific risks such as gender and racial biases relating to the contents of the input images, similar to a number of visual recognition systems [12, 21, 37, 97, 147].”
Background
LiT: Zero-Shot Transfer with Locked-image text Tuning
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01759 · Cited by 402 · cited 3×
“Finally, state-of-the-art zero-shot models on ImageNet such as BASIC [82] and LiT [146] are particularly optimized on classification tasks as they are trained on JFT-3B [145], a dataset with images and labels.”
BackgroundMethod
Learning feed-forward one-shot learnersPreprint
Bertinetto, L., Henriques, J. F., Valmadre, J., Torr, P. H. S., & Vedaldi, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05233 · Cited by 398 · cited 1×
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
Method
NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions
Xiao, J., Shang, X., Yao, A., & Chua, T. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00965 · Cited by 397 · cited 1×
“Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
Background
High-Performance Large-Scale Image Recognition Without NormalizationPreprint
Brock, A., De, S., Smith, S. L., & Simonyan, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.06171 · Cited by 394 · cited 2×
“Our vision encoder is a pretrained and frozen NormalizerFree ResNet (NFNet) [10] – we use the F6 model.”
MethodSelf-citation
Florence: A New Foundation Model for Computer Vision
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.11432 · Cited by 389 · cited 5×
“[140] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengchuan Zhang.”
BackgroundMethodInfluential
VirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01101 · Cited by 376 · cited 1×
“In our ablation studies (Section 3.3), we compare the proposed GATED XATTN - DENSE layers against recent alternatives [22, 68] and explore the effect of how frequently these additional layers are inserted to trade off between efficiency and expressivity.”
MethodInfluential
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA
Yang, Z., Gan, Z., Wang, J., Hu, X., Lu, Y., Liu, Z., & Wang, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20215 · Cited by 375 · cited 1×
“One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
Background
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · cited 3×
“[81] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving.”
BackgroundResultSelf-citation
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358 · cited 1×
“We ablate replacing our image-text pairs (ITP) by the publicly available LAION-400M dataset [96], which leads to a slight degradation in performance.”
Method
Large-Scale Adversarial Training for Vision-and-Language Representation LearningPreprint
Gan, Z., Chen, Y.-C., Li, L., Zhu, C., Cheng, Y., & Liu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06195 · Cited by 358 · cited 1×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Multimodal Few-Shot Learning with Frozen Language Models
Tsimpoukelli, M., Menick, J., Cabi, S., Eslami, S. M. A., Vinyals, O., & Hill, F. (2021) · arXiv · doi:10.48550/arxiv.2106.13884 · Cited by 355 · cited 3×
“One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
BackgroundSelf-citation
The hateful memes challenge: Detecting hate speech in multimodal memes
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D. (2020) · arXiv · doi:10.48550/arxiv.2005.04790 · Cited by 352 · cited 5×
“VisDial [20], a visual dialogue benchmark; HatefulMemes [54] a vision and text classification benchmark; NextQA [129] which specially focuses on causality and temporal relation; STAR [128], a multiple-choice question answering task; and RareAct [73], a benchmark measuring compositionality in action recognition.”
BackgroundMethodInfluential
A Short Note on the Kinetics-700-2020 Human Action Dataset
Smaira, L., Carreira, J., Noland, E., Clancy, E., Wu, A., & Zisserman, A. (2020) · arXiv · doi:10.48550/arxiv.2010.10864 · Cited by 334
Self-citation
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks
Sung, Y.-L., Cho, J., & Bansal, M. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00516 · Cited by 329
Multiview Transformers for Video Recognition
Yan, S., Xiong, X., Arnab, A., Lu, Z., Zhang, M., Sun, C., & Schmid, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00333 · Cited by 312 · cited 1×
“[134] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid.”
Background
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.05482 · Cited by 307 · cited 1×
“[127] Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael GontijoLopes, Ari S.”
Background
Unifying Vision-and-Language Tasks via Text GenerationPreprint
Cho, J., Lei, J., Tan, H. X., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.02779 · Cited by 291 · cited 4×
“We show in row (ii) the importance of our gradient accumulation strategy compared to using round-robin updates [17].”
BackgroundMethodResultInfluential
Meta-Learning Probabilistic Inference for Prediction
Gordon, J., Bronskill, J. F., Bauer, M., et al. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1805.09921 · Cited by 291 · cited 2×
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
BackgroundMethod
Pix2seq: A Language Modeling Framework for Object DetectionPreprint
Chen, T., Saxena, S., Li, L., Fleet, D. J., & Hinton, G. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10852 · Cited by 272
Scaling Up Vision-Language Pretraining for Image Captioning
Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01745 · Cited by 261 · cited 1×
“Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
Result
Self-Supervised MultiModal Versatile NetworksPreprint
Alayrac, J.-B., Recasens, A., Schneider, R. G., Arandjelović, R., Ramapuram, J., Fauw, J. D., Smaira, L., Dieleman, S., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16228 · Cited by 258 · cited 1×
“Another family of vision-language models is based on contrastive learning [2, 5, 49, 50, 57, 74, 82, 85, 138, 140, 146].”
BackgroundSelf-citation
CrossTransformers: spatially-aware few-shot transferPreprint
Doersch, C., Gupta, A., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.11498 · Cited by 255 · cited 2×
“[24] Carl Doersch, Ankush Gupta, and Andrew Zisserman.”
BackgroundMethodSelf-citation
MERLOT: Multimodal Neural Script Knowledge ModelsPreprint
Zellers, R., Lu, X., Hessel, J., Yu, Y., Park, J. S., Cao, J., Farhadi, A., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02636 · Cited by 239 · cited 2×
“[142] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.”
Background
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · cited 8×
“In the short term, effective approaches include relying on prompting to mitigate any biases and harmful outputs [86].”
BackgroundMethodInfluentialSelf-citation
Just Ask: Learning to Answer Questions from Millions of Narrated Videos
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00171 · Cited by 232
Self-citation
Fixing the train-test resolution discrepancyPreprint
Touvron, H., Vedaldi, A., Douze, M., & Jeǵou, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06423 · Cited by 228 · cited 2×
“[113] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou.”
Background
True Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226 · cited 1×
“However, in-context learning is known to be highly sensitive to various aspects of the demonstrations [80, 148], and its inference compute cost and absolute performance scale poorly with the number of shots beyond this low-data regime.”
Background
ReZero is All You Need: Fast Convergence at Large DepthPreprint
Bachlechner, T. C., Majumder, B. P., Mao, H. H., Cottrell, G. W., & McAuley, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04887 · Cited by 226 · cited 2×
“This multiplies the output of a newly added layer by tanh(α) before adding it to the input representation from the residual connection, where α is a layer-specific learnable scalar initialized to 0 [4].”
Background
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning
Chen, J., Guo, H., Yi, K., Li, B., & Elhoseiny, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01750 · Cited by 222
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token ModelingPreprint
Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12681 · Cited by 212 · cited 2×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
Background
Transformer Language Models without Positional Encodings Still Learn Positional Information
Haviv, A., Ram, O., Press, O., Izsak, P., & Levy, O. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2203.16634 · Cited by 210 · cited 2×
“[36] Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy.”
Background
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language
Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.00598 · Cited by 205 · cited 2×
“One recent line of work [26, 68, 78, 114, 136, 144] proposes to freeze the pretrained LM weights to prevent catastrophic forgetting [71].”
Background
All in One: Exploring Unified Video-Language Pre-Training
Wang, J., Ge, Y., Yan, R., Ge, Y., Lin, K. Q., Tsutsui, S., Lin, X., Cai, G., Wu, J., Shan, Y., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00638 · Cited by 199
UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and GenerationPreprint
Luo, H., Ji, L., Shi, B., Huang, H., Duan, N., Li, T., Li, J., Bharti, T., & Zhou, M. Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.06353 · Cited by 187 · cited 1×
“Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
Result
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound
Zellers, R., Lu, J., Lu, X., Yu, Y., Zhao, Y., Salehi, M., Kusupati, A., Hessel, J., Farhadi, A., & Choi, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01589 · Cited by 179 · cited 2×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundMethod
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Press, O., Smith, N. A., & Lewis, M. (2021) · arXiv · doi:10.48550/arxiv.2108.12409 · Cited by 161
Does Object Recognition Work for Everyone?Preprint
DeVries, T., Misra, I., Wang, C., & van der Maaten, L. J. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02659 · Cited by 141 · cited 3×
“However, current automatic vision system performance has been reported to vary with race, gender or when applied across different demographics and geographic regions [12, 21, 97].”
Background
Doing more with less: meta-reasoning and meta-learning in humans and machines
Griffiths, T. L., Callaway, F., Chang, M. B., Grant, E., Krueger, P. M., & Lieder, F. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2019.01.005 · Cited by 139 · cited 1×
“One key aspect of intelligence is the ability to quickly learn to perform a new task given a short instruction [33, 70].”
Background
STAR: A Benchmark for Situated Reasoning in Real-World VideosPreprint
Wu, B., Yu, S., Chen, Z., Tenenbaum, J. B., & Gan, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2405.09711 · Cited by 135
Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive ProcessesPreprint
Requeima, J., Gordon, J. A. R., Bronskill, J. F., Nowozin, S., & Turner, R. E. (2019) · arXiv (Cornell University) · doi:10.17863/cam.47574 · Cited by 134 · cited 2×
“In this paper, we take inspiration from the in-context [11] few-shot learning technique instead of more involved few-shot learning approaches based on metric learning [24, 103, 112, 117] or meta-learning [6, 7, 27, 31, 91, 155].”
BackgroundMethod
KAT: A Knowledge Augmented Transformer for Vision-and-Language
Gui, L., Wang, B., Huang, Q., Hauptmann, A., Bisk, Y., & Gao, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.70 · Cited by 130 · cited 1×
“[34] Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Hauptmann, Yonatan Bisk, and Jianfeng Gao.”
Background
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
Yang, Z., Lu, Y., Wang, J., Yin, X. P., Florencio, D. A. F., Wang, L., Zhang, C., Zhang, L., & Luo, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00864 · Cited by 130
A Multimodal Framework for the Detection of Hateful MemesPreprint
Lippe, P., Holla, N., Chandra, S., Rajamanickam, S., Antoniou, G., Shutova, E., & Yannakoudakis, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.12871 · Cited by 116 · cited 2×
“1 Restricted SotA† [140] [140] [124] [153] - - [51] [79] [79] [132] [137] [84] [62] 81.”
Background
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Dai, W., Hou, L., Shang, L.-F., et al. (2022) · Findings · doi:10.48550/arxiv.2203.06386 · Cited by 115 · cited 2×
“Similar to our work are VLMs able to generate text in an autoregressive manner [19, 25, 45, 67, 116].”
BackgroundResult
Large-Scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline
Murahari, V., Batra, D., Parikh, D., & Das, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_20 · Cited by 109
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
Zhu, X., Zhu, J., Li, H., Wu, X., Li, H., Wang, X., & Dai, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01630 · Cited by 108 · cited 1×
“Concurrent works [17, 58, 119, 124, 154] also propose to formulate numerous vision tasks as text generation problems.”
Background
VD-BERT: A Unified Vision and Dialog Transformer with BERT
Wang, Y., Joty, S., Lyu, M. R.-T., King, I., Xiong, C., & Hoi, S. C. H. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.269 · Cited by 103 · cited 3×
“[123] Yue Wang, Shafiq Joty, Michael Lyu, Irwin King, Caiming Xiong, and Steven Hoi.”
Background
Diagnosing Gender Bias in Image Recognition Systems
Schwemmer, C., Knight, C. R., Bello-Pardo, E. D., Oklobdzija, S. N., Schoonvelde, M., & Lockhart, J. W. (2020) · Socius Sociological Research for a Dynamic World · doi:10.1177/2378023120967171 · Cited by 101 · cited 3×
“[97] Carsten Schwemmer, Carly Knight, Emily D.”
Background
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers
Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B., & Nematzadeh, A. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00385 · Cited by 100 · cited 2×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundSelf-citation
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
Xu, H., Ghosh, G., Huang, P.-Y., Arora, P., Aminzadeh, M., Feichtenhofer, C., Metze, F., & Zettlemoyer, L. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.370 · Cited by 94 · cited 1×
“Others have explored visually-conditioned language generation [17, 114, 119, 124, 132] but have not yet shown good performance in low-data regimes.”
Background
Primer: Searching for Efficient Transformers for Language ModelingPreprint
So, D. R., Mańke, W., Liu, H., Dai, Z., Shazeer, N., & Le, Q. V. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08668 · Cited by 91 · cited 2×
“[104] David R So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V.”
BackgroundMethod
Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning SolutionPreprint
Zhu, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.08290 · Cited by 91
Combined scaling for zero-shot transfer learning
Pham, H., Dai, Z., Ghiasi, G., Kawaguchi, K., Liu, H., Yu, A. W., Yu, J., Chen, Y.-T., Luong, M.-T., Wu, Y., et al. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126658 · Cited by 85 · cited 5×
“Second, the classification performance of Flamingo lags behind that of state-of-the-art contrastive models [82, 85].”
BackgroundMethodInfluential
Understanding and Evaluating Racial Biases in Image Captioning
Zhao, D., Wang, A., & Russakovsky, O. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01456 · Cited by 85 · cited 1×
“Its ability to additionally handle visual inputs poses specific risks such as gender and racial biases relating to the contents of the input images, similar to a number of visual recognition systems [12, 21, 37, 97, 147].”
Background
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Bain, M., Nagrani, A., Varol, G., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00175 · Cited by 76 · cited 2×
“Another family of vision-language models is based on contrastive learning [2, 5, 49, 50, 57, 74, 82, 85, 138, 140, 146].”
BackgroundSelf-citation
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?Preprint
Wang, T., Roberts, A., Hesslow, D., Scao, T. L., Chung, H. W., Beltagy, I., Launay, J., & Raffel, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.05832 · Cited by 67 · cited 2×
“[120] Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel.”
Background
MURAL: Multimodal, Multitask Retrieval Across LanguagesPreprint
Jain, A., Guo, M., Srinivasan, K., Chen, T., Kudugunta, S., Jia, C., Yang, Y., & Baldridge, J. (2021) · arXiv.org · doi:10.48550/arxiv.2109.05125 · Cited by 65 · cited 2×
“[49] Aashi Jain, Mandy Guo, Krishna Srinivasan, Ting Chen, Sneha Kudugunta, Chao Jia, Yinfei Yang, and Jason Baldridge.”
Background
ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization
Xu, H., Chen, Y., Du, Y., Shao, N., Yanggang, W., Li, H., & Yang, Z. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.312 · Cited by 62 · cited 2×
“[131] Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang.”
Background
Teaching language models to support answers with verified quotes
Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.11147 · Cited by 53 · cited 3×
“[72] Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nat McAleese.”
BackgroundResultSelf-citation
UFO: A UniFied TransfOrmer for Vision-Language Representation LearningPreprint
Wang, J., Hu, X., Gan, Z., Yang, Z., Dai, X., Liu, Z., Lu, Y., & Wang, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10023 · Cited by 50 · cited 2×
“In particular, BERT [23] inspired a large body of vision-language work [16, 28, 29, 38, 59, 61, 66, 101, 106, 107, 109, 118, 121, 142, 143, 151].”
BackgroundMethod
MAGMA – Multimodal Augmentation of Generative Models through Adapter-based Finetuning
Eichenberg, C., Black, S., Weinbach, S., Parcalabescu, L., & Frank, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.179 · Cited by 49 · cited 2×
“[26] Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank.”
Background
CM3: A Causal Masked Multimodal Model of the Internet
Aghajanyan, A., Huang, B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.07520 · Cited by 42 · cited 3×
“We emphasize few-shot learning and vision tasks while CM3 [1] primarily evaluates on language-only benchmarks in a zero-shot or fine-tuned setup.”
BackgroundInfluential
Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs
Islam, M. A., Kowal, M. D., Jia, S., Derpanis, K. G., & Bruce, N. D. B. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00083 · Cited by 36
ORBIT: A Real-World Few-Shot Dataset for Teachable Object Recognition
Massiceti, D., Zintgraf, L., Bronskill, J., Theodorou, L., Harris, M. T., Cutrell, E., Morrison, C., Hofmann, K., & Stumpf, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01064 · Cited by 33
Achieving Human Parity on Visual Question Answering
Yan, M., Xu, H., Li, C., Tian, J., Bi, B., Wang, W., Xu, X., Zhang, J., Huang, S., Huang, F., et al. (2022) · ACM Transactions on Information Systems · doi:10.1145/3572833 · Cited by 28 · cited 5×
“6 Unrestricted SotA [133] [133] [119] [153] [65] [65] - - [123] - - - [152]”
BackgroundInfluential
RareAct: A video dataset of unusual interactionsPreprint
Miech, A., Alayrac, J.-B., Ivan Laptev, A., Šivic, J., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01018 · Cited by 14 · cited 4×
“Note: We omit RareAct, our 16th benchmark, as it is a zero-shot benchmark with no available fine-tuned results to compare to.”
BackgroundMethodInfluentialSelf-citation
Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model
Qiao, Y., Chen, H., Wang, J., Zhao, S.-S., Chen, Y., Ye, X., Li, Z., Qi, X., Gao, P., & Xie, G.-T. (2021) · arXiv.org · doi:10.48550/arxiv.2106.15332 · Cited by 9 · cited 3×
“1 Restricted SotA† [140] [140] [124] [153] - - [51] [79] [79] [132] [137] [84] [62] 81.”
Background

Sorted by citations

Foundation models for generalist medical artificial intelligence
Moor, M., Banerjee, O., Abad, Z. S. H., Krumholz, H. M., Leskovec, J., Topol, E. J., & Rajpurkar, P. (2023) · Nature · doi:10.1038/s41586-023-05881-4 · Cited by 2,028
Scientific discovery in the age of artificial intelligence.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06221-2 · Cited by 1,823
A Survey on In-context Learning
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.64 · Cited by 649
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429
Talking about Large Language Models
Shanahan, M. (2024) · Communications of the ACM · doi:10.1145/3624724 · Cited by 381

See all 966

Authors: Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J. L., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., & Simonyan, K.
ORCID: Alayrac, Jean-Baptiste
Miech, Antoine
Lenc, Karel
Vinyals, Oriol
Zisserman, Andrew
Venue: Advances in Neural Information Processing Systems 35
Publisher: Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published: 2022
Pages: 23716-23736
ISSN: 1049-5258 (ISSN-L)
DOI: 10.52202/068431-1723
OpenAlex: W7133196460
Cited by: 966
References: 154
Citations by year: 2026: 847 · 2025: 107 · 2024: 3 · 2023: 7 · 2022: 2
FWCI: 15.48
Type: conference paper
Language: en
Open access: closed
Locations: neural information processing systems (publishedVersion)
Oxford University Research Archive (ORA) (University of Oxford) (publishedVersion)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords: vision-language models, few-shot learning, image-text models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Flamingo: A Visual Language Model for Few-Shot Learning

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Metadata

Record

About these checks