The DOI matches its Crossref record; 115 of the 115 references it cites were found in a registry or an index.
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
Open accessAbstract
We launch EVA, a vision-centric foundation model to Explore the limits of Visual representation at scAle using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVIS dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can greatly stabilize the training and outperform the training from scratch counterpart with much fewer samples and less compute, providing a new direction for scaling up and accelerating the costly training of multi-modal foundation models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“With further scaling on compute, data, and model sizes, PLMs have led to not only continuous performance improvements [49, 71, 72], but also a surprising emergence of in-context learning capability [9, 25, 101, 102].”
BackgroundLiu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01167 · Cited by 8,494
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1×
“Recently, there are a few trials leveraging the semantic information from image-image or image-text contrastive learning [13, 22, 73] for MIM pretraining [43, 106, 124], which perform fairly well in vision downstream tasks.”
BackgroundUcf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
Masked-attention Mask Transformer for Universal Image Segmentation
Cheng, B., Misra, I., Schwing, A. G., Kirillov, A., & Girdhar, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00135 · Cited by 3,565
Swin Transformer V2: Scaling Up Capacity and Resolution
Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.09883 · Cited by 2,679
Self-citationThe Kinetics Human Action Video Dataset
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 2,593
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Self-citationPhotorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.11487 · Cited by 2,528
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 2×
“Meanwhile, CLIP features also play a central role in representation leaning [66, 98], AI generated content [74, 79, 80] and large dataset filtering [10, 81, 82], etc.”
BackgroundMethodSelf-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505
Swin Transformer V2: Scaling Up Capacity and Resolution
Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01170 · Cited by 2,484
Self-citationSelf-Training With Noisy Student Improves ImageNet Classification
Xie, Q., Luong, M.-T., Hovy, E., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01070 · Cited by 2,393
Soft-NMS — Improving Object Detection with One Line of Code
Bodla, N., Singh, B., Chellappa, R., & Davis, L. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.593 · Cited by 2,274 · cited 1×
“For COCO, Soft-NMS [8] is also applied.”
Deep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977
Cascade R-CNN: High Quality Object Detection and Instance Segmentation
Cai, Z., & Vasconcelos, N. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2956516 · Cited by 1,935
Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning
Sharma, P., Ding, N., Goodman, S., & Soricut, R. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1238 · Cited by 1,879
Semantic Understanding of Scenes Through the ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Xiao, T., Fidler, S., Barriuso, A., & Torralba, A. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1140-0 · Cited by 1,831
An Empirical Study of Training Self-Supervised Vision Transformers
Chen, X., Xie, S., & He, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00950 · Cited by 1,648
The kinetics human action video dataset
Kay, W. (2024) · TIB Data Manager · doi:10.57702/jtuxnt5m · Cited by 1,646
Hybrid Task Cascade for Instance Segmentation
Chen, K., Pang, J., Wang, J., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Shi, J., Ouyang, W., et al. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00511 · Cited by 1,587
BEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392
COCO-Stuff: Thing and Stuff Classes in Context
Caesar, H., Uijlings, J., & Ferrari, V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00132 · Cited by 1,277
LVIS: A Dataset for Large Vocabulary Instance Segmentation
Gupta, A., Dollar, P., & Girshick, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00550 · Cited by 1,274
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Going deeper with Image Transformers
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jegou, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00010 · Cited by 1,259
SimMIM: a Simple Framework for Masked Image Modeling
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00943 · Cited by 1,255
Self-citationThe Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00823 · Cited by 1,250
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., & Shum, H.-Y. (2022) · arXiv · Cited by 1,154 · cited 8×
“(timestamp: Nov 10, 2022) methods / results reference. a: BEiT-3 [104], b: iBOT [130], c: Open CLIP-H [48], d: Text4Vis [112], e: MaskFeat [106], f: Group DETRv2 [19], g: FocalNet [119], h: FD-SwinV2-G [110], i: Mask DINO [57], j: LVIS 2021 competition 1 st [36], k: ViT-Adapter [23]. pure MIM pre-trained model could n…”
BackgroundMethodInfluentialSimple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation
Ghiasi, G., Cui, Y., Srinivas, A., Qian, R., Lin, T.-Y., Cubuk, E. D., Le, Q. V., & Zoph, B. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00294 · Cited by 1,148
PointRend: Image Segmentation As Rendering
Kirillov, A., Wu, Y., He, K., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00982 · Cited by 1,135
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01501 · Cited by 1,090
Grounded Language-Image Pre-training
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01069 · Cited by 1,086
(2022) · doi:10.1007/978-3-031-20053-3_27 · Cited by 1,060
Huang, Z., Huang, L., Gong, Y., Huang, C., & Wang, X. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00657 · Cited by 1,049
Self-citationDynamic Head: Unifying Object Detection Heads with Attentions
Dai, X., Chen, Y., Xiao, B., Chen, D., Liu, M., Yuan, L., & Zhang, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00729 · Cited by 1,044
LAION-5B: An open large-scale dataset for training next generation image-text models
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.08402 · Cited by 1,022
(2020) · International Conference on Machine Learning · Cited by 1,012
(2020) · doi:10.1109/sc41405.2020.00024 · Cited by 928
CoAtNet: Marrying Convolution and Attention for All Data Sizes
Dai, Z., Liu, H., Le, Q. V., & Tan, M. (2021) · arXiv · doi:10.48550/arxiv.2106.04803 · Cited by 925
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training
Tong, Z., Song, Y., Wang, J., & Wang, L. (2022) · arXiv · doi:10.48550/arxiv.2203.12602 · Cited by 909 · cited 2×
“EVA processes video data simply via spatial-temporal attention as [35, 97] with no specific architectural adaptation for video related tasks.”
BackgroundMethodZhai, X., Kolesnikov, A., Houlsby, N., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01179 · Cited by 891
Rasley, J., Rajbhandari, S., Ruwase, O., & He, Y. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3406703 · Cited by 823
Exploring Plain Vision Transformer Backbones for Object Detection
Li, Y., Mao, H., Girshick, R., & He, K. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20077-9_17 · Cited by 809
Do ImageNet Classifiers Generalize to ImageNet?
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2019) · arXiv · doi:10.48550/arxiv.1902.10811 · Cited by 790
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Objects365: A Large-Scale, High-Quality Dataset for Object Detection
Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., & Sun, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00852 · Cited by 673 · cited 4×
“For intermediate fine-tuning on Objects365 [87], the model is trained with a batch size of 128 for 380k iterations.”
MethodInfluentialConceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts
Changpinyo, S., Sharma, P., Ding, N., & Soricut, R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00356 · Cited by 612
Masked Feature Prediction for Self-Supervised Visual Pre-Training
Wei, C., Fan, H., Xie, S., Wu, C.-Y., Yuille, A., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01426 · Cited by 587
End-to-End Semi-Supervised Object Detection with Soft Teacher
Xu, M., Zhang, Z., Hu, H., Wang, J., Wang, L., Wei, F., Bai, X., & Liu, Z. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00305 · Cited by 565
Training Deep Nets with Sublinear Memory Cost
Chen, T., Xu, B., Zhang, C., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1604.06174 · Cited by 561
Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation
Li, F., Zhang, H., Xu, H., Liu, S., Zhang, L., Ni, L. M., & Shum, H.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00297 · Cited by 560
CoCa: Contrastive Captioners are Image-Text Foundation Models
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., & Wu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.01917 · Cited by 503
BEiT v2: Masked Image Modeling with Vector-Quantized Visual TokenizersPreprint
Peng, Z., Dong, L., Bao, H., Ye, Q.-X., & Wei, F. (2022) · arXiv.org · doi:10.48550/arxiv.2208.06366 · Cited by 468
Touvron, H., Cord, M., & Jégou, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20053-3_30 · Cited by 409
Florence: A New Foundation Model for Computer Vision
Yuan, L., Chen, D., Chen, Y.-L., Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al. (2021) · arXiv · doi:10.48550/arxiv.2111.11432 · Cited by 389
LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., & Komatsuzaki, A. (2021) · arXiv · doi:10.48550/arxiv.2111.02114 · Cited by 358
Learning Robust Global Representations by Penalizing Local Predictive Power
Wang, H., Ge, S., Xing, E. P., & Lipton, Z. C. (2019) · arXiv · doi:10.48550/arxiv.1905.13549 · Cited by 337
A Short Note on the Kinetics-700-2020 Human Action Dataset
Smaira, L., Carreira, J., Noland, E., Clancy, E., Wu, A., & Zisserman, A. (2020) · arXiv · doi:10.48550/arxiv.2010.10864 · Cited by 334
Expanding Language-Image Pretrained Models for General Video Recognition
Ni, B., Peng, H., Chen, M., Zhang, S., Meng, G., Fu, J., Xiang, S., & Ling, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19772-7_1 · Cited by 326
Context Autoencoder for Self-supervised Representation Learning
Chen, X., Ding, M., Wang, X., Xin, Y., Mo, S., Wang, Y., Han, S., Luo, P., Zeng, G., & Wang, J. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-023-01852-4 · Cited by 319
Self-citationProbabilistic two-stage detectionPreprint
Zhou, X., Koltun, V., & Krähenbühl, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.07461 · Cited by 300
(2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 291
Masked Autoencoders as Spatiotemporal Learners
Feichtenhofer, C., Fan, H., Li, Y., & He, K. (2022) · neural information processing systems · doi:10.52202/068431-2605 · Cited by 273
A Short Note about Kinetics-600
Carreira, J., Noland, E., Banki-Horvath, A., Hillier, C., & Zisserman, A. (2018) · arXiv · doi:10.48550/arxiv.1808.01340 · Cited by 270
Yang, J., Li, C., Dai, X., Yuan, L., & Gao, J. (2022) · arXiv · doi:10.48550/arxiv.2203.11926 · Cited by 265
EVA-02: A visual representation for neon genesis
Fang, Y., Sun, Q., Wang, X., Huang, T., Wang, X., & Cao, Y. (2024) · Image and Vision Computing · doi:10.1016/j.imavis.2024.105171 · Cited by 264
Self-citationdata2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., & Auli, M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.03555 · Cited by 231
HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions
Rao, Y., Zhao, W., Tang, Y., Zhou, J., Lim, S. N., & Lu, J. (2022) · neural information processing systems · doi:10.52202/068431-0752 · Cited by 217
iBOT: Image BERT Pre-Training with Online Tokenizer
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A., & Kong, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07832 · Cited by 201
Vision Transformer Adapter for Dense Predictions
Chen, Z., Duan, Y., Wang, W., He, J., Lu, T., Dai, J., & Qiao, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.08534 · Cited by 195
Self-citationYang, J., Li, C., Dai, X., & Gao, J. (2022) · neural information processing systems · doi:10.52202/068431-0304 · Cited by 193
Beyer, L., Hénaff, O. J., Kolesnikov, A., Zhai, X., & Oord, A. V. D. (2020) · arXiv · doi:10.48550/arxiv.2006.07159 · Cited by 176
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language TasksPreprint
Wang, W., Bao, H., Li, D., Björck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.10442 · Cited by 163
Self-citationGLIPv2: Unifying Localization and Vision-Language Understanding
Zhang, H., Zhang, P., Hu, X., Chen, Y.-C., Li, L., Dai, X., Wang, L., Yuan, L., Hwang, J.-N., & Gao, J. (2022) · neural information processing systems · doi:10.52202/068431-2614 · Cited by 162
Revealing the Dark Secrets of Masked Image Modeling
Xie, Z., Geng, Z., Hu, J., Zhang, Z., Hu, H., & Cao, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01391 · Cited by 155
Self-citationSeMask: Semantically Masked Transformers for Semantic Segmentation
Jain, J., Singh, A. T., Orlov, N., Huang, Z., Li, J., Walton, S., & Shi, H. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00083 · Cited by 142
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature DistillationPreprint
Wei, Y., Hu, H., Xie, Z., Zhang, Z., Cao, Y., Bao, J., Chen, D., & Guo, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.14141 · Cited by 133
Self-citationSOLO: A Simple Framework for Instance Segmentation
Wang, X., Zhang, R., Shen, C., Kong, T., & Li, L. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3111116 · Cited by 132
Self-citationMVP: Multimodality-Guided Visual Pre-training
Wei, L., Xie, L., Zhou, W., Li, H., & Tian, Q. C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20056-4_20 · Cited by 90
MILAN: Masked Image Pretraining on Language Assisted RepresentationPreprint
Hou, Z., Sun, F., Chen, Y.-K., Xie, Y., & Kung, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.06049 · Cited by 81
On Data Scaling in Masked Image Modeling
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Wei, Y., Dai, Q., & Hu, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00999 · Cited by 72
Self-citationCo-training Transformer with Videos and Images Improves Action RecognitionPreprint
Zhang, B., Yu, J., Fifty, C., Han, W., Dai, A. M., Pang, R., & Sha, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.07175 · Cited by 65
Corrupted Image Modeling for Self-Supervised Visual Pre-TrainingPreprint
Fang, Y., Dong, L., Bao, H., Wang, X., & Wei, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2202.03382 · Cited by 61
Self-citationUniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormerPreprint
Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Wang, L., & Qiao, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.09552 · Cited by 59
Exploring Target Representations for Masked AutoencodersPreprint
Liu, X., Zhou, J., Kong, T., Lin, X., & Ji, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.03917 · Cited by 43
PeCo: Perceptual Codebook for BERT Pre-training of Vision TransformersPreprint
Dong, X., Bao, J., Zhang, T., Chen, D., Zhang, W., Yuan, L., Chen, D., Wen, F., Yu, N., & Guo, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12710 · Cited by 32
Improved Multiscale Vision Transformers for Classification and DetectionPreprint
Li, Y., Wu, C.-Y., Fan, H., Mangalam, K., Xiong, B., Malik, J., & Feichtenhofer, C. (2021) · arXiv (Cornell University) · Cited by 23
Group DETR v2: Strong Object Detector with Encoder-Decoder Pretraining
Chen, Q., Wang, J., Han, C., Zhang, S., Li, Z., Chen, X., Chen, J., Wang, X., Han, S., Zhang, G., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.03594 · Cited by 19
Self-citation
Research works printed but not linked: 18
- ilharco, 2021, Openclip
- ba, 2016, Layer normalization, ArXiv Preprint
- fu, 2021, Lvis challenge track technical report 1st place solution: Distribution balanced and boundary refinement for large vocabulary instance segmentation, ArXiv Preprint
- dosovitskiy, 0, An image is worth 16×16 words: Transformers for image recognition at scale, ICLRE
- vaswani, 2017, Attention is all you need, NeurIPS, 30
- byeon, 2022, Coyo-700m Image-text pair dataset
- tu, 2022, Maxvit: Multi-axis vision transformer, ArXiv Preprint
- 10.18653/v1/N18-2074 · DOI · Paper page
- recht, 2019, Do imagenet classifiers generalize to imagenet?
- wightman, 2019, pytorch-image-models
- wu, 2022, Transferring textual knowledge for visual recognition, ArXiv Preprint
- rae, 2021, Scaling language models: Methods, analysis & insights from training gopher, ArXiv Preprint
- radford, 2018, Improving language understanding by generative pre-training
- 0, L/14 h/14 and g/14 trained on laion-2b
- 0, Connecting text and images
- pham, 2021, Combined scaling for open-vocabulary image classification, ArXiv Preprint
- pham, 2021, Combined scaling for zero-shot transfer learning, ArXiv Preprint
- paszke, 2019, Pytorch: An imperative style, high-performance deep learning library, NeurIPS
Cited by
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2310.03744 · Cited by 1,733
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748
Towards a general-purpose foundation model for computational pathology
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02857-3 · Cited by 1,560
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078
“The dataset used for training EVA-CLIP is a custom mixture, see paper for details (Fang et al., 2023).”
MethodA survey on multimodal large language models
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024) · National Science Review · doi:10.1093/nsr/nwae403 · Cited by 893
Metadata
- Authors
- Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., & Cao, Y.
- Venue
- 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2023-06
- Pages
- 19358-19369
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4386076522
- Cited by
- 506
- References
- 115
- FWCI
- 57.53
- Type
- conference paper
- Language
- en
- Open access
- green
- Funders
- National Key R&D Program of China (2020AAA0105200)
- NSFC (61733007)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.