Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 506

The DOI matches its Crossref record; 115 of the 115 references it cites were found in a registry or an index.

EVA: Exploring the Limits of Masked Visual Representation Learning at Scale

Open access
Fang, Yuxin, Wang, Wen, Xie, Binhui, Sun, Quan, Wu, Ledell, Wang, Xinggang, Huang, Tiejun, Wang, Xinlong, Cao, Yue·2023·2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Abstract

We launch EVA, a vision-centric foundation model to Explore the limits of Visual representation at scAle using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVIS dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can greatly stabilize the training and outperform the training from scratch counterpart with much fewer samples and less compute, providing a new direction for scaling up and accelerating the costly training of multi-modal foundation models.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52729.2023.01855

Cites

115 of 115 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 115 references

Research works printed but not linked: 18

  1. ilharco, 2021, Openclip
  2. ba, 2016, Layer normalization, ArXiv Preprint
  3. fu, 2021, Lvis challenge track technical report 1st place solution: Distribution balanced and boundary refinement for large vocabulary instance segmentation, ArXiv Preprint
  4. dosovitskiy, 0, An image is worth 16×16 words: Transformers for image recognition at scale, ICLRE
  5. vaswani, 2017, Attention is all you need, NeurIPS, 30
  6. byeon, 2022, Coyo-700m Image-text pair dataset
  7. tu, 2022, Maxvit: Multi-axis vision transformer, ArXiv Preprint
  8. 10.18653/v1/N18-2074 · DOI · Paper page
  9. recht, 2019, Do imagenet classifiers generalize to imagenet?
  10. wightman, 2019, pytorch-image-models
  11. wu, 2022, Transferring textual knowledge for visual recognition, ArXiv Preprint
  12. rae, 2021, Scaling language models: Methods, analysis & insights from training gopher, ArXiv Preprint
  13. radford, 2018, Improving language understanding by generative pre-training
  14. 0, L/14 h/14 and g/14 trained on laion-2b
  15. 0, Connecting text and images
  16. pham, 2021, Combined scaling for open-vocabulary image classification, ArXiv Preprint
  17. pham, 2021, Combined scaling for zero-shot transfer learning, ArXiv Preprint
  18. paszke, 2019, Pytorch: An imperative style, high-performance deep learning library, NeurIPS

Cited by

See all 506

Metadata

Authors
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., & Cao, Y.
Venue
2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2023-06
Pages
19358-19369
ISSN
1063-6919 (ISSN-L)
OpenAlex
W4386076522
Cited by
506
References
115
FWCI
57.53
Type
conference paper
Language
en
Open access
green
Funders
  • National Key R&D Program of China (2020AAA0105200)
  • NSFC (61733007)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. https://app.cochat.ai/papers/10.1109/cvpr52729.2023.01855
The paper
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., & Cao, Y. (2023). EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr52729.2023.01855

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.