Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 1,255

The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.

MetaFormer is Actually What You Need for Vision

Open access
Yu, Weihao, Luo, Mi, Zhou, Pan, Si, Chenyang, Zhou, Yichen, Wang, Xinchao, Feng, Jiashi, Yan, Shuicheng·2022·2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52688.2022.01055
Open this paper in CoChat — free account

Abstract

Transformers have shown great potential in computer vision tasks. A common belief is their attention-based token mixer module contributes most to their competence. However, recent works show the attention-based module in transformers can be replaced by spatial MLPs and the resulted models still perform quite well. Based on this observation, we hypothesize that the general architecture of the transformers, instead of the specific token mixer module, is more essential to the model's performance. To verify this, we deliberately replace the attention module in transformers with an embarrassingly simple spatial pooling operator to conduct only basic token mixing. Surprisingly, we observe that the derived model, termed as PoolFormer, achieves competitive performance on multiple computer vision tasks. For example, on ImageNet-1K, PoolFormer achieves 82.1 % top-1 accuracy, surpassing well-tuned vision transformer/MLP-like baselines DeiT-B/ResMLP-B24 by 0.3%/1.1% accuracy with 35%/52% fewer parameters and 49%/61% fewer MACs. The effectiveness of Pool-Former verifies our hypothesis and urges us to initiate the concept of “MetaFormer”, a general architecture abstracted from transformers without specifying the token mixer. Based on the extensive experiments, we argue that MetaFormer is the key player in achieving superior results for recent transformer and MLP-like models on vision tasks. This work calls for more future research dedicated to improving MetaFormer instead of focusing on the token mixer modules. Additionally, our proposed PoolFormer could serve as a starting baseline for future MetaFormer architecture design.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr52688.2022.01055

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 60 references

Research works printed but not linked: 14

  1. liu, 2021, Pay attention to mlps, ArXiv Preprint
  2. lei ba, 2016, Layer normalization, ArXiv Preprint
  3. hou, 2021, Vision permutator: A permutable mlp-like architecture for visual recognition, ArXiv Preprint
  4. touvron, 0, Training data-efficient image transformers & distillation through attention, International Conference on Machine Learning, 10347
  5. wightman, 2021, Resnet strikes back: An improved training procedure in timm, ArXiv Preprint
  6. wightman, 2019, pytorch-image-models
  7. vaswani, 0, Attention is all you need, Advances in neural information processing systems, 5998
  8. contributors, 2020, MMSegmentation Openmmlab semantic segmentation toolbox and benchmark
  9. d'ascoli, 2021, Convit: Improving vision transformers with soft convolutional inductive biases, Proceedings of the 38th International Conference on Machine Learning volume 139 of Proceedings of Machine Learning Research, 2286
  10. dong, 2021, Attention is not all you need: Pure attention loses rank doubly exponentially with depth, ArXiv Preprint
  11. dosovitskiy, 0, An image is worth 16×16 words: Transformers for image recognition at scale, International Conference on Learning Representations
  12. chen, 2018, A^ 2-nets: Double attention networks, Advances in neural information processing systems, 31, 352
  13. raghu, 2021, Do vision transformers see like convolutional neural networks?, ArXiv Preprint
  14. rao, 2021, Global filter networks for image classification, ArXiv Preprint

Cited by

See all 1,255

Metadata

Authors
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., & Yan, S.
Venue
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2022-06
Pages
10809-10819
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52688.2022.01055
OpenAlex
W4312820606
Cited by
1,255
References
60
FWCI
103.05
Type
conference paper
Language
en
Open access
green
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: MetaFormer is Actually What You Need for Vision. https://app.cochat.ai/papers/10.1109/cvpr52688.2022.01055
The paper
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., & Yan, S. (2022). MetaFormer is Actually What You Need for Vision. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr52688.2022.01055

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.