Cited by 324Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

The DOI matches its Crossref record; 73 of the 78 references it cites were found in a registry or an index.

AutoFormer: Searching Transformers for Visual Recognition

Open access
Chen, Minghao, Peng, Houwen, Fu, Jianlong, Ling, Haibin·2021·2021 IEEE/CVF International Conference on Computer Vision (ICCV)

Abstract

Recently, pure transformer-based models have shown great potentials for vision tasks such as image classification and detection. However, the design of transformer networks is challenging. It has been observed that the depth, embedding dimension, and number of heads can largely affect the performance of vision transformers. Previous models configure these dimensions based upon manual crafting. In this work, we propose a new one-shot architecture search framework, namely AutoFormer, dedicated to vision transformer search. AutoFormer entangles the weights of different blocks in the same layers during supernet training. Benefiting from the strategy, the trained supernet allows thousands of subnets to be very well-trained. Specifically, the performance of these subnets with weights inherited from the supernet is comparable to those retrained from scratch. Besides, the searched models, which we refer to AutoFormers, surpass the recent state-of-the-arts such as ViT and DeiT. In particular, AutoFormer-tiny/small/base achieve 74.7%/81.7%/82.4% top-1 accuracy on ImageNet with 5.7M/22.9M/53.7M parameters, respectively. Lastly, we verify the transferability of AutoFormer by providing the performance on downstream benchmarks and distillation experiments. Code and models are available at https://github.com/microsoft/Cream.

Overview

Integrity signals

Registry

Crossref · 10.1109/iccv48922.2021.01205

Cites

73 of 73 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 78 references

Research works printed but not linked: 2

  1. li, 2020, Blockwisely supervised neural architecture search with knowledge distillation, CVPR
  2. huang, 2019, Ccnet: Crisscross attention for semantic segmentation, ICCV

Cited by

Sorted by citations

Items 201–300 of 324

Metadata

Authors
Chen, M., Peng, H., Fu, J., & Ling, H.
Affiliations
  • Stony Brook University (US)
  • Microsoft Research Asia (China) (CN)
Venue
2021 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2021-10
Pages
12250-12260
OpenAlex
W3204801262
Cited by
323
References
78
Citations by year
2026: 28 · 2025: 75 · 2024: 103 · 2023: 79 · 2022: 33 · 2021: 5
FWCI
17.99
Type
conference paper
Language
en
Open access
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
vision transformer, image classification, transferability, knowledge distillation, Autoformer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review