The DOI matches its Crossref record; 73 of the 78 references it cites were found in a registry or an index.
AutoFormer: Searching Transformers for Visual Recognition
Open accessAbstract
Recently, pure transformer-based models have shown great potentials for vision tasks such as image classification and detection. However, the design of transformer networks is challenging. It has been observed that the depth, embedding dimension, and number of heads can largely affect the performance of vision transformers. Previous models configure these dimensions based upon manual crafting. In this work, we propose a new one-shot architecture search framework, namely AutoFormer, dedicated to vision transformer search. AutoFormer entangles the weights of different blocks in the same layers during supernet training. Benefiting from the strategy, the trained supernet allows thousands of subnets to be very well-trained. Specifically, the performance of these subnets with weights inherited from the supernet is comparable to those retrained from scratch. Besides, the searched models, which we refer to AutoFormers, surpass the recent state-of-the-arts such as ViT and DeiT. In particular, AutoFormer-tiny/small/base achieve 74.7%/81.7%/82.4% top-1 accuracy on ImageNet with 5.7M/22.9M/53.7M parameters, respectively. Lastly, we verify the transferability of AutoFormer by providing the performance on downstream benchmarks and distillation experiments. Code and models are available at https://github.com/microsoft/Cream.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The methods include pruning [40, 41], Quantization [42-44], Knowledge Distillation [45] and Neural Architecture Search [46].”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [46]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
(2014) · Cited by 33,419
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,889
Research works printed but not linked: 2
- li, 2020, Blockwisely supervised neural architecture search with knowledge distillation, CVPR
- huang, 2019, Ccnet: Crisscross attention for semantic segmentation, ICCV
Cited by
Sorted by citations
Items 301–324 of 324
Wang, J., Luo, M., Yue, Z., Xia, M. Y., Shi, C., & Chen, F. (2026) · IEEE Sensors Journal · doi:10.1109/jsen.2026.3655120
Towards Generalizable Time Series Forecasting Via IB-Regularized Transformer-Diffusion
Na, D., Kang, J., & Kwon, J. (2025) · Proceedings · doi:10.1109/icdm65498.2025.00153
Proxy-Mamba: Training-Free Architecture Search for Mamba via Gradient-Weight Correlation
Zhang, H., Wang, L., & Sun, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7075-1_33
Meng, L., Yuan, Y., Gao, J., Chen, C., & Zhang, J. (2026) · Nondestructive Testing And Evaluation · doi:10.1080/10589759.2026.2653140
ADCformer: multivariate time series forecasting with adaptive differential channels transformer
Xia, Y., Zhou, Y., Hu, X., & Duan, S. (2026) · Computing · doi:10.1007/s00607-026-01644-x
XAI-Pruner: Explainability-Driven Pruning of CNN and Transformer
Niu, D., Pan, H., Jiang, J., Xu, J., & Niu, X. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462953
Aruna Gladys, A., Aishwarya, R., Vetriselvi, V., Daphne, S., Baskar, P. N., & Rajan, A. A. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10598-4 · Cited by 1
Target-Adaptive Neural Architecture Search in YOLOv9 Machine Vision Models
Son, S. B., Cho, Y. A., Kim, J., & Park, S. (2026) · International Conference on Information Networking · doi:10.1109/icoin68469.2026.11480593
Beyond the Limits: Overcoming Negative Correlation of Activation-Based Training-Free NAS
Kang, H., Ma, L., Chen, P., Yu, G., Wang, X., & Huang, M. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.00082
Huang, J., Xue, B., Sun, Y., Zhang, M., & Yen, G. G. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113917
Towards distribution-aware active learning for data-efficient neural architecture predictor
Yu, C., Wang, Y., Tang, C., Ju, W., Liu, X., & Lv, J. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104892
Delibaşoğlu, İ., Chakraborty, S., & Heintz, F. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.115449
Liu, G., Li, Y., Zhang, X., Li, L., Shang, R., & Jiao, L. (2026) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2026.3708498
VibrationID: A Vibration-Based Behavior Classification Method for Identity Authentication
Cui, Z., Yang, Z., & He, Z. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2487-6_6
Zhang, B., Ren, A., Liu, D., Xiong, L., He, H., Zhong, K., & Duan, M. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.103932
A survey of deep learning in image processing: developments, applications and challenges
Xiong, Z., Tang, X., & Verbeek, F. J. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11618-2
Neural Architecture Search via Differentiable Proxies in AUTOML
Rejina, P., V (2026) · Eduschool International Journal of Data Science and Machine learning (EIJDSML) · doi:10.63090/eijdsml/3139.7344.0004
Covariance-Inverse Fusion: Toward Comprehensive Proxies for Zero-Shot Neural Architecture Search
Wu, N., Huang, H. P., Kang, L., Xu, Y., Feng, F., & Wu, C. (2026) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2026.3719625
Tackling Workload Forecasting Challenges with an Offline-Online Dynamic Framework
Jiang, J., Liu, Y., Li, J., Han, L., Lu, W., Deng, Q., Zhu, Z., Lu, X., & Ma, L. (2026) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65706.2026.00270
Zhang, H., Yang, Y., Liu, S., Ma, L., Wong, K., & Lin, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37041-9_18
Tyagi, S., Rawat, P., Yadav, D., & Upadhyay, P. (2026) · Expert Systems · doi:10.1111/exsy.70422
Gokul, K. P., & Dhanalakshmy, D. M. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-6513-8_17
Zhang, Y., Leung, C., Bakhi, A., Coelho, B., & Itoh, M. (2026) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2026.3743062
Evolutionary Transformer Architecture Search via LLM-Guided Coarse-to-Fine Transformations
Liu, L., Yen, G. G., Li, C., & He, Z. (2026) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec69861.2026.11727121
Metadata
- Authors
- Chen, M., Peng, H., Fu, J., & Ling, H.
- Affiliations
- Stony Brook University (US)
- Microsoft Research Asia (China) (CN)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 12250-12260
- OpenAlex
- W3204801262
- Cited by
- 323
- References
- 78
- Citations by year
- 2026: 28 · 2025: 75 · 2024: 103 · 2023: 79 · 2022: 33 · 2021: 5
- FWCI
- 17.99
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- vision transformer, image classification, transferability, knowledge distillation, Autoformer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
