AutoFormer: Searching Transformers for Visual Recognition
Open accessAbstract
Recently, pure transformer-based models have shown great potentials for vision tasks such as image classification and detection. However, the design of transformer networks is challenging. It has been observed that the depth, embedding dimension, and number of heads can largely affect the performance of vision transformers. Previous models configure these dimensions based upon manual crafting. In this work, we propose a new one-shot architecture search framework, namely AutoFormer, dedicated to vision transformer search. AutoFormer entangles the weights of different blocks in the same layers during supernet training. Benefiting from the strategy, the trained supernet allows thousands of subnets to be very well-trained. Specifically, the performance of these subnets with weights inherited from the supernet is comparable to those retrained from scratch. Besides, the searched models, which we refer to AutoFormers, surpass the recent state-of-the-arts such as ViT and DeiT. In particular, AutoFormer-tiny/small/base achieve 74.7%/81.7%/82.4% top-1 accuracy on ImageNet with 5.7M/22.9M/53.7M parameters, respectively. Lastly, we verify the transferability of AutoFormer by providing the performance on downstream benchmarks and distillation experiments. Code and models are available at https://github.com/microsoft/Cream.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The methods for reducing transformers' size, latency and energy consumption include pruning, quantization, knowledge distillation and neural architecture search.
“The methods include pruning [40, 41], Quantization [42-44], Knowledge Distillation [45] and Neural Architecture Search [46].”
From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [46]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
(2014) · Cited by 33,419 · see citing works
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755 · see citing works
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,890 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · see citing works
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · see citing works
Wang, X., Girshick, R., Gupta, A., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00813 · Cited by 11,596 · see citing works
Howard, A., Sandler, M., Chen, B., Wang, W., Chen, L.-C., Tan, M., Chu, G., Vasudevan, V., Zhu, Y., Pang, R., et al. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00140 · Cited by 9,708 · see citing works
Learning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00907 · Cited by 6,631 · see citing works
mixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831 · see citing works
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
Yun, S., Han, D., Chun, S., Oh, S. J., Yoo, Y., & Choe, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00612 · Cited by 5,400 · see citing works
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,091 · see citing works
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
Self-citationTransformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595 · see citing works
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · see citing works
Automated Flower Classification over a Large Number of Classes
Nilsback, M.-E., & Zisserman, A. (2008) · 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing · doi:10.1109/icvgip.2008.47 · Cited by 3,442 · see citing works
MnasNet: Platform-Aware Neural Architecture Search for Mobile
Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., & Le, Q. V. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00293 · Cited by 3,416 · see citing works
Randaugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00359 · Cited by 3,223 · see citing works
CCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045 · see citing works
CvT: Introducing Convolutions to Vision Transformers
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., & Zhang, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00009 · Cited by 2,649 · see citing works
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415 · see citing works
Deep Networks with Stochastic Depth
Huang, G., Sun, Y., Liu, Z., Sedra, D., & Weinberger, K. Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_39 · Cited by 2,104 · see citing works
Designing Network Design Spaces
Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., & Dollar, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01044 · Cited by 2,084 · see citing works
(2013) · Cited by 2,056 · see citing works
Neural Architecture Search: A Survey
Elsken, T., Metzen, J. H., & Hutter, F. (2018) · arXiv · doi:10.48550/arxiv.1808.05377 · Cited by 1,763 · see citing works
Deeply-Supervised NetsPreprint
Lee, C., Xie, S., Gallagher, P., Zhang, Z., & Tu, Z. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.5185 · Cited by 1,661 · see citing works
Parkhi, O., Vedaldi, A., Zisserman, A., & Jawahar, C. V. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248092 · Cited by 1,564 · see citing works
What Does BERT Look at? An Analysis of BERT’s Attention
Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w19-4828 · Cited by 1,514 · see citing works
(2018) · International Conference on Machine Learning · Cited by 1,465 · see citing works
Bottleneck Transformers for Visual Recognition
Srinivas, A., Lin, T.-Y., Parmar, N., Shlens, J., Abbeel, P., & Vaswani, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01625 · Cited by 1,385 · see citing works
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
Wu, B., Keutzer, K., Dai, X., Zhang, P., Wang, Y., Sun, F., Wu, Y., Tian, Y., Vajda, P., & Jia, Y. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01099 · Cited by 1,302 · see citing works
Visualizing the Loss Landscape of Neural Nets
Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2017) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000461393 · Cited by 1,148 · see citing works
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1580 · Cited by 1,146 · see citing works
Exploring Self-Attention for Image Recognition
Zhao, H., Jia, J., & Koltun, V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01009 · Cited by 1,002 · see citing works
Stand-Alone Self-Attention in Vision Models
Ramachandran, P., Parmar, N., Vaswani, A., Bello, I., Levskaya, A., & Shlens, J. (2019) · arXiv · doi:10.48550/arxiv.1906.05909 · Cited by 977 · see citing works
Xie, L., & Yuille, A. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.154 · Cited by 904 · see citing works
Single Path One-Shot Neural Architecture Search with Uniform Sampling
Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., & Sun, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_32 · Cited by 882 · see citing works
A Convergence Theory for Deep Learning via Over-ParameterizationPreprint
Allen-Zhu, Z., Li, Y., & Song, Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03962 · Cited by 847 · see citing works
Once-for-All: Train One Network and Specialize it for Efficient Deployment
Cai, H., Gan, C., Wang, T., Zhang, Z., & Han, S. (2019) · arXiv · doi:10.48550/arxiv.1908.09791 · Cited by 845 · see citing works
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
Wang, H., Zhu, Y., Green, B., Adam, H., Yuille, A., & Chen, L.-C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58548-8_7 · Cited by 809 · see citing works
Learning Deep Transformer Models for Machine Translation
Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., & Chao, L. S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1176 · Cited by 673 · see citing works
Understanding and Simplifying One-Shot Architecture Search
Bender, G., Kindermans, P.-J., Zoph, B., Vasudevan, V., & Le, Q. V. (2018) · International Conference on Machine Learning · Cited by 558 · see citing works
Random Search and Reproducibility for Neural Architecture SearchPreprint
Li, L., & Talwalkar, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.07638 · Cited by 534 · see citing works
SMASH: One-Shot Model Architecture Search through HyperNetworksPreprint
Brock, A., Lim, T., Ritchie, J. M., & Weston, N. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05344 · Cited by 500 · see citing works
Gradient Descent Provably Optimizes Over-parameterized Neural NetworksPreprint
Du, S. S., Zhai, X., Póczos, B., & Singh, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02054 · Cited by 490 · see citing works
Co-Scale Conv-Attentional Image Transformers
Xu, W., Xu, Y., Chang, T. A., & Tu, Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00983 · Cited by 425 · see citing works
Are Sixteen Heads Really Better than One?
Michel, P., Levy, O., & Neubig, G. (2019) · arXiv · doi:10.48550/arxiv.1905.10650 · Cited by 404 · see citing works
Slimmable Neural NetworksPreprint
Yu, J., Yang, L., Xu, N., Yang, J., & Huang, T. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.08928 · Cited by 403 · see citing works
Universally Slimmable Networks and Improved Training Techniques
Yu, J., & Huang, T. S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00189 · Cited by 366 · see citing works
FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search
Chu, X., Zhang, B., & Xu, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01202 · Cited by 358 · see citing works
Regularized Evolution for Image Classifier Architecture Search
Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33014780 · Cited by 328 · see citing works
So, D. R., Liang, C., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1901.11117 · Cited by 277 · see citing works
BigNAS: Scaling up Neural Architecture Search with Big Single-Stage Models
Yu, J., Jin, P., Liu, H., Bender, G., Kindermans, P.-J., Tan, M., Huang, T. S., Song, X., Pang, R., & Le, Q. V. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58571-6_41 · Cited by 252 · see citing works
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
Wang, H., Wu, Z., Liu, Z., Cai, H., Zhu, L., Gan, C., & Han, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.686 · Cited by 227 · see citing works
PolyTransform: Deep Polygon Transformer for Instance Segmentation
Liang, J., Homayounfar, N., Ma, W.-C., Xiong, Y., Hu, R., & Urtasun, R. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00915 · Cited by 188 · see citing works
Block-Wisely Supervised Neural Architecture Search With Knowledge Distillation
Li, C., Peng, J., Yuan, L., Wang, G., Liang, X., Lin, L., & Chang, X. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00206 · Cited by 187 · see citing works
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182 · see citing works
Training Deeper Neural Machine Translation Models with Transparent Attention
Bapna, A., Chen, M., Firat, O., Cao, Y., & Wu, Y.-H. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1338 · Cited by 160 · see citing works
Self-citationLi, C., Tang, T., Wang, G., Peng, J., Wang, B., Liang, X., & Chang, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01206 · Cited by 109 · see citing works
Collecting a Large-scale Dataset of Fine-grained Cars
Krause, J., Deng, J., Stark, M. M., & Fei-Fei, L. (2013) · MPG.PuRe (Max Planck Society) · Cited by 100 · see citing works
Grafit: Learning fine-grained image representations with coarse labels
Touvron, H., Sablayrolles, A., Douze, M., Cord, M., & Jeǵou, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00091 · Cited by 77 · see citing works
Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture SearchPreprint
Peng, H., Du, H., Yu, H., Li, Q., Liao, J., & Fu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.15821 · Cited by 57 · see citing works
Self-citationAugment your batch: better training with larger batchesPreprint
Hoffer, E., Ben‐Nun, T., Hubara, I., Giladi, N., Hoefler, T., & Soudry, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.09335 · Cited by 45 · see citing works
Go Wide, Then Narrow: Efficient Training of Deep Thin NetworksPreprint
Zhou, D., Ye, M., Chen, C., Meng, T., Tan, M., Song, X., Le, Q. V., Liu, Q., & Schuurmans, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00811 · Cited by 13 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
5 of the 78 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682 · see citing works
Transformers in Vision: A Survey
Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022) · ACM Computing Surveys · doi:10.1145/3505244 · Cited by 3,595 · see citing works
SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer
Ma, J., Tang, L., Fan, F., Huang, J., Mei, X., & Ma, Y. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105686 · Cited by 1,557 · see citing works
Transformers in Time Series: A Survey
Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., & Sun, L. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/759 · Cited by 1,260 · see citing works
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention
Liu, X., Peng, H., Zheng, N., Yang, Y., Hu, H., & Yuan, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01386 · Cited by 956 · see citing works
Self-citationTinyViT: Fast Pretraining Distillation for Small Vision Transformers
Wu, K., Zhang, J., Peng, H., Liu, M., Xiao, B., Fu, J., & Yuan, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19803-8_5 · Cited by 480 · see citing works
Self-citationA survey of the vision transformers and their CNN-transformer based variants
Khan, A., Rauf, Z., Sohail, A., Khan, A. R., Asif, H., Asif, A., & Farooq, U. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10595-0 · Cited by 496 · see citing works
Rethinking and Improving Relative Position Encoding for Vision Transformer
Wu, K., Peng, H., Chen, M., Fu, J., & Chao, H. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00988 · Cited by 393 · see citing works
Self-citationA review of convolutional neural network architectures and their optimizations
Cong, S., & Zhou, Y. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10213-5 · Cited by 397 · see citing works
Rethinking Vision Transformers for MobileNet Size and Speed
Li, Y., Ju, H., Wen, Y., Evangelidis, G., Salahi, K., Wang, Y., Tulyakov, S., & Ren, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01549 · Cited by 369 · see citing works
CrossFormer++: A Versatile Vision Transformer Hinging on Cross-Scale Attention
Wang, W., Chen, W., Qiu, Q., Chen, L., Wu, B., Lin, B., He, X., & Liu, W. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3341806 · Cited by 335 · see citing works
SPViT: Enabling Faster Vision Transformers via Latency-Aware Soft Token Pruning
Kong, Z., Dong, P., Ma, X., Meng, X., Niu, W., Sun, M., Shen, X., Yuan, G., Ren, B., Tang, H., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20083-0_37 · Cited by 169 · see citing works
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154 · see citing works
MiniViT: Compressing Vision Transformers with Weight Multiplexing
Zhang, J., Peng, H., Wu, K., Liu, M., Xiao, B., Fu, J., & Yuan, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01183 · Cited by 130 · see citing works
Self-citationHeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers
Dong, P., Sun, M., Lu, A., Xie, Y., Liu, K., Kong, Z., Meng, X., Li, Z., Lin, X., Fang, Z., et al. (2023) · 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071047 · Cited by 128 · see citing works
Lu, Z., Cheng, R., Jin, Y., Tan, K. C., & Deb, K. (2023) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2022.3233364 · Cited by 122 · see citing works
Neural Architecture Search for Transformers: A Survey
Chitty-Venkata, K. T., Emani, M., Vishwanath, V., & Somani, A. K. (2022) · IEEE Access · doi:10.1109/access.2022.3212767 · Cited by 118 · see citing works
GT-HAD: Gated Transformer for Hyperspectral Anomaly Detection
Lian, J., Wang, L., Sun, H., & Huang, H. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3355166 · Cited by 118 · see citing works
Li, C., Tang, T., Wang, G., Peng, J., Wang, B., Liang, X., & Chang, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01206 · Cited by 109 · see citing works
Jiang, W., Liu, B., Liang, Y., Gao, H., Lin, P., Zhang, D., & Hu, G. (2023) · Applied Energy · doi:10.1016/j.apenergy.2023.122155 · Cited by 106 · see citing works
GLiT: Neural Architecture Search for Global and Local Image Transformer
Chen, B., Li, P., Li, C., Li, B., Bai, L., Chen, L., Sun, M., Yan, J., & Ouyang, W. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00008 · Cited by 100 · see citing works
Less Is More: Pay Less Attention in Vision Transformers
Pan, Z., Zhuang, B., He, H., Liu, J., & Cai, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20099 · Cited by 92 · see citing works
Convolution-Transformer for Image Feature Extraction
Yin, L., Wang, L., Lu, S., Wang, R., Yang, Y., Yang, B., Liu, S., Alsanad, A., AlQahtani, S. A., Yin, Z., et al. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2024.051083 · Cited by 89 · see citing works
ViTAS: Vision Transformer Architecture Search
Su, X. Q., You, S., Xie, J., Zheng, M., Wang, F., Qian, C., Zhang, C., Wang, X., & Xu, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19803-8_9 · Cited by 85 · see citing works
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78 · see citing works
Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space
Chavan, A., Shen, Z., Liu, Z., Liu, Z., Cheng, K.-T., & Xing, E. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00488 · Cited by 72 · see citing works
DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Vision Transformers
Li, C., Wang, G., Wang, B., Liang, X., Li, Z., & Chang, X. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3194044 · Cited by 72 · see citing works
Global Vision Transformer Pruning with Hessian-Aware Saliency
Yang, H., Yin, H., Shen, M., Molchanov, P. A., Li, H. H., & Kautz, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01779 · Cited by 70 · see citing works
Liu, D., Zhao, L., Wang, Y., & Kato, J. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109550 · Cited by 65 · see citing works
Dynamic Spatial Sparsification for Efficient Vision Transformers and Convolutional Neural Networks
Rao, Y., Liu, Z.-Y., Zhao, W.-L., Zhou, J., & Lu, J.-W. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3263826 · Cited by 60 · see citing works
A survey of model compression strategies for object detection
Lyu, Z., Yu, T., Pan, F., Zhang, Y., Luo, J., Zhang, D., Chen, Y., Zhang, B., & Li, G. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17192-x · Cited by 65 · see citing works
Training-free Transformer Architecture Search
Zhou, Q., Sheng, K., Zheng, X., Li, K., Sun, X., Tian, Y., Chen, J., & Ji, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01062 · Cited by 64 · see citing works
Neural Architecture Search Benchmarks: Insights and Survey
Chitty-Venkata, K. T., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · IEEE Access · doi:10.1109/access.2023.3253818 · Cited by 64 · see citing works
DeViT: Decomposing Vision Transformers for Collaborative Inference in Edge Devices
Xu, G., Hao, Z., Luo, Y., Hu, H., An, J., & Mao, S. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3315138 · Cited by 63 · see citing works
Efficient Multimodal Fusion via Interactive Prompting
Li, Y., Quan, R., Zhu, L., & Yang, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00256 · Cited by 60 · see citing works
“In this section, we experiment with automatic fusion structure search via AutoFormer [5].”
MethodNoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers
Liu, Y., Yang, H., Dong, Z., Keutzer, K., Du, L., & Zhang, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01946 · Cited by 60 · see citing works
Pruning Self-Attentions Into Convolutional Layers in Single Path
He, H., Cai, J., Liu, J., Pan, Z., Zhang, J., Tao, D., & Zhuang, B. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3355890 · Cited by 59 · see citing works
You, H., Xiong, Y., Dai, X., Wu, B., Zhang, P., Fan, H., Vajda, P., & Lin, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01387 · Cited by 60 · see citing works
Multi-scale convolution enhanced transformer for multivariate long-term time series forecasting
Li, Y., Xu, Y., Xu, Y., Li, X., & Zhang, C. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106745 · Cited by 57 · see citing works
Kim, G., Choi, J. G., & Lim, S. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108475 · Cited by 56 · see citing works
Taloma, R. L., Cuomo, F., Comminiello, D., & Pisani, P. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11093-7 · Cited by 53 · see citing works
Res-trans networks for lung nodule classification
Liu, D., Liu, F., Tie, Y., Qi, L., & Wang, F. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02576-5 · Cited by 51 · see citing works
EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm
Zhang, J., Li, X., Wang, Y., Wang, C., Yang, Y., Liu, Y., & Tao, D. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02034-6 · Cited by 53 · see citing works
Satellite Video Object Tracking Based on Location Prompts
Wang, J., Liu, F., Jiao, L., Gao, Y., Wang, H., Li, L., Chen, P., Liu, X., & Li, S. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3358549 · Cited by 44 · see citing works
Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting From Multimodal Data
Nguyen, H. H., Blaschko, M. B., Saarakkala, S. S., & Tiulpin, A. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3312524 · Cited by 41 · see citing works
Two-stage fine-grained image classification model based on multi-granularity feature fusion
Xu, Y., Wu, S., Yang, M., Yang, M., Wu, Z., Yao, Y., & Wei, Z. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.110042 · Cited by 41 · see citing works
Wang, R., Bai, Q., Ao, J., Zhou, L., Xiong, Z., Wei, Z., Zhang, Y., Ko, T., & Li, H. (2022) · Interspeech 2022 · doi:10.21437/interspeech.2022-10269 · Cited by 40 · see citing works
BViT: Broad Attention-Based Vision Transformer
Li, N., Chen, Y., Li, W., Ding, Z., Zhao, D., & Nie, S. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3264730 · Cited by 38 · see citing works
STDAN: Deformable Attention Network for Space-Time Video Super-Resolution
Wang, H., Xiang, X., Tian, Y., Yang, W., & Liao, Q. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3243029 · Cited by 36 · see citing works
Dynamic Mixture of Counter Network for Location-Agnostic Crowd Counting
Wang, M., Cai, H., Dai, Y., & Gong, M. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00025 · Cited by 35 · see citing works
Automated Progressive Learning for Efficient Training of Vision Transformers
Li, C., Zhuang, B., Wang, G., Liang, X., Chang, X., & Yang, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01216 · Cited by 34 · see citing works
Ying, W., Zheng, Q., Wu, Y., Yang, K., Zhou, Z., Chen, J., & Ye, Z. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110869 · Cited by 32 · see citing works
ADNNet: Attention-based deep neural network for Air Quality Index prediction
Wu, X., Gu, X., & See, K. W. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125128 · Cited by 32 · see citing works
AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search
Lee, J., & Ham, B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00563 · Cited by 31 · see citing works
TS-Fastformer: Fast Transformer for Time-series Forecasting
Lee, S., Hong, J., Liu, L., & Choi, W. (2023) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3630637 · Cited by 31 · see citing works
Auto-Prox: Training-Free Vision Transformer Architecture Search via Automatic Proxy Discovery
Wei, Z., Dong, P., Hui, Z., Li, A., Li, L., Lu, M., Pan, H., & Li, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29511 · Cited by 30 · see citing works
Tang, Z., Zhang, J., Hu, M., Ning, Z., Shi, J., Zhai, R., Liu, C., Zhang, J., & Wang, G. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131923 · Cited by 30 · see citing works
DeepWind: a heterogeneous spatio-temporal model for wind forecasting
Wang, B., Shi, J., Tan, B., Ma, M., Hong, F., Yu, Y., & Li, T. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111385 · Cited by 28 · see citing works
NViT: Vision Transformer Compression and Parameter Redistribution.Preprint
Yang, H., Yin, H., Molchanov, P. A., Li, H., & Kautz, J. (2021) · arXiv (Cornell University) · Cited by 28 · see citing works
KMT-PLL: K-Means Cross-Attention Transformer for Partial Label Learning
Fan, J., Huang, L., Gong, C., You, Y., Gan, M., & Wang, Z. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3347792 · Cited by 27 · see citing works
From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks
Geng, X., Wang, Z., Chen, C., Xu, Q., Xu, K., Jin, C., Gupta, M., Yang, X., Chen, Z., Sabry Aly, M. M., et al. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3394494 · Cited by 26 · see citing works
Ma, S., Nie, J., Guan, S., He, Z., & Gao, M. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3443910 · Cited by 25 · see citing works
Convolution-Enhanced Vision Transformer Network for Smoke Recognition
Cheng, G., Zhou, Y., Gao, S., Li, Y., & Yu, H. (2023) · Fire Technology · doi:10.1007/s10694-023-01378-8 · Cited by 25 · see citing works
ViT-MVT: A Unified Vision Transformer Network for Multiple Vision Tasks
Xie, T., Dai, K., Jiang, Z., Li, R., Mao, S., Wang, K., & Zhao, L. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3342141 · Cited by 26 · see citing works
TFormer: A Transmission-Friendly ViT Model for IoT Devices
Lu, Z., Ding, C., Juefei-Xu, F., Boddeti, V. N., Wang, S., & Yang, Y. (2023) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3222765 · Cited by 24 · see citing works
Poly-PC: A Polyhedral Network for Multiple Point Cloud Tasks at Once
Xie, T., Wang, S., Wang, E. K., Yang, L., Jiang, Z., Zhang, X., Dai, K., Li, R., & Cheng, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00125 · Cited by 24 · see citing works
Elegans-AI: How the connectome of a living organism could model artificial neural networks
Bardozzo, F., Terlizzi, A., Simoncini, C., Lió, P., & Tagliaferri, R. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127598 · Cited by 24 · see citing works
A multi-task spatio-temporal fusion network for offshore wind power ramp events forecasting
Song, W., Yan, J., Han, S., Liu, S., Wang, H., Dai, Q., Huo, X., & Yongqian, L. I. U. (2024) · Renewable Energy · doi:10.1016/j.renene.2024.121774 · Cited by 23 · see citing works
Chen, X.-X., Liu, C.-S., Xia, H., & Chi, Z. (2025) · Control Engineering Practice · doi:10.1016/j.conengprac.2024.106165 · Cited by 21 · see citing works
Generalized Global Ranking-Aware Neural Architecture Ranker for Efficient Image Classifier Search
Guo, B., Chen, T., He, S., Liu, H., Xu, L., Ye, P., & Chen, J. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548149 · Cited by 23 · see citing works
A survey of visual neural networks: current trends, challenges and opportunities
Feng, P., & Tang, Z. (2022) · Multimedia Systems · doi:10.1007/s00530-022-01003-8 · Cited by 23 · see citing works
Joint Graph Learning and Matching for Semantic Feature Correspondence
Liu, H., Wang, T., Li, Y., Lang, C., Jin, Y., & Ling, H. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109059 · Cited by 22 · see citing works
Self-citationTang, C., Zhang, L. L., Jiang, H., Xu, J., Cao, T., Zhang, Q., Yang, Y., Wang, Z., & Yang, M. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00536 · Cited by 25 · see citing works
Knowledge Guided Evolutionary Transformer for Remote Sensing Scene Classification
Zhao, J., Jiao, L., Wang, C., Liu, X., Liu, F., Li, L., Ma, M., & Yang, S. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3407138 · Cited by 23 · see citing works
TimeRAG: Boosting LLM Time Series Forecasting via Retrieval-Augmented Generation
Yang, S., Wang, D., Zheng, H., & Jin, R. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889933 · Cited by 21 · see citing works
PIT: Optimization of Dynamic Sparse Deep Learning Models via Permutation Invariant Transformation
Zheng, N., Jiang, H., Zhang, Q., Han, Z., Ma, L., Yang, Y., Yang, F., Zhang, C., Qiu, L., Yang, M., et al. (2023) · ACM SIGOPS Symposium on Operating Systems Principles (SOSP) · doi:10.1145/3600006.3613139 · Cited by 23 · see citing works
Lu, X., Suganuma, M., & Okatani, T. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00116 · Cited by 23 · see citing works
An overview of transformers for video anomaly detection
Dilek, E., & Dener, M. (2025) · Neural computing & applications (Print) · doi:10.1007/s00521-025-11218-1 · Cited by 22 · see citing works
Multi-Dimensional Vision Transformer Compression via Dependency Guided Gaussian Process Search
Hou, Z., & Kung, S.-Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00411 · Cited by 20 · see citing works
NAS-PED: Neural Architecture Search for Pedestrian Detection
Tang, Y., Liu, M., Li, B., Wang, Y., & Ouyang, W. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3507918 · Cited by 19 · see citing works
Shen, Z., Liu, Z., & Xing, E. P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20053-3_42 · Cited by 19 · see citing works
SRDD: a lightweight end-to-end object detection with transformer
Zhu, Y., Xia, Q., & Jin, W. (2022) · Connection Science · doi:10.1080/09540091.2022.2125499 · Cited by 18 · see citing works
Training-Free Transformer Architecture Search With Zero-Cost Proxy Guided Evolution
Zhou, Q., Sheng, K., Zheng, X., Li, K., Tian, Y., Chen, J., & Ji, R. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3378781 · Cited by 18 · see citing works
Dual Cross-Attention Transformer Networks for Temporal Predictive Modeling of Industrial Process
Wang, J., Xie, Y., Xie, S., & Chen, X. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3385820 · Cited by 18 · see citing works
Sun, H., He, X., Li, H., Kong, J., Qiao, M., Cheng, X., Li, P., Zhang, J., Liu, R., & Shang, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127347 · Cited by 18 · see citing works
MDL-NAS: A Joint Multi-domain Learning Framework for Vision Transformer
Wang, S., Xie, T., Cheng, J., Zhang, X., & Liu, H. (2023) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52729.2023.01924 · Cited by 12 · see citing works
Gao, S., Liu, Y., Wang, J., Wang, Z., Xu, W., Yue, R., Cui, R., Liu, Y., & Fan, X. (2024) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2024.110349 · Cited by 18 · see citing works
HybridBranchNet: A novel structure for branch hybrid convolutional neural networks architecture
Parcham, E., & Fateh, M. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.05.025 · Cited by 17 · see citing works
GAIA-Universe: Everything is Super-Netify
Peng, J., Chang, Q., Yin, H., Bu, X., Sun, J., Xie, L., Zhang, X., Tian, Q. C., & Zhang, Z. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3276392 · Cited by 16 · see citing works
Multiscale Spatial-Channel Transformer Architecture Search for Remote Sensing Image Change Detection
Zhang, M., Liu, L., Lei, Z., Ma, K., Feng, J., Liu, Z., & Jiao, L. (2023) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2023.3347765 · Cited by 16 · see citing works
Zhang, S., Zhu, C., & Guo, X. (2024) · Energies · doi:10.3390/en17091996 · Cited by 16 · see citing works
Yuan, L., Wang, H., & Zhang, F. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125588 · Cited by 16 · see citing works
UFO: Unified Feature Optimization
Xi, T., Sun, Y., Yu, D., Li, B., Peng, N., Zhang, G., Zhang, X., Wang, Z., Chen, J., Wang, J., et al. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19809-0_27 · Cited by 16 · see citing works
A comprehensive review of lightweight deep learning models for edge computing with future directions
Lai, N., Dewi, D. A., Maidin, S. S., Xiao, W., Zhao, S., & Hu, Q. (2026) · Discover Computing · doi:10.1007/s10791-026-10021-3 · Cited by 20 · see citing works
Cai, K., Zhang, X., Zhang, M., Ge, Q., Li, S., Qiao, B., & Liu, Y. (2023) · Sustainable Environment Research · doi:10.1186/s42834-023-00175-w · Cited by 15 · see citing works
A frequency domain-based machine learning architecture for short-term wave height forecasting
Zhan, K., Li, C., & Zhu, R. (2023) · Ocean Engineering · doi:10.1016/j.oceaneng.2023.115844 · Cited by 15 · see citing works
MoBoo: Memory-Boosted Vision Transformer for Class-Incremental Learning
Ni, B., Nie, X., Zhang, C., Xu, S., Zhang, X., Meng, G., & Xiang, S. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3417431 · Cited by 16 · see citing works
Auto-Spikformer: Spikformer architecture search
Che, K.-W., Zhou, Z., Niu, J., Ma, Z., Fang, W., Chen, Y.-Q., Shen, S., Yuan, L., & Tian, Y. (2023) · Frontiers in Neuroscience · doi:10.3389/fnins.2024.1372257 · Cited by 18 · see citing works
Aeroelastic force prediction via temporal fusion transformers
Montoya, M. C., Mishra, A., Verma, S., Mures, O. A., & Rubio-Medrano, C. E. (2024) · Computer-Aided Civil and Infrastructure Engineering · doi:10.1111/mice.13381 · Cited by 15 · see citing works
Remote Sensing Video Tracking: Current Status, Challenges, and Future
Liu, F., Wang, J., Jiao, L., Zhang, J., Wang, H., Li, S., Li, L., Chen, P., Liu, X., Ma, W., et al. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3573572 · Cited by 15 · see citing works
- Authors
- Chen, M., Peng, H., Fu, J., & Ling, H.
- Affiliations
- Stony Brook University (US)
- Microsoft Research Asia (China) (CN)
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 12250-12260
- OpenAlex
- W3204801262
- Cited by
- 323
- References
- 78
- Citations by year
- 2026: 28 · 2025: 75 · 2024: 103 · 2023: 79 · 2022: 33 · 2021: 5
- FWCI
- 17.99
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- vision transformer, image classification, transferability, knowledge distillation, Autoformer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
