The DOI matches its Crossref record; 67 of the 67 references it cites were found in a registry or an index.
Swin Transformer V2: Scaling Up Capacity and Resolution
Open accessAbstract
We present techniques for scaling Swin Transformer [35] up to 3 billion parameters and making it capable of training with images of up to 1,536x1,536 resolution. By scaling up capacity and resolution, Swin Transformer sets new records on four representative vision benchmarks: 84.0% top-1 accuracy on ImageNet- V2 image classification, 63.1 / 54.4 box / mask mAP on COCO object detection, 59.9 mIoU on ADE20K semantic segmentation, and 86.8% top-1 accuracy on Kinetics-400 video action classification. We tackle issues of training instability, and study how to effectively transfer models pre-trained at low resolutions to higher resolution ones. To this aim, several novel technologies are proposed: 1) a residual post normalization technique and a scaled cosine attention approach to improve the stability of large vision models; 2) a log-spaced continuous position bias technique to effectively transfer models pre-trained at low-resolution images and windows to their higher-resolution counterparts. In addition, we share our crucial implementation details that lead to significant savings of GPU memory consumption and thus make it feasi-ble to train large vision models with regular GPUs. Using these techniques and self-supervised pre-training, we suc-cessfully train a strong 3 billion Swin Transformer model and effectively transfer it to various vision tasks involving high-resolution images or windows, achieving the state-of-the-art accuracy on a variety of benchmarks. Code is avail-able at https://github.com/microsoft/Swin-Transformer.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Research works printed but not linked: 14
- 2021, Microsoft, Using deepspeed and megatron to train megatron-turing nlg 530b the world's largest and most powerful generative language model
- liang, 2021, Cb-netv2 A composite backbone network architecture for obj ect detection
- li, 2021, Localvit Bringing locality to vision trans-formers
- 2020, Microsoft, Turing-NLG A 17-Billion-Parameter Language Model by Microsoft
- xiong, 2020, On layer normalization in the transformer architecture
- lei ba, 2016, Layer normalization
- touvron, 2020, Training data-efficient image transformers & distillation through attention, ArXiv Preprint
- wu, 0, Group normalization, Proceedings of the European Conference on Computer Vision (ECCV), 3
- vaswani, 2017, Attention is all you need, Advances in neural information processing systems, 5998
- radford, 2019, Language Models are Unsupervised Multitask Learners
- brock, 2021, Characterizing signal propagation to close the performance gap in unnormalized resnets, ArXiv Preprint
- cheng, 2021, Per-pixel classification is not all you need for semantic segmentation, ar Xiv
- ryoo, 2021, Tokenlearner What can 8 learned tokens do for images and videos?
- riquelme, 2021, Scaling Vision with Sparse Mixture of Experts
Cited by
YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00721 · Cited by 13,198
YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2022) · arXiv · doi:10.48550/arxiv.2207.02696 · Cited by 13,205
YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information
Wang, C.-Y., Yeh, I.-H., & Mark Liao, H.-Y. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72751-1_1 · Cited by 4,823
Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks
Chen, J., Kao, S.-H., He, H., Zhuo, W., Wen, S., Lee, C.-H., & Chan, S.-H. G. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01157 · Cited by 2,343
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs
Ding, X., Zhang, X., Han, J., & Ding, G. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01166 · Cited by 1,447
Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data
Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., & Zhao, H. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00987 · Cited by 1,296
SimMIM: a Simple Framework for Masked Image Modeling
Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., & Hu, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00943 · Cited by 1,255
Self-citationInternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01385 · Cited by 1,080
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention
Liu, X., Peng, H., Zheng, N., Yang, Y., Hu, H., & Yuan, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01386 · Cited by 956
Self-citationExploring Plain Vision Transformer Backbones for Object Detection
Li, Y., Mao, H., Girshick, R., & He, K. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20077-9_17 · Cited by 809
DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection
Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L. M., & Shum, H.-Y. (2022) · arXiv · Cited by 1,154
“Without using any other tricks, we achieve the result of 63.1 on val2017 and 63.2 on test-dev without test time augmentation (TTA) (see Appendix A), outperforming the previous state-of-the-art result 63.1 achieved by SwinV2 [22] with a much neater solution.”
BackgroundMethodResultInfluentialRep ViT: Revisiting Mobile CNN From ViT Perspective
Wang, A., Chen, H., Lin, Z., Han, J., & Ding, G. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01506 · Cited by 752
Pinto-Coelho, L. (2023) · Bioengineering · doi:10.3390/bioengineering10121435 · Cited by 676
Point Transformer V3: Simpler, Faster, Stronger
Wu, X., Jiang, L., Wang, P.-S., Liu, Z., Liu, X., Qiao, Y., Ouyang, W., He, T., & Zhao, H. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00463 · Cited by 648
Self-citationDETRs with Collaborative Hybrid Assignments Training
Zong, Z., Song, G., & Liu, Y. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00621 · Cited by 592
Masked Feature Prediction for Self-Supervised Visual Pre-Training
Wei, C., Fan, H., Xie, S., Wu, C.-Y., Yuille, A., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01426 · Cited by 587
Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation
Li, F., Zhang, H., Xu, H., Liu, S., Zhang, L., Ni, L. M., & Shum, H.-Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00297 · Cited by 560
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends
Gui, J., Chen, T., Zhang, J., Cao, Q., Sun, Z., Luo, H., & Tao, D. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3415112 · Cited by 552
Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks
Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O. K., Singhal, S., Som, S., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01838 · Cited by 561
Self-citationVideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking
Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., & Qiao, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01398 · Cited by 536
EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., & Cao, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01855 · Cited by 506
Self-citationOneFormer: One Transformer to Rule Universal Image Segmentation
Jain, J., Li, J., Chiu, M., Hassani, A., Orlov, N., & Shi, H. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00292 · Cited by 504
TinyViT: Fast Pretraining Distillation for Small Vision Transformers
Wu, K., Zhang, J., Peng, H., Liu, M., Xiao, B., Fu, J., & Yuan, L. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19803-8_5 · Cited by 426
TransNeXt: Robust Foveal Visual Perception for Vision Transformers
Shi, D. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01683 · Cited by 423
FuXi: a cascade machine learning forecasting system for 15-day global weather forecast
Chen, L., Zhong, X., Zhang, F., Cheng, Y., Xu, Y., Qi, Y., & Li, H. (2023) · npj Climate and Atmospheric Science · doi:10.1038/s41612-023-00512-1 · Cited by 367
MambaVision: A Hybrid Mamba-Transformer Vision Backbone
Hatamizadeh, A., & Kautz, J. (2025) · 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52734.2025.02352 · Cited by 358
Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering
Liu, Y., Li, G., & Lin, L. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3284038 · Cited by 348
MedNeXt: Transformer-Driven Scaling of ConvNets for Medical Image Segmentation
Roy, S., Koehler, G., Ulrich, C., Baumgartner, M., Petersen, J., Isensee, F., Jäger, P. F., & Maier-Hein, K. H. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43901-8_39 · Cited by 347
Rethinking Vision Transformers for MobileNet Size and Speed
Li, Y., Hu, J., Wen, Y., Evangelidis, G., Salahi, K., Wang, Y., Tulyakov, S., & Ren, J. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01549 · Cited by 329
Context Autoencoder for Self-supervised Representation Learning
Chen, X., Ding, M., Wang, X., Xin, Y., Mo, S., Wang, Y., Han, S., Luo, P., Zeng, G., & Wang, J. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-023-01852-4 · Cited by 319
Object Detection Using Deep Learning, CNNs and Vision Transformers: A Review
Amjoud, A. B., & Amrouch, M. (2023) · IEEE Access · doi:10.1109/access.2023.3266093 · Cited by 318
Rethinking Mobile Block for Efficient Attention-based Models
Zhang, J., Li, X., Li, J., Liu, L., Xue, Z., Zhang, B., Jiang, Z., Huang, T., Wang, Y., & Wang, C. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00134 · Cited by 313
Remote Sensing Object Detection in the Deep Learning Era—A Review
Gui, S., Song, S., Qin, R., & Tang, Y. (2024) · Remote Sensing · doi:10.3390/rs16020327 · Cited by 305
Efficient and Explicit Modelling of Image Hierarchies for Image Restoration
Li, Y., Fan, Y., Xiang, X., Demandolx, D., Ranjan, R., Timofte, R., & Van Gool, L. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01753 · Cited by 302
Mukhamediev, R. I., Popova, Y., Kuchin, Y., Zaitseva, E., Kalimoldayev, A., Symagulov, A., Levashenko, V., Abdoldina, F., Gopejenko, V., Yakunin, K., et al. (2022) · Mathematics · doi:10.3390/math10152552 · Cited by 292
VOLO: Vision Outlooker for Visual Recognition
Yuan, L., Hou, Q., Jiang, Z., Feng, J., & Yan, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3206108 · Cited by 288
SwiftFormer: Efficient Additive Attention for Transformer-based Real-time Mobile Vision Applications
Shaker, A., Maaz, M., Rasheed, H., Khan, S., Yang, M.-H., & Khan, F. S. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01598 · Cited by 279
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond
Zhang, Q., Xu, Y., Zhang, J., & Tao, D. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-022-01739-w · Cited by 277
A foundation model for the Earth system
Bodnar, C., Bruinsma, W. P., Lucic, A., Stanley, M., Allen, A., Brandstetter, J., Garvan, P., Riechert, M., Weyn, J. A., Dong, H., et al. (2025) · Nature · doi:10.1038/s41586-025-09005-y · Cited by 267
Lightweight Deep Learning for Resource-Constrained Environments: A Survey
Liu, H.-I., Galindo, M., Xie, H., Wong, L.-K., Shuai, H.-H., Li, Y.-H., & Cheng, W.-H. (2024) · ACM Computing Surveys · doi:10.1145/3657282 · Cited by 259
Scaling Self-Supervised Learning for Histopathology with Masked Image Modeling
Filiot, A., Ghermi, R., Olivier, A., Jacob, P., Fidon, L., Camara, A., Mac Kain, A., Saillard, C., & Schiratti, J.-B. (2023) · doi:10.1101/2023.07.21.23292757 · Cited by 148
Transformer for object detection: Review and benchmark
Li, Y., Miao, N., Ma, L., Shuang, F., & Huang, X. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107021 · Cited by 135
CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flow
Weinzaepfel, P., Lucas, T., Leroy, V., Cabon, Y., Arora, V., Brégier, R., Csurka, G., Antsfeld, L., Chidlovskii, B., & Revaud, J. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01647 · Cited by 110
Token Merging: Your ViT But Faster
Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., & Hoffman, J. (2022) · arXiv · doi:10.48550/arxiv.2210.09461 · Cited by 140
“5, we show the results of applying our method off-the-shelf and during MAE fine-tuning using ViT-L from Spatiotemporal MAE compared to the relevant state-of-the-art on Kinetics-400 classification: Swin (Liu et al., 2022c) pretrained on ImageNet-21k, MViTv2 (Li et al., 2022) pretrained with MaskFeats (Wei et al., 2022)…”
BackgroundMethodInfluentialWang, J., Dai, H., Chen, T., Liu, H., Zhang, X., Zhong, Q., & Lu, R. (2023) · Scientific Reports · doi:10.1038/s41598-023-33804-w · Cited by 46
Yang, Z., Tian, T., Kong, J., & Chen, H. (2025) · Analytical Chemistry · doi:10.1021/acs.analchem.4c06677 · Cited by 46
PAtt-Lite: Lightweight Patch and Attention MobileNet for Challenging Facial Expression Recognition
Ngwe, J. L., Lim, K. M., Lee, C. P., Ong, T. S., & Alqahtani, A. (2024) · IEEE Access · doi:10.1109/access.2024.3407108 · Cited by 45
Exploring Vision-Language Models for Imbalanced Learning
Wang, Y., Yu, Z., Wang, J., Heng, Q., Chen, H., Ye, W., Xie, R., Xie, X., & Zhang, S. (2023) · arXiv · doi:10.48550/arxiv.2304.01457 · Cited by 45
Controllable Guide-Space for Generalizable Face Forgery Detection
Guo, Y., Zhen, C., & Yan, P. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01903 · Cited by 38
COŞKUN, D., KARABOĞA, D., BAŞTÜRK, A., AKAY, B., NALBANTOĞLU, Ö. U., DOĞAN, S., PAÇAL, İ., & KARAGÖZ, M. A. (2023) · Turkish Journal of Electrical Engineering and Computer Sciences · doi:10.55730/1300-0632.4048 · Cited by 38
FuXi-Extreme: Improving extreme rainfall and wind forecasts with diffusion model
Zhong, X., Chen, L., Liu, J., Lin, C., Qi, Y., & Li, H. (2024) · Science China Earth Sciences · doi:10.1007/s11430-023-1427-x · Cited by 37
Yang, Y., Jerger, A., Feng, S., Wang, Z., Brasfield, C., Cheung, M. S., Zucker, J., & Guan, Q. (2024) · Communications Biology · doi:10.1038/s42003-024-07359-z · Cited by 24
Vision Transformers: From Semantic Segmentation to Dense Prediction
Zhang, L., Lu, J., Zheng, S., Zhao, X., Zhu, X., Fu, Y., Xiang, T., Feng, J., & Torr, P. H. S. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02173-w · Cited by 23
Cross-Layer Reliability Evaluation and Efficient Hardening of Large Vision Transformers Models
Roquet, L., Fernandes dos Santos, F., Rech, P., Traiola, M., Sentieys, O., & Kritikakou, A. (2024) · Proceedings of the 61st ACM/IEEE Design Automation Conference · doi:10.1145/3649329.3655688 · Cited by 16
Expediting Large-Scale Vision Transformer for Dense Prediction Without Fine-Tuning
Yuan, Y., Liang, W., Ding, H., Liang, Z., Zhang, C., & Hu, H. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327511 · Cited by 14
Self-citationCREaTor: zero-shot cis-regulatory pattern modeling with attention mechanisms
Li, Y., Ju, F., Chen, Z., Qu, Y., Xia, H., He, L., Wu, L., Zhu, J., Shao, B., & Deng, P. (2023) · Genome Biology · doi:10.1186/s13059-023-03103-8 · Cited by 12
Chen, N., Xu, T., Sun, M., Yao, C., & Yang, D. (2025) · Intelligent Computing · doi:10.34133/icomputing.0143 · Cited by 5
Beyond Accuracy: A Deployment-Oriented Benchmark of Deepfake Detection Models
Ain, Q. U., Ning, H., Philipo, A. G., Daneshmand, M., & Ding, J. (2025) · Preprint (TechRxiv) · doi:10.36227/techrxiv.176315939.91577661/v1 · Cited by 2
Xu, Z., Choi, Y., Yi, C., Park, C., Park, J., Park, H., & Song, S. (2026) · Aerospace · doi:10.3390/aerospace13030222
Exact-Solution Volume and Length Generalization in Transformers
Zhu, Y. J., & Chiang, D. (2026) · arXiv · doi:10.48550/arxiv.2610.07676
“QKNorm ℓ 2 -normalizes complete query and key vectors and learns the attention scale (Henry et al., 2020), while scaled cosine attention likewise removes dependence on their magnitudes (Liu et al., 2022).”
Metadata
- Authors
- Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L., Wei, F., & Guo, B.
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 11999-12009
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4312349930
- Cited by
- 2,484
- References
- 67
- FWCI
- 197.21
- Type
- conference paper
- Language
- en
- Open access
- green
- Funders
- Microsoft
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.