Pvt v2: Improved baselines with pyramid vision transformer
Open accessAbstract
Transformers have recently lead to encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs: (i) a linear complexity attention layer, (ii) an overlapping patch embedding, and (iii) a convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linearity and provides significant improvements on fundamental vision tasks such as classification, detection, and segmentation. In particular, PVT v2 achieves comparable or better performance than recent work such as the Swin transformer. We hope this work will facilitate state-of-the-art transformer research in computer vision. Code is available at https://github.com/whai362/PVT .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
PVTv2 can process images of any size and is computationally efficient even with high-resolution inputs.
“Additionally, PVTv2 can process images of any size, showcasing computational efficiency even with high-resolution inputs [18].”
From Pose Estimation for Cross-Domain Non-Cooperative Spacecraft Based on Spatial-Aware Keypoints Regression (2024) · cites this paper as [18]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1704.04861 · Cited by 19,015 · see citing works
Self-citation(2010) · Cited by 14,134 · see citing works
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · see citing works
Proceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,869 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556 · see citing works
Cascade R-CNN: Delving Into High Quality Object Detection
Cai, Z., & Vasconcelos, N. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00644 · Cited by 7,370 · see citing works
mixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831 · see citing works
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309 · see citing works
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · see citing works
Self-citationProceedings of the 21st International Conference on Neural Information Processing Systems
Koller, D., Schuurmans, D., Bengio, Y., & Bottou, L. (2008) · Cited by 4,461 · see citing works
Random Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7000 · Cited by 3,617 · see citing works
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · see citing works
Scene Parsing through ADE20K Dataset
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., & Torralba, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.544 · Cited by 3,326 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W. F., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206848 · Cited by 2,714 · see citing works
CvT: Introducing Convolutions to Vision Transformers
Wu, H., Xiao, B., Codella, N., Liu, M., Dai, X., Yuan, L., & Zhang, L. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00009 · Cited by 2,649 · see citing works
Zhang, S., Chi, C., Yao, Y., Lei, Z., & Li, S. Z. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00978 · Cited by 2,531 · see citing works
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415 · see citing works
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification
Chen, C.-F. R., Fan, Q., & Panda, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00041 · Cited by 2,242 · see citing works
MMDetection: Open MMLab Detection Toolbox and Benchmark
Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.07155 · Cited by 2,227 · see citing works
Self-citationDesigning Network Design Spaces
Radosavovic, I., Kosaraju, R. P., Girshick, R., He, K., & Dollar, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01044 · Cited by 2,084 · see citing works
Proceedings of the 19th International Conference on Artificial Intelligence and Statistics 2016
Murray, I., & Graham, M. (2016) · Cited by 2,079 · see citing works
Sparse R-CNN: End-to-End Object Detection with Learnable Proposals
Sun, P., Zhang, R., Jiang, Y., Kong, T., Xu, C., Zhan, W., Tomizuka, M., Li, L., Yuan, Z., Wang, C., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01422 · Cited by 1,617 · see citing works
Self-citationPanoptic Feature Pyramid Networks
Kirillov, A., Girshick, R., He, K., & Dollár, P. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00656 · Cited by 1,460 · see citing works
Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., & Wang, Y. (2021) · arXiv · doi:10.48550/arxiv.2103.00112 · Cited by 1,394 · see citing works
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference
Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jegou, H., & Douze, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01204 · Cited by 935 · see citing works
Twins: Revisiting the Design of Spatial Attention in Vision Transformers
Chu, X., Tian, Z., Wang, Y., Zhang, B., Ren, H., Wei, X., Xia, H., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2104.13840 · Cited by 817 · see citing works
Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear UnitsPreprint
Hendrycks, D., & Gimpel, K. (2016) · arXiv (Cornell University) · Cited by 753 · see citing works
Conditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489 · cited 2× · see citing works
“CPVT [6] replaces the fixed size position embedding in ViT with conditional position encodings, making it easier to process images of arbitrary resolution.”
BackgroundMethodXiang, L., Wang, W., Hu, X., Li, J., Tang, J., & Yang, J. J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01146 · Cited by 476 · see citing works
Self-citationCo-Scale Conv-Attentional Image Transformers
Xu, W., Xu, Y., Chang, T. A., & Tu, Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00983 · Cited by 425 · see citing works
LocalViT: Analyzing Locality in Vision TransformersPreprint
Li, Y., Zhang, K., Cao, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05707 · Cited by 279 · see citing works
How Much Position Information Do Convolutional Neural Networks Encode?
Islam, M. A., Jia, S., & Bruce, N. D. B. (2020) · arXiv · doi:10.48550/arxiv.2001.08248 · Cited by 253 · see citing works
Polyp-PVT: Polyp Segmentation with Pyramid Vision TransformersPreprint
Dong, B., Wang, W., Fan, D.-P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.06932 · Cited by 172 · see citing works
Self-citationCvT: Introducing Convolutions to Vision TransformersPreprint
Wu, H., Xiao, B., Codella, N., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.15808 · Cited by 165 · see citing works
Sparse R-CNN: End-to-End Object Detection with Learnable ProposalsPreprint
Sun, P., Zhang, R., Jiang, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.12450 · Cited by 103 · see citing works
Do We Really Need Explicit Position Encodings for Vision TransformersPreprint
Chu, X., Zhang, B., Tian, Z., Wei, X. K., & Xia, H. (2021) · arXiv (Cornell University) · Cited by 89 · see citing works
LeViT: a Vision Transformer in ConvNet's Clothing for Faster InferencePreprint
Graham, B. V., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jeǵou, H., & Douze, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01136 · Cited by 88 · see citing works
Twins: Revisiting the design of spatial attention in vision transformersPreprint
Chu, X. (2024) · arXiv (Cornell University) · doi:10.57702/b8b2svfj · Cited by 39 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- He, K. M.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, 2980-2988, 2017.
Cited by
Sorted by citations
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · see citing works
“This is a worth exploring direction, we recently propose a solu-tion PVTv2 [67].”
BackgroundSelf-citationInternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01385 · Cited by 1,080 · see citing works
Self-citationEfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention
Liu, X., Peng, H., Zheng, N., Yang, Y., Hu, H., & Yuan, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01386 · Cited by 956 · see citing works
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02283 · Cited by 734 · see citing works
Self-citationSalient Object Detection via Integrity Learning
Zhuge, M., Fan, D.-P., Liu, N., Zhang, D., Xu, D., & Shao, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3179526 · Cited by 403 · see citing works
Self-citationSemantic segmentation using Vision Transformers: A survey
Thisanke, H., Deshan, C., Chamith, K., Seneviratne, S., Vidanaarachchi, R., & Herath, D. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106669 · Cited by 288 · see citing works
Transformer for object detection: Review and benchmark
Li, Y., Miao, N., Ma, L., Shuang, F., & Huang, X. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107021 · Cited by 135 · see citing works
SAM struggles in concealed scenes — empirical study on “Segment Anything”
Ji, G.-P., Fan, D.-P., Xu, P., Zhou, B., Cheng, M.-M., & Van Gool, L. (2023) · Science China Information Sciences · doi:10.1007/s11432-023-3881-x · Cited by 101 · see citing works
Self-citationPVSPE: A pyramid vision multitask transformer network for spacecraft pose estimation
Yang, H., Xiao, X., Yao, M., Xiong, Y., Cui, H., & Fu, Y. (2024) · Advances in Space Research · doi:10.1016/j.asr.2024.05.011 · Cited by 41 · see citing works
Vision Transformers: From Semantic Segmentation to Dense Prediction
Zhang, L., Lu, J., Zheng, S., Zhao, X., Zhu, X., Fu, Y., Xiang, T., Feng, J., & Torr, P. H. S. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02173-w · Cited by 23 · see citing works
“With a linear embedding layer applied to the flattened pixel vectors of every patch, we then obtain a sequence of feature embedding vectors as the input to a ViT.”
PixArt-$α$: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis
Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.00426 · Cited by 30 · see citing works
Self-citationQiu, Y., Lu, H., Mei, J., Bao, S., & Xu, J. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124734 · Cited by 5 · see citing works
Wang, Z., Liu, Y., & Zhang, E. (2024) · Aerospace · doi:10.3390/aerospace11110948 · Cited by 5 · see citing works
- Authors
- Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L.
- Affiliations
- Shanghai Artificial Intelligence Laboratory (CN)
- University of Hong Kong (HK)
- Nanjing University of Science and Technology (CN)
- ETH Zurich (CH)
- Sensetime (China) (CN)
- Department of Computer Science and Technology, Nanjing University, Nanjing 210023, China
- Inception Institute of Artificial Intelligence (AE)
- Corresponding author
- Wang, Wenhai
- Venue
- Computational Visual Media
- Publisher
- Tsinghua University Press
- Published
- 2022-09
- Volume
- 8
- Issue
- 3
- Pages
- 415-424
- ISSN
- 2096-0433 (ISSN-L)
- 2096-0662
- OpenAlex
- W3175515048
- Cited by
- 2,439
- References
- 53
- Citations by year
- 2026: 513 · 2025: 728 · 2024: 629 · 2023: 430 · 2022: 125 · 2021: 14
- FWCI
- 156.39
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Computational Visual Media (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- ETH Zürich Research Collection
- Funders
- National Natural Science Foundation of China (61672273, 61832008, BK20160021)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- Pyramid Vision Transformer, vision transformer, computational complexity reduction, image classification, object detection, semantic segmentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
