The DOI matches its Crossref record; 79 of the 84 references it cites were found in a registry or an index.
SPViT: Enabling Faster Vision Transformers via Latency-Aware Soft Token Pruning
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Some pruning methods dynamically vary the number of tokens (e.g., Kong et al. (2022)). This increases accuracy but is generally impractical, as it prevents batched inference or training without padding tokens.”
From Token Merging: Your ViT But Faster (2022) · cites this paper as Kong et al. (2022)
In the citing paper
“We omit DeiT-Ti results from the main paper because we are not able to reproduce the baseline accuracy in SPViT (Kong et al., 2022).”
From Token Merging: Your ViT But Faster (2022) · cites this paper as Kong et al., 2022
In the citing paper
“Pruning methods require token padding during training, and thus don't improve training speed, while ToMe at r = 13 trains 1.5 × faster than the baseline DeiT-S.”
From Token Merging: Your ViT But Faster (2022) · cites this paper as Kong et al., 2022
In the citing paper
“Moreover, most token pruning works are forced to use padding tokens or attention masking during training, negating the benefits of pruning in the first place.”
From Token Merging: Your ViT But Faster (2022) · cites this paper as Kong et al., 2022
In the citing paper
“In vision transformers, it has primarily been used to reduce computational cost by removing visually redundant tokens [12, 13, 31, 49, 73, 76, 85, 111].”
From Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality (2025) · cites this paper as [73]
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Squeeze-and-Excitation Networks
Hu, J., Shen, L., & Sun, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00745 · Cited by 35,667
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Advances in neural information processing systems 7
Cios, K. J., & Shields, M. E. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,093
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
Proceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,865
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259 · cited 4×
“We compare our method with several representative methods including DynamicViT [69], IA-RED2 [64], RegNetY [67], CrossViT [7], VTP [112], ATS [28], CvT [85], PVT [83], T2T-ViT [104], UP-DeiT [95], PS-ViT [78], Evo-ViT [89], TNT [34], HVT [65], Swin [49], CoaT [88], CPVT [18], EViT [46], UVC [97], MD-DeiT [39],and S2Vi…”
BackgroundMethodInfluentialRethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008 · cited 1×
“former (ViT), which solely exploits the self-attention mechanism that inherits from the Transformer architecture, has set up many state-of-the-art (SOTA) records in image classifications [22,79,7], object detection [3,19,1,60], tracking [15,91,59], semantic segmentation [110,16], depth estimation [94,45], image retrie…”
BackgroundSelf-citationZhao, H., Jiang, L., Jia, J., Torr, P. H. S., & Koltun, V. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01595 · Cited by 3,005
Research works printed but not linked: 33
- Bao, H., Dong, L., Piao, S., Wei, F.: BEit: BERT pre-training of image transformers. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=p-BhZSz59o4
- Chen, B., et al.: Psvit: better vision transformer via token pooling and attention sharing. arXiv preprint arXiv:2108.03428 (2021) · arXiv
- Chen, P., Chen, Y., Liu, S., Yang, M., Jia, J.: Exploring and improving mobile level vision transformers. arXiv preprint arXiv:2108.13015 (2021) · arXiv
- Chen, T., Chen, X., Ma, X., Wang, Y., Wang, Z.: Coarsening the granularity: towards structurally sparse lottery tickets. In: Proceedings of the International Conference on Machine Learning (ICML) (2022)
- Chen, T., Saxena, S., Li, L., Fleet, D.J., Hinton, G.: Pix2seq: a language modeling framework for object detection. arXiv preprint arXiv:2109.10852 (2021) · arXiv
- Chen, X., Hsieh, C.J., Gong, B.: When vision transformers outperform resnets without pre-training or strong data augmentations. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=LtKcMgGOeLt
- Cheng, B., Schwing, A., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021). https://openreview.net/forum?id=0lz69oI5iZP
- Chu, X., et al.: Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102.10882 (2021) · arXiv
- Dosovitskiy, A., et al.: An image is worth 16 $$\times $$ 16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (2021). https://openreview.net/forum?id=YicbFdNTTy
- El-Nouby, A., Neverova, N., Laptev, I., Jégou, H.: Training vision transformers for image retrieval. arXiv preprint arXiv:2102.05644 (2021) · arXiv
- Fayyaz, M., et al.: Ats: adaptive token sampling for efficient vision transformers. arXiv preprint arXiv:2111.15667 (2021) · arXiv
- Gao, P., Lu, J., Li, H., Mottaghi, R., Kembhavi, A.: Container: context aggregation network. arXiv preprint arXiv:2106.01401 (2021) · arXiv
- Jia, D., et al.: Efficient vision transformers via fine-grained manifold distillation. arXiv preprint arXiv:2107.01378 (2021) · arXiv
- Jiang, Z., et al.: All tokens matter: token labeling for training better vision transformers. arXiv preprint arXiv:2104.10858 (2021) · arXiv
- Kornblith, S., Norouzi, M., Lee, H., Hinton, G.: Similarity of neural network representations revisited. In: International Conference on Machine Learning, pp. 3519–3529. PMLR (2019)
- Liang, Y., GE, C., Tong, Z., Song, Y., Wang, J., Xie, P.: EVit: expediting vision transformers via token reorganizations. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=BjyvwnXXVn_
- Liu, N., et al.: Lottery ticket preserves weight correlation: is it desirable or not? In: International Conference on Machine Learning (ICML), pp. 7011–7020. PMLR (2021)
- Liu, Y., Sangineto, E., Bi, W., Sebe, N., Lepri, B., De Nadai, M.: Efficient training of visual transformers with small-size datasets. arXiv preprint arXiv:2106.03746 (2021) · arXiv
- Ma, X., et al.: Sanity checks for lottery tickets: Does your winning ticket really win the jackpot? In: Advances in Neural Information Processing Systems (NeurIPS) 34 (2021)
- Pan, B., Jiang, Y., Panda, R., Wang, Z., Feris, R., Oliva, A.: Ia-red$$^2$$: Interpretability-aware redundancy reduction for vision transformers. In: Advances in Neural Information Processing Systems (2021)
- Prillo, S., Eisenschlos, J.: Softsort: a continuous relaxation for the argsort operator. In: International Conference on Machine Learning, pp. 7793–7802. PMLR (2020)
- Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., Dosovitskiy, A.: Do vision transformers see like convolutional neural networks? arXiv preprint arXiv:2108.08810 (2021) · arXiv
- Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., Hsieh, C.J.: Dynamicvit: efficient vision transformers with dynamic token sparsification. In: Advances in Neural Information Processing Systems (2021)
- Renggli, C., Pinto, A.S., Houlsby, N., Mustafa, B., Puigcerver, J., Riquelme, C.: Learning to merge tokens in vision transformers. arXiv preprint arXiv:2202.12015 (2022) · arXiv
- Ryoo, M.S., Piergiovanni, A., Arnab, A., Dehghani, M., Angelova, A.: Tokenlearner: what can 8 learned tokens do for images and videos? In: Advances in Neural Information Processing Systems (2021)
- Sanh, V., Wolf, T., Rush, A.M.: Movement pruning: adaptive sparsity by fine-tuning. arXiv preprint arXiv:2005.07683 (2020) · arXiv
- Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., Beyer, L.: How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270 (2021) · arXiv
- Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., J’egou, H.: Training data-efficient image transformers & distillation through attention. In: ICML (2021)
- Wu, B., et al.: Visual transformers: token-based image representation and processing for computer vision. arXiv preprint arXiv:2006.03677 (2020) · arXiv
- Yu, H., Wu, J.: A unified pruning framework for vision transformers. arXiv preprint arXiv:2111.15127 (2021) · arXiv
- Yu, Q., Xia, Y., Bai, Y., Lu, Y., Yuille, A., Shen, W.: Glance-and-gaze vision transformer. In: Advances in Neural Information Processing Systems (2021)
- Yuan, L., et al.: Tokens-to-token vit: training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986 (2021) · DOI · Paper page
- Zhu, M., Han, K., Tang, Y., Wang, Y.: Visual transformer pruning. In: KDD 2021 Workshop on Model Mining (2021)
Cited by
Sorted by citations
TinyViT: Fast Pretraining Distillation for Small Vision Transformers
Wu, K., Zhang, J., Peng, H., Liu, M., Xiao, B., Fu, J., & Yuan, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19803-8_5 · Cited by 521
A Survey on Efficient Vision Transformers: Algorithms, Techniques, and Performance Benchmarking
Papa, L., Russo, P., Amerini, I., & Zhou, L. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3392941 · Cited by 200
Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., & Chang, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73004-7_2 · Cited by 359
Less is More: Focus Attention for Efficient DETR
Zheng, D., Dong, W., Hu, H., Chen, X., & Wang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00614 · Cited by 162
Self-citationHeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers
Dong, P., Sun, M., Lu, A., Xie, Y., Liu, K., Kong, Z., Meng, X., Li, Z., Lin, X., Fang, Z., et al. (2023) · 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca56546.2023.10071047 · Cited by 128
Self-citation
Metadata
- Authors
- Kong, Z., Dong, P., Ma, X., Meng, X., Niu, W., Sun, M., Shen, X., Yuan, G., Ren, B., Tang, H., Qin, M., & Wang, Y.
- Affiliations
- Northeastern University (US)
- Clemson University (US)
- Peking University (CN)
- William & Mary (US)
- ETH Zurich (CH)
- Corresponding author
- Wang, Yanzhi
- ORCID
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer Nature Switzerland
- Published
- 2022
- Pages
- 620-640
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W4312910119
- Cited by
- 169
- References
- 84
- Citations by year
- 2026: 25 · 2025: 61 · 2024: 55 · 2023: 25 · 2022: 3
- FWCI
- 13.06
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Lecture notes in computer science (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- vision transformer, token selection, multi-head attention, edge deployment, mobile devices, FPGA, real-time inference, DeiT, Swin Transformer, computation cost reduction, accuracy-latency trade-off
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
