A-ViT: Adaptive Tokens for Efficient Vision Transformer
Abstract
We introduce A - ViT, a method that adaptively adjusts the inference cost of vision transformer (ViT) for images of different complexity. A - ViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds. We refor-mulate Adaptive Computation Time (ACT [17]) for this task, extending halting to discard redundant spatial tokens. The appealing architectural properties of vision transformers enables our adaptive token reduction mechanism to speed up inference without modifying the network architecture or inference hardware. We demonstrate that A - ViT requires no extra parameters or sub-network for halting, as we base the learning of adaptive halting on the original network parameters. We further introduce distributional prior regularization that stabilizes training compared to prior ACT approaches. On the image classification task (ImageNet1K), we show that our proposed A - ViT yields high efficacy in filtering informative spatial features and cutting down on the overall compute. The proposed method improves the throughput of DeiT-Tiny by 62% and DeiT-Small by 38% with only 0.3% accuracy drop, outperforming prior art by a large margin.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several token pruning methods prune different numbers of tokens depending on the input content, making batched inference infeasible.
“and several prune different numbers of tokens depending on the input content, making batched inference infeasible.”
From Token Merging: Your ViT But Faster (2022) · cites this paper as Yin et al., 2022
Most token pruning methods cannot be applied to speed up training.
“most cannot be applied to speed up training;”
From Token Merging: Your ViT But Faster (2022) · cites this paper as Yin et al., 2022
Current token pruning methods require re-training the model to be effective, some with extra parameters.
“current methods require re-training the model to be effective (some with extra parameters);”
From Token Merging: Your ViT But Faster (2022) · cites this paper as Yin et al., 2022
Token pruning's information loss limits how many tokens can reasonably be reduced.
“Yet, token pruning has several disadvantages: the information loss from pruning limits how many tokens you can reasonably reduce;”
From Token Merging: Your ViT But Faster (2022) · cites this paper as Yin et al., 2022
Pruning methods require token padding during training, and thus don't improve training speed.
“Pruning methods require token padding during training, and thus don't improve training speed, while ToMe at r = 13 trains 1.5 × faster than the baseline DeiT-S.”
From Token Merging: Your ViT But Faster (2022) · cites this paper as Yin et al., 2022
A-ViT is not trained from scratch and performs slightly better than DynamicViT in its setting.
“† A-ViT is not trained from scratch and performs slightly better than DynamicViT in its setting.”
From Token Merging: Your ViT But Faster (2022) · cites this paper as Yin et al., 2022
Most token pruning works are forced to use padding tokens or attention masking during training, negating the benefits of pruning in the first place.
“Moreover, most token pruning works are forced to use padding tokens or attention masking during training, negating the benefits of pruning in the first place.”
From Token Merging: Your ViT But Faster (2022) · cites this paper as Yin et al., 2022
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 59 · Cited by 360
- Authors
- Yin, H., Vahdat, A., Alvarez, J. M., Mallya, A., Kautz, J., & Molchanov, P.
- Affiliations
- Nvidia (United Kingdom) (GB)
- NVIDIA
- Venue
- 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2022-06
- Pages
- 10799-10808
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W4312849330
- Cited by
- 360
- References
- 59
- Citations by year
- 2026: 71 · 2025: 117 · 2024: 105 · 2023: 65 · 2022: 2
- FWCI
- 26.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- token reduction, inference efficiency, image classification, DeiT
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
