The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; 2 of the 9 sources its claims cite were checked for support.
FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer
Open accessAbstract
Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers. In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers. In addition, observing an extreme non-uniform distribution in attention maps, we propose Log-Int-Softmax (LIS) to sustain that and simplify inference by using 4-bit quantization and the BitShift operator. Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bit-width on attention maps. For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our knowledge, we are the first to achieve lossless accuracy degradation (~1%) on fully quantized vision transformers. The code is available at https://github.com/megvii-research/FQ-ViT.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
15 Own result · 9 Own interpretation · 6 No basis given · 5 Sources not found · 1 Supported · 1 Partly supported · 1 No full text of the sources · accuracy not yet measured
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Averaging on all attention maps in ViT-L, about 98.8% of values are smaller than 1/16.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Meanwhile, our FQ-ViT significantly exceeds PTQ for ViT [Liu et al., 2021b], whose LayerNorm and Softmax are not quantized.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Image Classification on ImageNet
What the sources say
- Not matchedcites [Liu et al., 2021b]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Firstly, we find a serious interchannel variation of LayerNorm inputs, which some channel ranges even exceed 40× of the median.”
- Also stated in: § 3.2 Power-of-Two Factor for LayerNorm Quantization · plain
“Looking into the inputs of LayerNorm layers, we find there is a serious inter-channel variation.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.2 Power-of-Two Factor for LayerNorm Quantization · plain
In the paper
“Moreover, following the purpose of ranking-aware loss [Liu et al., 2021b], log2 quantization can retain much order consistency between full precision and quantized attention maps.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
What the sources say
- Not matchedcites [Liu et al., 2021b]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results show that our fully quantized vision transformers achieve comparable performance with the full precision models.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 5 Conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Secondly, we find that the values of the attention map have an extreme non-uniform distribution, with most values clustered in 0 ∼ 0.01, and a few high attention values close to 1.”
- Also stated in: § Log2 Quantization for Attention Map · plain
“We observe a distribution centering at a fairly small value, while only a few outliers have larger values close to 1.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Log2 Quantization for Attention Map · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Meanwhile, thanks to the nature of powers of two, PTF α can be efficiently combined with layerwise quantization by BitShift operator, avoiding floatingpoint calculations of group-wise or channel-wise quantization.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Previous work [Liu et al., 2021b] finds there have been a significant accuracy degradation when quantizing LayerNorm and Softmax of vision transformers.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
What the sources say
- Not matchedcites [Liu et al., 2021b]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in Lay-erNorm inputs”
- Also stated in: § DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin · plain
“Firstly, we find a serious interchannel variation of LayerNorm inputs, which some channel ranges even exceed 40× of the median.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Secondly, it also introduces the merits of converting the MatMul to BitShift between the quantized attention map (Attn Q ) and values (V Q ) as:”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 27
Every source the paper cites for this claim was checked and supports it.
In the paper
“Unlike BatchNorm [Ioffe and Szegedy, 2015], commonly used in CNNs, LayerNorm cannot be folded into the previous layer due to its dynamic computational property, so we have to quantize it separately.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
What the sources say
- Supportedcites [Ioffe and Szegedy, 2015] source
The citation identifies BatchNorm as the contrast to LayerNorm, not as the source of the authors’ LayerNorm quantization claim. Section 3.1 (p. 4) says BatchNorm uses fixed statistics at inference and becomes a linear transform; Section 3.2 (p. 5) describes its use in convolutional networks. These passages support the BatchNorm side of the comparison. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A possible solution is using group-wise quantization [Shen et al., 2020] or channel-wise quantization [Li et al., 2019], which assign different quantization parameter to different group or channel.However, these will still induce the calculation of mean and variance in the floating-point domain, resulting in a high hardware overhead.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
What the sources say
- Partly supportedcites [Shen et al., 2020] sourceExperimental — accuracy not yet measured
Section 2.3 (p. 3) says each subgroup can have its own quantization range, supporting the group-wise part of the claim. Section 3.2 (p. 5) warns that more groups require more lookup tables and can adversely affect hardware performance, but does not attribute that cost to floating-point calculations of mean and variance. The mean and variance discussed in Section 2.2 (p. 3) concern Hessian eigenvalues, not group-wise quantization overhead. (LLM, conf=0.88)
- Not matchedcites [Li et al., 2019]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared with 4-bit uniform quantization which only assigns 1 bin for such many values, log2 method can allocate 12 bins for them.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our FQ-ViT significantly improves the quantization accuracy and achieves 47.2 mAP on Mask R-CNN (Swin-S) and 50.8 mAP on Cascade Mask R-CNN (Swin-S) with 8-bit on weights/activations and 4-bit on attention maps.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Object Detection on COCO
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We propose Log-Int-Softmax (LIS), a novel method that can perform 4-bit quantization on attention maps.With LIS, we can store attention maps on an aggressively lowbit and replace multiplication with BitShift operator.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bitwidth on attention maps.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Abstract
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is obvious that all current methods can't capture fully quantized vision transformers, while our FQ-ViT does it and achieves a nearly lossless quantization even with an aggressively low-bit on attention maps.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Image Classification on ImageNet
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is observed that the channel-wise ranges fluctuate more wildly in vision transformers than those in ResNets.For instance, the maximum range/median range of ResNet152 is only 21.6/4.2,while it goes up to 622.5/15.5 in Swin-B.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As experimenting on quantization of attention maps from 8-bit to 4-bit with uniform quantization, all vision transformers show severe performance drop.For example, DeiT-T only results in 8.69% top-1 accuracy on Im-ageNet with 4-bit uniform quantized attention maps, decreasing 63.05% from 8-bit case.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Consequently, we save the extreme degradation in 4-bit quantization of attention maps and achieve equal performance as 8-bit uniform quantization with 50% less memory footprint.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, our FQ-ViT achieves 81.20% accuracy on DeiT-B in the case of all modules quantized to 8-bit, and it can still achieve 80.85% accuracy when the attention maps are compressed to 4-bit.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Image Classification on ImageNet
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers.”
- Also stated in: § DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin · boosted
“we propose Power-of-Two Factor (PTF) to quantize the inputs of the LayerNorm.In this way, the quantization error is greatly reduced, and the overall computational efficiency is the same as that of layer-wise quantization thanks to the BitShift operator.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when quantized to lower bit-width, as the 6-bit and 4-bit cases show, uniform quantization sharply degrades and even deactivates all the attention area.On the contrary, LIS still presents acceptable performance similar to 8-bit.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 4.4 Visualization of Quantized Attention Map
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Firstly, the model with PTF and LIS outperforms the baseline and achieves almost lossless accuracy.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 4.3 Ablation Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Secondly, thanks to the BitShift operator, PTF only introduces little amount of BitOPs, while LIS reduces that.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 4.3 Ablation Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When both using 8-bit, uniform quantization focuses on the high activation area, while LIS keeps more texture in the low activation area, which retains more relative rank of the attention map.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 4.4 Visualization of Quantized Attention Map
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“As model performance proved to benefit from higher resolution and smaller patch size [Dosovitskiy et al., 2021], when the increasing resolution and reducing patch size, the storage and computation of attention maps become the bottleneck which directly affect the throughput and latency of inference.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.3 Log-Int-Softmax for Softmax Quantization
What the sources say
- Not matchedcites [Dosovitskiy et al., 2021]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Based on such extreme inter-channel variation, layer-wise quantization, which applies the same quantization parameters to all channels, will lead to an intolerable quantization error.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We propose Power-of-Two Factor (PTF), a simple yet efficient post-training method that can achieve accurate quantization on LayerNorm inputs with only one layerwise quantization scale.”
- Also stated in: § 3.2 Power-of-Two Factor for LayerNorm Quantization · plain
“The core idea of PTF is to equip different channels with different factors, rather than different quantization parameters.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 3.2 Power-of-Two Factor for LayerNorm Quantization · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Firstly, comparing to Equation ( 6), the fixed output range (0, 1) of Softmax makes the log2 function calibration-free: Attn Q = Q(Attn|b) = clip(⌊-log 2 (Attn)⌉, 0, 2 b -1).( 15) This ensures that the quantization of attention maps will not be affected by the fluctuation of calibration data.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“During inference, layer-wise parameters s and zp can be extracted, so the computation of µ,σ could be done in the integer domain rather than floating-point, which reduces the energy and area costs [Lian et al., 2019].”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
What the sources say
- Not checked — no full text of the sourcecites [Lian et al., 2019] source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We achieve integer-only inference on Softmax modules, significantly reducing the inference consumption.”
- Also stated in: § Integer-only Inference · plain
“we propose Log-Int-Softmax, an integer-only, faster, low consuming Softmax”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Integer-only Inference · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Combining these methods, we are the first work to achieve post-training quantization for fully quantized vision transformers.”
- Also stated in: § Abstract · hedged
“To our knowledge, we are the first to achieve lossless accuracy degradation (∼1%) on fully quantized vision transformers.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Abstract · hedged
In the paper
“However, this work does not quantize Softmax and Layer-Norm modules, resulting in an incomplete quantization.”
- Also stated in: § 4.2 Comparison with State-of-the-art Methods · plain
“Note that PTQ for ViT [Liu et al., 2021b] is closest to our work, however it does not quantize the LayerNorm and Softmax, while we quantize all modules.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 2.2 Network Quantization
What the sources say
- Not matchedcites [Liu et al., 2021b]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.2 Comparison with State-of-the-art Methods · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we propose Log-Int-Softmax (LIS), which provides higher quantization resolution for small values and presents a more efficient integer inference for Softmax.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
The extreme non-uniform distribution of attention maps observed in ViT-L is representative of attention maps in other vision transformer architectures.
“Inspired by the idea of sparse attention in Dynam-icViT [Rao et al., 2021], we probe into the distribution of attention maps, as Figure 3 shows.We observe a distribution centering at a fairly small value, while only a few outliers have larger values close to 1. Averaging on all attention maps in ViT-L, about 98.8% of values are smaller than 1/16.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For ViTs, FQ-ViT [30] preliminarily explores the feasibility of integer arithmetic for part of the operations ( e . g ., Softmax), but it is simply built on I-BERT [19] and ignores the notable GELUoperation, leaving a huge gap between it and integeronly inference.”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It is worth noting that although these three component approximations can potentially enable integeronly inference of ViTs (partly verified by FQ-ViT [30]), the computation of high-order polynomials is inefficient in inference, and they are developed for language models that do not fit the data distribution of ViTs, leading to mismatched approximations.”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LIS also encounters the above problems, since it is simply built on top of I-BERT.”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“LIS in FQ-ViT [30]: using Log-Int-Softmax, which builds opon I-BERT and adds the logarithmic function.”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“In addition, we also display the cases of ResNets [He et al., 2016] for comparison.”
ResultInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“Image Classification on ImageNet To demonstrate the effectiveness of proposed methods, we conduct extensive experiments in ImageNet [Krizhevsky et al., 2012] with various vision transformers, i.”
MethodInfluentialAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 4×
“Object Detection on COCO We also conduct experiments on the object detection benchmark COCO [Lin et al., 2014].”
MethodInfluentialSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · arXiv · Cited by 35,608
Self-citationSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 19×
“…architectures have achieved competitive performance in various computer vision (CV) tasks, including image classification [Dosovitskiy et al. , 2021; Tou-vron et al. , 2021], object detection [Carion et al. , 2020; Liu et al. , 2021a], semantic segmentation [Zheng et al. , 2021] and so on.”
BackgroundMethodResultInfluentialSelf-citationBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2×
“The above process can be written as: Unlike BatchNorm [Ioffe and Szegedy, 2015], commonly used in CNNs, LayerNorm cannot be folded into the previous layer due to its dynamic computational property, so we have to quantize it separately.”
MethodEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · cited 1×
“…architectures have achieved competitive performance in various computer vision (CV) tasks, including image classification [Dosovitskiy et al. , 2021; Tou-vron et al. , 2021], object detection [Carion et al. , 2020; Liu et al. , 2021a], semantic segmentation [Zheng et al. , 2021] and so on.”
BackgroundLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Research works printed but not linked: 2
- [36]36. Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distilla- tion through attention. In International Conference on Machine Learning. PMLR, 2021.
- [38]38. Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantiza- tion via bit-split and stitching. In International Conference on Machine Learning. PMLR, 2020.
Cited by
Sorted by citations
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Li, Z., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01565 · Cited by 141
“FQ-ViT [30] introduces powers-of-two scale quantization and log-int quantization for LayerNorm and Softmax, respectively.”
BackgroundMethodInfluentialSelf-citationRepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers
Li, Z., Xiao, J., Yang, L., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01580 · Cited by 127
“To this end, several efforts identify the key components that limit the quantization performance of ViTs, such as LayerNorm, Softmax, and GELU, and pro-pose PTQ schemes accordingly [22, 24].”
BackgroundMethodInfluentialSelf-citationFoundation models in smart agriculture: Basics, opportunities, and challenges
Li, J., Xu, M., Xiang, L., Chen, D., Zhuang, W., Yin, X., & Li, Z. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109032 · Cited by 159
Self-citationNoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers
Liu, Y., Yang, H., Dong, Z., Keutzer, K., Du, L., & Zhang, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01946 · Cited by 60
“FQ-ViT [19], EasyQuant [32], and PTQ4ViT [37], on ImageNet.”
Saha, S., & Xu, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130417 · Cited by 85
SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inference
Wang, W., Zhou, S., Sun, W., Sun, P., & Liu, Y. (2023) · 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD) · doi:10.1109/iccad57390.2023.10323725 · Cited by 51
“Proposed by [22] for Layer-Norm quantization, PTF equips different channels with different factors to address the inter-channel variation in the inputs of LayerNorm layers.”
Self-citationWhich Tokens to Use? Investigating Token Reduction in Vision Transformers
Haurum, J. B., Escalera, S., Taylor, G. W., & Moeslund, T. B. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00085 · Cited by 47
“Within the computer vision domain several method paradigms have been investigated such as model pruning [7, 9, 46, 56, 71] and quantization [35, 39], structured token downsampling inspired by the pooling layers in CNNs [20, 23, 41], randomly dropping patches [1, 40], part selection modules [22, 26, 27, 63, 66], and dy…”
A survey of FPGA and ASIC designs for transformer inference acceleration and optimization
Kang, B. J., Lee, H. I., Yoon, S. K., Kim, Y. C., Jeong, S. B., Jeong, S. B., & Kim, H. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103247 · Cited by 54
PSAQ-ViT V2: Toward Accurate and General Data-Free Quantization for Vision Transformers
Li, Z., Chen, M., Xiao, J., & Gu, Q. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3301007 · Cited by 41
“FQ-ViT [7] designs specific quantization strategies for Softmax and LN components to obtain the fully quantized ViTs.”
Self-citationVision transformer models for mobile/edge devices: a survey
Lee, S. I., Koo, K., Lee, J. H., Lee, G., Jeong, S., Seongjun, O., & Kim, H. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01312-0 · Cited by 44
“Another critical approach is network quantization [33], which reduces the precision of the numerical values in the model.”
Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge
Shen, X., Dong, P., Lu, L., Kong, Z., Li, Z., Lin, M., Wu, C., & Wang, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29860 · Cited by 50
Self-citationDong, P., Zhuang, J., Yang, Z., Ji, S., Li, Y., Xu, D., Huang, H., Hu, J., Jones, A. K., Shi, Y., et al. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3443692 · Cited by 35
Q-DETR: An Efficient Low-Bit Quantized Detection Transformer
Xu, S. Y., Li, Y., Lin, M., Gao, P., Guo, G., Lü, J., & Zhang, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00374 · Cited by 34
“We also report the detection performance of the 8-bit post-training quantization networks, such as percentile [19] , VT-PTQ [26].”
MethodBoost Vision Transformer with GPU-Friendly Sparsity and Quantization
Yu, C., Chen, T., Gan, Z., & Fan, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02170 · Cited by 33
“we choose the FQ-ViT [25], Q-ViT [23], PTQ-ViT [28] and PTQ4ViT [65] as the reference methods from quantization category.”
BackgroundMethodBiViT: Extremely Compressed Binary Vision Transformers
He, Y., Lou, Z., Zhang, L., Liu, J., Wu, W., Zhou, H., & Zhuang, B. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00520 · Cited by 31
“However, current Transformer quantization literature mainly focuses on fixed-point quantization, either through Quantization-Aware Training (QAT) [39, 43] or Post-Training Quantization (PTQ) [53, 45, 76].”
BackgroundMethodInfluentialBinaryViT: Pushing Binary Vision Transformers Towards Convolutional Models
Le, P.-H. C., & Li, X. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00492 · Cited by 32
PTQ4SAM: Post-Training Quantization for Segment Anything
Lv, C., Chen, H., Guo, J., Guo, J., Guo, J., Ding, Y., Liu, X., Liu, X., & Liu, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01509 · Cited by 35
“14 end for 18: end for 19: return M Q [7, 27, 30, 68] identify the extremely unbalanced power-law distribution as the rationale of quantization difficulty and devise a specialized quantizer to address it.”
BackgroundMethodRate-Distortion Optimized Post-Training Quantization for Learned Image Compression
Shi, J., Lu, M., & Ma, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3323015 · Cited by 28
“In the past years, as the wide spread of self-attention-based Vision Transformer (ViT) in various tasks [9], [24], [31], [32], numerous works [9], [31], [33], [34] attempted to extend the self-attention mechanism in LIC for content-adaptive embedding.”
BackgroundMethodShi, H., Xin, C., Mao, W., & Wang, Z. (2024) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2024.3422684 · Cited by 30
“Moreover, FQ-ViT [7] proposes Power-of-Two Factor (PTF) and Log-Int-Softmax (LIS) to quantize ViTs’ non-linear Layer-Norm and Softmax, respectively, offering fully quantized ViTs.”
BackgroundMethodInfluentialPan, Z., Cai, J., & Zhuang, B. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01545 · Cited by 21
“Another line of works focus on improving ViT efficiency by token pruning [35, 41], quantization [24, 27] and dynamic inference [41, 51], etc.”
BackgroundWang, G., Cai, S., Li, W., Lyu, D., & He, G. (2024) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2024.3426653 · Cited by 25
“This approach leverages PTQ adopting int8 as the base precision since SOTA ViT PTQ methods can maintain accuracy comparable with full precision at int8 [11], [12].”
MethodInfluentialTFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models
Huang, Y., Gong, R., Liu, J., Chen, T., & Liu, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00703 · Cited by 38
“It can be categorized into quantization-aware training (QAT) [7, 16, 28, 49, 53] and post-training quantization (PTQ) [15, 24, 26, 31, 47].”
BackgroundTrio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer
Shi, H., Shao, H., Mao, W., & Wang, Z. (2024) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2024.3485192 · Cited by 21
“Specifically, we first adopt the leading one detector (LOD) to find the index i of the first non-zero bit of X Q , then add i with the value of the ( i - 1) th bit to obtain the result.”
InfluentialQuantized Feature Distillation for Network Quantization
Zhu, K., He, Y.-Y., & Wu, J. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26354 · Cited by 18
“Recent methods (Yuan et al. 2021; Liu et al. 2021b; Lin et al. 2022) tried post training quantization (Nagel et al.”
BackgroundMethodInfluentialADFQ-ViT: Activation-Distribution-Friendly post-training Quantization for Vision Transformers
Jiang, Y., Sun, N., Xie, X., Yang, F., & Li, T. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107289 · Cited by 20
AdaLog: Post-training Quantization for Vision Transformers with Adaptive Logarithm Quantizer
Wu, Z., Chen, J., Zhong, H., Huang, D., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73383-3_24 · Cited by 20
OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning
Thwal, C. M., Nguyen, M. N. H., Tun, Y. L., Kim, S. T., Thai, M. T., & Hong, C. S. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.11.044 · Cited by 18
“FQ-ViT (Lin et al., 2022) is then implemented under the PTQ, a training-free paradigm, by introducing the power-of-two factor (PTF) and log-int-softmax (LIS) quantization methods for LayerNorm and Softmax modules, which are not quantized in PTQ4ViT.”
BackgroundMethodAn FPGA-Based Reconfigurable Accelerator for Convolution-Transformer Hybrid EfficientViT
Shao, H., Shi, H., Mao, W., & Wang, Z. (2024) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas58744.2024.10557992 · Cited by 15
“Besides, the non-linear operations in ViTs, e.g., LayerNorm (LN), GELU [5], and especially Softmax, are hardware unfriendly and quantization sensitive [6], [7], hindering ViTs’ achievable task accuracy and hardware efficiency.”
BackgroundJumping through Local Minima: Quantization in the Loss Landscape of Vision Transformers
Frumkin, N., Gope, D., & Marculescu, D. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01557 · Cited by 14
“In our framework, we learn these initial quantization parameters using a fast, layer-wise framework such as FQ-ViT [20], and then use evolutionary search to adjust the scales of each attention block’s projection layers.”
MethodTowards Accurate Post-Training Quantization of Vision Transformers via Error Reduction
Zhong, Y., You, H., Hu, J., Zhang, Y., & Ji, R. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3528042 · Cited by 26
“Unfortunately, the unique architectural intricacies of ViTs, such as MHSA and LN, make that conventional PTQ methods, tailored for CNNs, often un-derperform when applied to ViTs [25], [29], [40], [41], [42].”
InfluentialNAS-BNN: Neural Architecture Search for Binary Neural Networks
Lin, Z., Wang, Y., Zhang, J., et al. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111086 · Cited by 15
High-throughput systolic array-based accelerator for hybrid transformer-CNN networks
Song, Q., Dai, Y., Lu, H., & Jin, G. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102194 · Cited by 14
“The accuracy of ViT inference after FQ-ViT quantization was almost unaffected.”
InfluentialWang, J., Zhang, L., Li, X., Yang, H., & Liu, Y. (2023) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2023.3329039 · Cited by 15
“We use two models to evaluate the proposed sparse nonlinear Transformer method for short and long input sequences, and they are quantized using FQ-ViT [28].”
MethodRAWAtten: Reconfigurable Accelerator for Window Attention in Hierarchical Vision Transformers
Li, W., Luo, Y., & Yu, S. (2023) · Design, Automation, and Test in Europe · doi:10.23919/date56975.2023.10137196 · Cited by 13
“Post-training quantization [17] is applied to quantize all model parameters to INT8, resulting in accuracy loss of 0.89% for Swin-T and 0.53% for Swin-S.”
MethodInfluentialAn Efficient Window-Based Vision Transformer Accelerator via Mixed-Granularity Sparsity
Dong, Q., Zhang, S., & Wang, Z. (2025) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2025.3527541 · Cited by 16
“We can see that with approximation calculations and appropriate quantization bit-width for non-linear operations, the model maintains the 8-bit FXP accuracy of 80.06%, 0.98% higher than that of FQ-ViT [40].”
InfluentialOPAL: Outlier-Preserved Microscaling Quantization Accelerator for Generative Large Language Models
Koo, J., Park, D., Jung, S., & Kung, J. (2024) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3649329.3657323 · Cited by 16
“Therefore, we modified the log2-based softmax method [8], an approach for PTQ, on optimizing LLM inference.”
CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-training Quantization of ViTs
Ramachandran, A., Kundu, S., & Krishna, T. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72855-6_18 · Cited by 14
Are Conventional SNNs Really Efficient? A Perspective from Network Quantization
Shen, G., Zhao, D., Li, T., Li, J., & Zeng, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02600 · Cited by 13
“To address the severe energy and computational constraints, the traditional machine learning domain has introduced a suite of model optimization techniques, including quantization [19, 20, 22] and network pruning [9, 10, 30].”
Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers
Li, Z., Lu, A., Xie, Y., Kong, Z., Sun, M., Tang, H., Xue, Z. J., Dong, P., Ding, C., Wang, Y., et al. (2024) · International Conference on Supercomputing · doi:10.1145/3650200.3656622 · Cited by 12
Self-citationDisaster Detection on the Fly: Optimized Transformers for UAVs
Jankovic, B., Jangirova, S., Ullah, W., Khan, L. U., & Guizani, M. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3596681 · Cited by 13
“However, the significant number of parameters and the computational overhead of transformers are challenging during their deployment on resource-constrained hardware devices [25].”
PQ-SAM: Post-training Quantization for Segment Anything Model
Liu, X., Ding, X., Yu, L., Xi, Y., Li, W., Tu, Z., Hu, J., Chen, H., Yin, B., & Xiong, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72684-2_24 · Cited by 11
“Nevertheless, exiting PTQ methods [22, 23, 25, 50] for ViT are focusing on conventional ViT models for classification, which are different from the ViT in SAM.”
Lee, J., Kwon, Y., Park, S., Yu, M., Park, J., & Song, H. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3403844 · Cited by 22
“Recently, quantization for ViTs has been studied [28]– [30], [47]–[49].”
BackgroundMethodZhang, H., Sboev, A. G., Борисович, Р. Р., & Yu, Q. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107789 · Cited by 10
STFormer: An efficient visual Transformer model with sparse attention and adaptive token aggregation
Feng, X., Zhong, S., Wang, Y., & Ming, D. (2025) · Pattern Recognition Letters · doi:10.1016/j.patrec.2025.03.035 · Cited by 9
Wu, J., Song, M., Zhao, J., Gao, Y., Li, J., & So, H. K. (2025) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3714416 · Cited by 10
Agglomerative Token Clustering
Haurum, J. B., Escalera, S., Taylor, G. W., & Moeslund, T. B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72998-0_12 · Cited by 11
“These attempts range from model pruning [5, 8, 9, 42, 56, 71], quantization [33, 37], structured downsampling [19, 25, 39, 46], sparsification of the attention module [4,65], part selection modules [22,27,28,61,62], and dynamically adjusting the size of input patches [1,7,12,35,63,64,72,74].”
BackgroundAQ-DETR: Low-Bit Quantized Detection Transformer with Auxiliary Queries
Wang, R., Sun, H., Yang, L., Lin, S., Liu, C., Gao, Y., Hu, Y. B., & Zhang, B. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i14.29487 · Cited by 9
“We also report the detection performance of the 8-bit post-training quantization networks, such as percentile (Lin et al. 2021), VT-PTQ (Liu et al. 2021).”
MethodAdaptive patch selection to improve Vision Transformers through Reinforcement Learning
Cauteruccio, F., Marchetti, M., Traini, D., Ursino, D., & Virgili, L. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06516-z · Cited by 11
BinaryFormer: A Hierarchical-Adaptive Binary Vision Transformer (ViT) for Efficient Computing
Wang, M., Xu, Z., Zheng, B., & Xie, W. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3396520 · Cited by 8
“In addition, the theoretical model size as suggested in [10], [15], [20], and [32] is calculated to evaluate the cost of the memory space required for model deployment.”
BackgroundMethodInfluentialMaulana, D., & Ramasamy, R. K. (2026) · Computers · doi:10.3390/computers15010069 · Cited by 7
“Edge devices (Jetson Nano, Raspberry Pi, Coral TPU) impose strict limitations on power consumption ( 10 W), memory ( 8 GB), and computational throughput [18].”
Wu, Z., Zhang, J., Chen, J., Guo, J., Huang, D., & Wang, Y. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00905 · Cited by 10
“PTQ methods can be further divided into two categories, i.e ., the one that only involves calibration [27, 29, 35, 50], and the reconstruction-based one [20, 36, 49, 54], where the later generally achieves superior accuracy by introducing an efficient fine-tuning process.”
QAttn: Efficient GPU Kernels for mixed-precision Vision Transformers
Kluska, P., Castelló, A., Scheidegger, F., Malossi, A. C. I., & Quintana–Ort́ı, E. S. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00368 · Cited by 6
“Lastly, methods like FQ-ViT [26] or PTQ4ViT [51] tried to find optimal scaling factors for the weights of ViT to avoid performance degradation.”
LRP-QViT: Mixed-Precision Vision Transformer Quantization Using Layer Importance Score
Ranjan, N., & Savakis, A. E. (2025) · IEEE International Conference on Digital Signal Processing (DSP) · doi:10.1109/dsp65409.2025.11075195 · Cited by 8
“Post-training quantization (PTQ) methods [4], [5], [19] enable efficient quantization without retraining, mitigating performance loss through twin uniform quantization [19], block-wise calibration [20], and reparameterization [5], [6].”
Ahmed, S., Arafat, A. A., Najafi, D., Mahmood, A. U., Rizve, M. N., Nahian, M. A., Zhou, R., Angizi, S., & Rakin, A. S. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.02806 · Cited by 8
“A widely used approach for compressing Vision Transformers (ViT) is pruning [16, 30– 33, 36 ], which focuses on reducing the Another widely adopted approach for compressing ViT models is quantization [8, 11–13, 15, 18], which converts floating-point parameters to lower precision values.”
PTQ4RIS: Post-Training Quantization for Referring Image Segmentation
Jiang, X., Yang, H., Zhu, K., Qiu, X., Zhao, S., & Zhou, S. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11128841 · Cited by 7
“Based on this, FQ-ViT [19] introduces a power-of-two factor and Log-Int-Softmax to achieve more comprehensive quantization.”
Self-citationHuo, Y., Kang, Y., Yang, D., & Zhu, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130061 · Cited by 5
M2-ViT: Accelerating Hybrid Vision Transformers With Two-Level Mixed Quantization
Liang, Y., Shi, H., & Wang, Z. (2025) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2024.3525184 · Cited by 8
“Similarly, for weights, since summations are constrained to channels within filters, they are typically filter-wise quantized, where all weight channels within each filter share the same scaling factor [8].”
BackgroundMethodInfluentialSpatial–Temporal Spiking Feature Pruning in Spiking Transformer
Zhou, Z., Che, K., Niu, J., Yao, M., Li, G., Yuan, L., Luo, G., & Zhu, Y. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3500018 · Cited by 6
“In both NLP and CV, many techniques have been proposed to improve Transformer efficiency, such as token pruning [64], [65], [66], [67], [68], [69], [70], [71], [72], token merging [73], [74], linear attention [75], [76] and model quantization [77], [78].”
Allspark: Workload Orchestration for Visual Transformers on Processing In-Memory Systems
Ge, M., Wang, J., Chen, B., et al. (2024) · IEEE Transactions on Computers · doi:10.1109/tc.2024.3483633 · Cited by 5
“Following the integer quantization methods i-BERT [39] and FQ-ViT [40], we synthesize dedicated ACU of 8-bit integers based on 28nm process technology using commercial logic synthesis tools to obtain processing delay for nonlinear computations.”
MethodTanama, C., Peng, K., Marinov, Z., Stiefelhagen, R., & Roitberg, A. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342203 · Cited by 5
“From a model perspective, quantization-based model compression [13], [14], [15], [16], [17], [18], [19], [20], [21], [22] is a well-established technique for reducing the number of bits required by the model.”
MethodPost-Training Quantization for Vision Transformer in Transformed Domain
Feng, K., Chen, Z., Gao, F., Wang, Z., Xu, L., & Lin, W. (2023) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme55011.2023.00252 · Cited by 5
“Our proposed method is compared with four existing post-training quantization methods in the literature, including PTQforViT [8], FQViT [10], PTQ4ViT [11], and PSAQ-ViT [9].”
MethodAdaptive layer and token selection for efficient fine-tuning of vision transformers
Devoto, A., Alvetreti, F., Pomponi, J., Lorenzo, P. D., Minervini, P., & Scardapane, S. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131216 · Cited by 11
Ding, R., Liang, Y. Q., Zhao, S., Nie, J., Chen, L., Liu, H., & Zhou, X. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107558 · Cited by 6
Token Compensator: Altering Inference Cost of Vision Transformer Without Re-tuning
Jie, S., Tang, Y., Guo, J., Deng, Z., Han, K., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72640-8_5 · Cited by 6
RL-PTQ: RL-based Mixed Precision Quantization for Hybrid Vision Transformers
Kwon, E., Zhou, M., Xu, W., Rosing, T., & Kang, S. (2024) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3649329.3656231 · Cited by 4
“Without this consideration, channels with a narrow range may face issues where all values are treated as zeros during layer-wise quantization.”
Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
Du, D., Gong, G., & Chu, X. (2024) · arXiv · doi:10.48550/arxiv.2405.00314 · Cited by 4
“4) Outlier-Aware Training : PackQViT [69] introduces an outlier-aware training approach, differing from the method in [43] which calculates the power-of-two factor during calibration.”
MethodInfluentialPark, D., Park, H., Yeo, I., Kim, J., Lee, S., Lee, S., Shin, H., Pae, S.-I., Fan, D., Kung, J., et al. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3633487 · Cited by 6
“Unlike conventional convolution blocks, Transformers contain operations that are di ffi cult to quantize, such as softmax and Gaussian error linear unit (GELU), requiring deeper consideration when selecting and applying quantization algorithms [81], [82].”
AHCPTQ: Accurate and Hardware-Compatible Post-Training Quantization for Segment Anything Model
Zhang, W., Zhong, Y., Ando, S., & Yoshioka, K. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02078 · Cited by 7
“PTQ requires only a small calibration dataset, significantly reducing data and computational demands while maintaining competitive accuracy [4, 6, 20, 23, 27, 34, 47–49].”
PoMQ-ViT: Mixed-Precision Quantization Vision Transformer with Pareto Optimization
Wu, Z., & Zhao, Z. (2025) · Applied Sciences · doi:10.3390/app15189856 · Cited by 3
He, X., Ye, L., Liu, H., Gong, C., & He, W. (2025) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2025.103530 · Cited by 7
Temporal Feature Matters: A Framework for Diffusion Model Quantization
Huang, Y., Gong, R., Liu, X., Liu, J., Li, Y., Lu, J., & Tao, D. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3585692 · Cited by 6
“It can be categorized into quantization-aware training (QAT) [15], [27], [53], [96], [105] and post-training quantization (PTQ) [25], [45], [49], [61], [89].”
Wu, P., Qu, Y., Zhao, Z., Liu, Z., & Yu, H. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04254-w · Cited by 3
“Moreover, we develop a novel Fully Quantized Convolutional Vision Transformer (FQ-Conv-ViT) model for DR detection, which further implements the accurate quantization on LayerNorm [30] inputs and reduces the computational complexity by employing Log-Int-Softmax (LIS).”
Throughput Maximization for Transformer Inference on Processing Near-Memory Architectures
Ge, M., Zhong, Y., Chen, S., & Kang, Y. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3574235 · Cited by 3
“Following the integer quantization methods i-BERT [59] and FQ-ViT [60], we synthesize dedicated ACU of 8-bit integers based on 28 nm process technology using commercial logic synthesis tools to obtain processing delay for nonlinear computations.”
A 28nm 343.5fps/W Vision Transformer Accelerator with Integer-Only Quantized Attention Block
Lin, C.-C., Lu, W., Huang, P.-T., & Hung-Ming, C. (2024) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas59952.2024.10595969 · Cited by 3
Mixed Precision Quantization with Hardware-Friendly Activation Functions for Hybrid ViT Models
Kang, B. J., Choi, D. H., & Kim, H. (2024) · International Conference on Electronics, Information, and Communication (ICEIC) · doi:10.1109/iceic61013.2024.10457283 · Cited by 4
FHPG: A unified framework for transformer with pruning and quantization
Ren, R. (2026) · Pattern Recognition Letters · doi:10.1016/j.patrec.2026.01.020 · Cited by 2
PTQAT: A Hybrid Parameter-Efficient Quantization Algorithm for 3D Perception Tasks
Wang, X., Lin, Z., Xia, Z., & Wang, Y. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00423 · Cited by 2
“FQ-ViT [21] proposes a Power-of-Two Factor method to address significant inter-channel variations in LayerNorm inputs, and introduced Log-Int-Softmax to preserve the non-uniform distribution of attention maps.”
Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformers
Zhong, Y., Zhou, Y., Zhang, Y., Sui, W., Li, S., Li, Y., Chao, F., & Ji, R. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01160 · Cited by 4
“The spe-cific architecture of ViTs, such as LayerNorm and the self-12480 attention module, urges distinct PTQ methods compared to CNNs [18, 24, 46, 51, 53, 86].”
I&S-ViT: An Inclusive & Stable Method for Post-Training ViTs Quantization
Zhong, Y., Hu, J., Lin, M., Chen, M., & Ji, R. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3610466 · Cited by 7
“Unfortunately, the customized operators, such as LayerNorm andMHSAinViTs,createmaladjustmentswhenmakingadirect extensionofPTQmethodsfromCNNstoViTs[2] , [4], [19], [20], [52].”
InfluentialDong, J., Lou, W., Wu, H., Zheng, Z., Qin, Y., Gong, L., Wang, C., & Zhou, X. (2025) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2025.3604705 · Cited by 4
“We evaluate several ViT architectures, including ViT [7] and DeiT [28], and compare our results against prior state-of-the-art (SOTA) quantization meth-ods [20], [24], [26], [27].”
PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inference
Li, Y., Tang, C., Meng, Y., Fan, J., Chai, Z., Ma, X., Wang, Z., & Zhu, W. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3605239 · Cited by 4
“FQ-ViT [29] adaptively quantizes all structures of the ViTs model while maintaining results similar to those of the full precision model.”
Post-Training Quantization for Vision Mamba with K-Scaled Quantization and Reparameterization
Shi, B.-Y., Lo, Y.-C., Wu, A.-Y. A., & Tsai, Y. (2025) · Machine learning for signal processing ... · doi:10.1109/mlsp62443.2025.11204250 · Cited by 3
Zhao, J., Zeng, C., Wang, M., Han, L., Shang, Y., Zhang, M., & Nie, L. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3599479 · Cited by 2
“Especially, for ViTs we additionally select several ViTs-specific quantization techniques, i.e., FQ-ViT [25], PTQ4ViT [30], APQ-ViT [43] and RepQ-ViT [27]. b) Models: For LLMs, we mainly choose LLaMA (7B,13B and 30B) [44] and LLaMA-2 (7B,13B) [45] to evaluate our proposed methods, since LLaMA and…”
Deep Compression on Segment Anything Model for Efficient Industrial Manufacturing
Zheng, Y., Liu, J., Li, Q., Li, J., & Xi, Z. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3585613 · Cited by 2
“While FQ-ViT [57] applies a powers-of-two scale quantization to LayerNorm and uses log-int quantization for Softmax, RepQ-ViT [58] takes a di ff erent approach.”
Efficient Integer Quantization for Compressed DETR Models
Liu, P., Li, C., Zhang, N., Yang, J., & Wang, L. (2025) · Entropy · doi:10.3390/e27040422 · Cited by 3
Binary Transformer Based on the Alignment and Correction of Distribution
Wang, K., Wang, M., Wan, Z., & Shen, T. (2024) · Sensors · doi:10.3390/s24248190 · Cited by 3
“Quantization technology [10] has been considered the most effective approach for reducing the amount of calculations in transformer.”
MGRQ: Post-Training Quantization For Vision Transformer With Mixed Granularity Reconstruction
Yang, L., Li, Z., Xiao, J., Gong, H., & Gu, Q. (2024) · Proceedings - International Conference on Image Processing · doi:10.1109/icip51287.2024.10648284 · Cited by 3
Self-citationFQ4DM: Full Quantization for Diffusion Model
Wang, C.-E., Tai, Y.-S., & Wu, A.-Y. A. (2024) · Machine learning for signal processing ... · doi:10.1109/mlsp58920.2024.10734817 · Cited by 2
“Contrarily, our method aims to quantize entire models, markedly diminishing computation time as well as memory usage [25].”
MethodFinding Optimal Numerical Format for Sub-8-Bit Post-Training Quantization of Vision Transformers
Lee, J., Hwang, Y., & Choi, J. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096798 · Cited by 2
“Many prior studies have proposed variants of fixed and floating-point formats to represent values with a limited number of bits [7, 8, 9, 10], but these studies suffer significant accuracy degradation when the bit-precision is reduced below 8-bit due to sub-optimal representation restraining…”
BackgroundINSPIRE: In-Sensor Compressed Weight Retrieval for Enhancing ViT Efficiency at Edge
Ahmed, S., Najafi, D., Nahian, M. A., Khoshavi, N., Arafat, A. A., Rizve, M. N., Nikdast, M., Rakin, A. S., & Angizi, S. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539411 · Cited by 1
“Tailored quantization methods for ViTs include PTQ4ViT [18], which applies twin uniform quantization with Hessian-guided scaling, FQ-ViT [19] utilizes powers-of-two scale quantization, RepQ-ViT [20] decouples quantization from inference to manage extreme activation distributions, and [21] enables…”
Kaneko, H., & Meng, L. (2026) · The Journal of Supercomputing · doi:10.1007/s11227-026-08450-4 · Cited by 1
Where Do Tokens Go? Understanding Pruning Behaviors in STEP at High Resolutions
Szczepanski, M., Poreba, M., & Haroun, K. (2026) · SN Computer Science · doi:10.1007/s42979-025-04707-6 · Cited by 1
“Although various efforts have been made to improve their efficiency, achieving a balance between computational complexity, latency, and performance remains difficult, including quantization [12–16] knowledge distillation [17, 18] and pruning.”
Long, Y., Zhao, L., Liu, T., Yuan, Z., Diao, W., Yan, M., & Wang, Z. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3656351 · Cited by 1
“In the challenging scenario of W4 / A4 quantization, FQ-ViT [18] is almost inoperable, achieving an accuracy of only 0.10% on multiple datasets; RepQ-ViT [38] and I&S-ViT [39] perform relatively well.”
InfluentialHC-PTQ: Poincaré-Based Hyperbolic Clustering for Data-Free Quantization of Vision Transformers
Mineo, R., Palazzo, S., Spampinato, C., & Rundo, F. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00421 · Cited by 1
“Fully quantized ViTs like FQ-ViT [19] and Percentile push low-bit quantization This ICCV Workshop paper is the Open Access version, provided by the Computer Vision Foundation.”
Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformers
Maisonnave, L., Haroun, K., & Pégeot, T. (2025) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw69036.2025.00425 · Cited by 1
“To handle non-linear operations, FQ-ViT [20] employs powers-of-two scaling for LayerNorm and logarithmic integer quantization for Soft-max outputs.”
Back to fundamentals: Low-level visual features guided progressive token pruning
Ouyang, Y., Liang, Y., Li, Q., Guo, X., Luo, Y., Wu, D., Wang, H. E., & Pan, Y. (2025) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2025.103579 · Cited by 2
QuantTune: Optimizing Model Quantization with Adaptive Outlier-Driven Fine Tuning
Chen, J.-M., Chao, Y.-H., Wang, Y., Shieh, M., Hsu, C., & Lin, W.-F. (2025) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr67560.2025.00029 · Cited by 3
“We further compare with FQ-ViT [21], PTQ4ViT [29], and a Gated-Attention outlier method [25].”
Zhang, X., Cheng, Y., Zou, D., Gu, K., Wang, M., & Wang, Z. (2025) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2025.103512 · Cited by 2
Liu, Z., Wu, P., Zhao, Z., & Yu, H. (2025) · Journal of Imaging · doi:10.3390/jimaging11070219 · Cited by 1
“In our model, certain activation functions and fully connected layers involve quantization operations, primarily inspired by FQ-ViT [21].”
Metadata
- Authors
- Lin, Y., Zhang, T., Sun, P., Li, Z., & Zhou, S.
- Affiliations
- Megvii (China) (CN)
- Venue
- Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
- Published
- 2022-07
- Pages
- 1173-1179
- OpenAlex
- W4285601701
- Cited by
- 198
- References
- 26
- Citations by year
- 2026: 49 · 2025: 77 · 2024: 47 · 2023: 25
- FWCI
- 13.51
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Keywords
- post-training quantization, 4-bit quantization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
