The DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index; 2 of the 9 sources its claims cite were checked for support.
FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer
Open accessAbstract
Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers. In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers. In addition, observing an extreme non-uniform distribution in attention maps, we propose Log-Int-Softmax (LIS) to sustain that and simplify inference by using 4-bit quantization and the BitShift operator. Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bit-width on attention maps. For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our knowledge, we are the first to achieve lossless accuracy degradation (~1%) on fully quantized vision transformers. The code is available at https://github.com/megvii-research/FQ-ViT.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
15 Own result · 9 Own interpretation · 6 No basis given · 5 Sources not found · 1 Supported · 1 Partly supported · 1 No full text of the sources · accuracy not yet measured
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Averaging on all attention maps in ViT-L, about 98.8% of values are smaller than 1/16.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Meanwhile, our FQ-ViT significantly exceeds PTQ for ViT [Liu et al., 2021b], whose LayerNorm and Softmax are not quantized.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Image Classification on ImageNet
What the sources say
- Not matchedcites [Liu et al., 2021b]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Firstly, we find a serious interchannel variation of LayerNorm inputs, which some channel ranges even exceed 40× of the median.”
- Also stated in: § 3.2 Power-of-Two Factor for LayerNorm Quantization · plain
“Looking into the inputs of LayerNorm layers, we find there is a serious inter-channel variation.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.2 Power-of-Two Factor for LayerNorm Quantization · plain
In the paper
“Moreover, following the purpose of ranking-aware loss [Liu et al., 2021b], log2 quantization can retain much order consistency between full precision and quantized attention maps.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
What the sources say
- Not matchedcites [Liu et al., 2021b]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results show that our fully quantized vision transformers achieve comparable performance with the full precision models.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 5 Conclusions
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Secondly, we find that the values of the attention map have an extreme non-uniform distribution, with most values clustered in 0 ∼ 0.01, and a few high attention values close to 1.”
- Also stated in: § Log2 Quantization for Attention Map · plain
“We observe a distribution centering at a fairly small value, while only a few outliers have larger values close to 1.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Log2 Quantization for Attention Map · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Meanwhile, thanks to the nature of powers of two, PTF α can be efficiently combined with layerwise quantization by BitShift operator, avoiding floatingpoint calculations of group-wise or channel-wise quantization.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Previous work [Liu et al., 2021b] finds there have been a significant accuracy degradation when quantizing LayerNorm and Softmax of vision transformers.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
What the sources say
- Not matchedcites [Liu et al., 2021b]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in Lay-erNorm inputs”
- Also stated in: § DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin · plain
“Firstly, we find a serious interchannel variation of LayerNorm inputs, which some channel ranges even exceed 40× of the median.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Secondly, it also introduces the merits of converting the MatMul to BitShift between the quantized attention map (Attn Q ) and values (V Q ) as:”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 27
Every source the paper cites for this claim was checked and supports it.
In the paper
“Unlike BatchNorm [Ioffe and Szegedy, 2015], commonly used in CNNs, LayerNorm cannot be folded into the previous layer due to its dynamic computational property, so we have to quantize it separately.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
What the sources say
- Supportedcites [Ioffe and Szegedy, 2015] source
The citation identifies BatchNorm as the contrast to LayerNorm, not as the source of the authors’ LayerNorm quantization claim. Section 3.1 (p. 4) says BatchNorm uses fixed statistics at inference and becomes a linear transform; Section 3.2 (p. 5) describes its use in convolutional networks. These passages support the BatchNorm side of the comparison. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“A possible solution is using group-wise quantization [Shen et al., 2020] or channel-wise quantization [Li et al., 2019], which assign different quantization parameter to different group or channel.However, these will still induce the calculation of mean and variance in the floating-point domain, resulting in a high hardware overhead.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
What the sources say
- Partly supportedcites [Shen et al., 2020] sourceExperimental — accuracy not yet measured
Section 2.3 (p. 3) says each subgroup can have its own quantization range, supporting the group-wise part of the claim. Section 3.2 (p. 5) warns that more groups require more lookup tables and can adversely affect hardware performance, but does not attribute that cost to floating-point calculations of mean and variance. The mean and variance discussed in Section 2.2 (p. 3) concern Hessian eigenvalues, not group-wise quantization overhead. (LLM, conf=0.88)
- Not matchedcites [Li et al., 2019]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared with 4-bit uniform quantization which only assigns 1 bin for such many values, log2 method can allocate 12 bins for them.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our FQ-ViT significantly improves the quantization accuracy and achieves 47.2 mAP on Mask R-CNN (Swin-S) and 50.8 mAP on Cascade Mask R-CNN (Swin-S) with 8-bit on weights/activations and 4-bit on attention maps.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Object Detection on COCO
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We propose Log-Int-Softmax (LIS), a novel method that can perform 4-bit quantization on attention maps.With LIS, we can store attention maps on an aggressively lowbit and replace multiplication with BitShift operator.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bitwidth on attention maps.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Abstract
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is obvious that all current methods can't capture fully quantized vision transformers, while our FQ-ViT does it and achieves a nearly lossless quantization even with an aggressively low-bit on attention maps.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Image Classification on ImageNet
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is observed that the channel-wise ranges fluctuate more wildly in vision transformers than those in ResNets.For instance, the maximum range/median range of ResNet152 is only 21.6/4.2,while it goes up to 622.5/15.5 in Swin-B.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As experimenting on quantization of attention maps from 8-bit to 4-bit with uniform quantization, all vision transformers show severe performance drop.For example, DeiT-T only results in 8.69% top-1 accuracy on Im-ageNet with 4-bit uniform quantized attention maps, decreasing 63.05% from 8-bit case.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Consequently, we save the extreme degradation in 4-bit quantization of attention maps and achieve equal performance as 8-bit uniform quantization with 50% less memory footprint.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, our FQ-ViT achieves 81.20% accuracy on DeiT-B in the case of all modules quantized to 8-bit, and it can still achieve 80.85% accuracy when the attention maps are compressed to 4-bit.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Image Classification on ImageNet
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers.”
- Also stated in: § DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin · boosted
“we propose Power-of-Two Factor (PTF) to quantize the inputs of the LayerNorm.In this way, the quantization error is greatly reduced, and the overall computational efficiency is the same as that of layer-wise quantization thanks to the BitShift operator.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin · boosted
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, when quantized to lower bit-width, as the 6-bit and 4-bit cases show, uniform quantization sharply degrades and even deactivates all the attention area.On the contrary, LIS still presents acceptable performance similar to 8-bit.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 4.4 Visualization of Quantized Attention Map
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Firstly, the model with PTF and LIS outperforms the baseline and achieves almost lossless accuracy.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 4.3 Ablation Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Secondly, thanks to the BitShift operator, PTF only introduces little amount of BitOPs, while LIS reduces that.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 4.3 Ablation Studies
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When both using 8-bit, uniform quantization focuses on the high activation area, while LIS keeps more texture in the low activation area, which retains more relative rank of the attention map.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 4.4 Visualization of Quantized Attention Map
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“As model performance proved to benefit from higher resolution and smaller patch size [Dosovitskiy et al., 2021], when the increasing resolution and reducing patch size, the storage and computation of attention maps become the bottleneck which directly affect the throughput and latency of inference.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.3 Log-Int-Softmax for Softmax Quantization
What the sources say
- Not matchedcites [Dosovitskiy et al., 2021]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Based on such extreme inter-channel variation, layer-wise quantization, which applies the same quantization parameters to all channels, will lead to an intolerable quantization error.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We propose Power-of-Two Factor (PTF), a simple yet efficient post-training method that can achieve accurate quantization on LayerNorm inputs with only one layerwise quantization scale.”
- Also stated in: § 3.2 Power-of-Two Factor for LayerNorm Quantization · plain
“The core idea of PTF is to equip different channels with different factors, rather than different quantization parameters.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 3.2 Power-of-Two Factor for LayerNorm Quantization · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Firstly, comparing to Equation ( 6), the fixed output range (0, 1) of Softmax makes the log2 function calibration-free: Attn Q = Q(Attn|b) = clip(⌊-log 2 (Attn)⌉, 0, 2 b -1).( 15) This ensures that the quantization of attention maps will not be affected by the fluctuation of calibration data.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Log2 Quantization for Attention Map
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“During inference, layer-wise parameters s and zp can be extracted, so the computation of µ,σ could be done in the integer domain rather than floating-point, which reduces the energy and area costs [Lian et al., 2019].”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 3.2 Power-of-Two Factor for LayerNorm Quantization
What the sources say
- Not checked — no full text of the sourcecites [Lian et al., 2019] source
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We achieve integer-only inference on Softmax modules, significantly reducing the inference consumption.”
- Also stated in: § Integer-only Inference · plain
“we propose Log-Int-Softmax, an integer-only, faster, low consuming Softmax”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Integer-only Inference · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Combining these methods, we are the first work to achieve post-training quantization for fully quantized vision transformers.”
- Also stated in: § Abstract · hedged
“To our knowledge, we are the first to achieve lossless accuracy degradation (∼1%) on fully quantized vision transformers.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Abstract · hedged
In the paper
“However, this work does not quantize Softmax and Layer-Norm modules, resulting in an incomplete quantization.”
- Also stated in: § 4.2 Comparison with State-of-the-art Methods · plain
“Note that PTQ for ViT [Liu et al., 2021b] is closest to our work, however it does not quantize the LayerNorm and Softmax, while we quantize all modules.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > 2.2 Network Quantization
What the sources say
- Not matchedcites [Liu et al., 2021b]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.2 Comparison with State-of-the-art Methods · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we propose Log-Int-Softmax (LIS), which provides higher quantization resolution for small values and presents a more efficient integer inference for Softmax.”
§ FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer > DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
The extreme non-uniform distribution of attention maps observed in ViT-L is representative of attention maps in other vision transformer architectures.
“Inspired by the idea of sparse attention in Dynam-icViT [Rao et al., 2021], we probe into the distribution of attention maps, as Figure 3 shows.We observe a distribution centering at a fairly small value, while only a few outliers have larger values close to 1. Averaging on all attention maps in ViT-L, about 98.8% of values are smaller than 1/16.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For ViTs, FQ-ViT [30] preliminarily explores the feasibility of integer arithmetic for part of the operations ( e . g ., Softmax), but it is simply built on I-BERT [19] and ignores the notable GELUoperation, leaving a huge gap between it and integeronly inference.”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It is worth noting that although these three component approximations can potentially enable integeronly inference of ViTs (partly verified by FQ-ViT [30]), the computation of high-order polynomials is inefficient in inference, and they are developed for language models that do not fit the data distribution of ViTs, leading to mismatched approximations.”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LIS also encounters the above problems, since it is simply built on top of I-BERT.”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“LIS in FQ-ViT [30]: using Log-Int-Softmax, which builds opon I-BERT and adds the logarithmic function.”
From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“In addition, we also display the cases of ResNets [He et al., 2016] for comparison.”
ResultInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“Image Classification on ImageNet To demonstrate the effectiveness of proposed methods, we conduct extensive experiments in ImageNet [Krizhevsky et al., 2012] with various vision transformers, i.”
MethodInfluentialAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · cited 4×
“Object Detection on COCO We also conduct experiments on the object detection benchmark COCO [Lin et al., 2014].”
MethodInfluentialSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · arXiv · Cited by 35,608
Self-citationSwin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · cited 19×
“…architectures have achieved competitive performance in various computer vision (CV) tasks, including image classification [Dosovitskiy et al. , 2021; Tou-vron et al. , 2021], object detection [Carion et al. , 2020; Liu et al. , 2021a], semantic segmentation [Zheng et al. , 2021] and so on.”
BackgroundMethodResultInfluentialSelf-citationBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2×
“The above process can be written as: Unlike BatchNorm [Ioffe and Szegedy, 2015], commonly used in CNNs, LayerNorm cannot be folded into the previous layer due to its dynamic computational property, so we have to quantize it separately.”
MethodEnd-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285 · cited 1×
“…architectures have achieved competitive performance in various computer vision (CV) tasks, including image classification [Dosovitskiy et al. , 2021; Tou-vron et al. , 2021], object detection [Carion et al. , 2020; Liu et al. , 2021a], semantic segmentation [Zheng et al. , 2021] and so on.”
BackgroundLayer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., & Kalenichenko, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00286 · Cited by 4,114
Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P. H. S., et al. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00681 · Cited by 4,008
TinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786 · cited 2×
“To facilitate deployment, several techniques have been proposed, including quantization [Zhou et al. , 2016; Nagel et al. , 2020; Shen et al. , 2020; Liu et al. , 2021b], pruning [Han et al. , 2016], distillation [Jiao et al. , 2020] and adaptation of architecture design [Graham et al. , 2021].”
BackgroundMethodDoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Zhou, S., Wu, Y., Ni, Z., Zhou, X., Wen, H., & Zou, Y. (2016) · arXiv · doi:10.48550/arxiv.1606.06160 · Cited by 1,690 · cited 3×
“To facilitate deployment, several techniques have been proposed, including quantization [Zhou et al., 2016; Nagel et al., 2020; Shen et al., 2020; Liu et al., 2021b], pruning [Han et al.”
BackgroundMethodSelf-citationMorgan, M. C. (2025) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.17575540 · Cited by 1,375
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference
Graham, B., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jegou, H., & Douze, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01204 · Cited by 935 · cited 3×
“To facilitate deployment, several techniques have been proposed, including quantization [Zhou et al. , 2016; Nagel et al. , 2020; Shen et al. , 2020; Liu et al. , 2021b], pruning [Han et al. , 2016], distillation [Jiao et al. , 2020] and adaptation of architecture design [Graham et al. , 2021].”
BackgroundMethodDeepTrust^RT: Confidential Deep Neural Inference Meets Real-Time!
Babar, M. F., & Hasan, M. (2024) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.ecrts.2024.13 · Cited by 708
DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., & Hsieh, C.-J. (2021) · arXiv · doi:10.48550/arxiv.2106.02034 · Cited by 448 · cited 3×
“DynamicViT [Rao et al., 2021] presents a dynamic token sparsification framework to prune redundant tokens progressively and dynamically, achieving competitive complexity and accuracy trade-off.”
BackgroundMethodQ-BERT: Hessian Based Ultra Low Precision Quantization of BERT
Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M. W., & Keutzer, K. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6409 · Cited by 437 · cited 4×
“To facilitate deployment, several techniques have been proposed, including quantization [Zhou et al., 2016; Nagel et al., 2020; Shen et al., 2020; Liu et al., 2021b], pruning [Han et al.”
MethodInfluentialLow-bit Quantization of Neural Networks for Efficient Inference
Choukroun, Y., Kravchik, E., Yang, F., & Kisilev, P. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00363 · Cited by 404 · cited 1×
“This paper employs several current post-training quantization methods, including MinMax, EMA [ Jacob et al. , 2018 ] , Per-centile [ Li et al. , 2019 ] , OMSE [ Choukroun et al. , 2019 ] , Bit-Split [ Wang et al. , 2020 ] and PTQ for ViT [ Liu et al. , 2021b ] .”
MethodUp or Down? Adaptive Rounding for Post-Training Quantization
Nagel, M., Amjad, R. A., van Baalen, M., Louizos, C., & Blankevoort, T. (2020) · arXiv · doi:10.48550/arxiv.2004.10568 · Cited by 252
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer
Xu, Y., Zhang, Z., Zhang, M., Sheng, K., Li, K., Dong, W., Zhang, L., Xu, C., & Sun, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20202 · Cited by 226
Fully Quantized Network for Object Detection
Li, R., Wang, Y., Liang, F., Qin, H., Yan, J., & Fan, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00292 · Cited by 214 · cited 4×
“This paper employs several current post-training quantization methods, including MinMax, EMA [ Jacob et al. , 2018 ] , Per-centile [ Li et al. , 2019 ] , OMSE [ Choukroun et al. , 2019 ] , Bit-Split [ Wang et al. , 2020 ] and PTQ for ViT [ Liu et al. , 2021b ] .”
BackgroundMethodInfluentialPost-Training Quantization for Vision Transformer
Liu, Z., Wang, Y., Han, K., Ma, S., & Gao, W. (2021) · arXiv · doi:10.48550/arxiv.2106.14156 · Cited by 192
I-BERT: Integer-only BERT Quantization
Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., & Keutzer, K. (2021) · arXiv · doi:10.48550/arxiv.2101.01321 · Cited by 186
High-Performance FPGA-Based CNN Accelerator With Block-Floating-Point Arithmetic
Lian, X., Liu, Z., Song, Z., Dai, J., Zhou, W., & Ji, X. (2019) · IEEE Transactions on Very Large Scale Integration (VLSI) Systems · doi:10.1109/tvlsi.2019.2913958 · Cited by 162 · cited 3×
“In this case, the models are not fully quantized, resulting in the need to retain floating-point units in the hardware, which will bring large consumption and significantly reduce the in-a ference speed [Lian et al. , 2019].”
BackgroundInfluentialLeViT: a Vision Transformer in ConvNet's Clothing for Faster InferencePreprint
Graham, B. V., El-Nouby, A., Touvron, H., Stock, P., Joulin, A., Jeǵou, H., & Douze, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01136 · Cited by 88
A Deep Look into Logarithmic Quantization of Model Parameters in Neural Networks
Cai, J., Takemoto, M., & Nakajo, H. (2018) · Proceedings of the 10th International Conference on Advances in Information Technology · doi:10.1145/3291280.3291800 · Cited by 31 · cited 2×
“, 2018] and log2 [Cai et al., 2018] are typically used.”
BackgroundEvo-ViT: Slow-Fast Token Evolution for Dynamic Vision TransformerPreprint
Xu, Y., Zhang, Z., Zhang, M., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.01390 · Cited by 15 · cited 1×
“Evo-ViT [Xu et al., 2021] proposes a slow-fast updating mechanism that guarantees information flow and spatial structure, trimming down both the training and inference complexity.”
Background
Research works printed but not linked: 2
- [36]36. Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distilla- tion through attention. In International Conference on Machine Learning. PMLR, 2021.
- [38]38. Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantiza- tion via bit-split and stitching. In International Conference on Machine Learning. PMLR, 2020.
Cited by
Sorted by citations
A Survey of Techniques for Optimizing Transformer Inference
Chitty-Venkata, K. T., Mittal, S., Emani, M., Vishwanath, V., & Somani, A. K. (2023) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2023.102990 · Cited by 154
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Li, Z., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01565 · Cited by 141
“FQ-ViT [30] introduces powers-of-two scale quantization and log-int quantization for LayerNorm and Softmax, respectively.”
BackgroundMethodInfluentialSelf-citationRepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers
Li, Z., Xiao, J., Yang, L., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01580 · Cited by 127
“To this end, several efforts identify the key components that limit the quantization performance of ViTs, such as LayerNorm, Softmax, and GELU, and pro-pose PTQ schemes accordingly [22, 24].”
BackgroundMethodInfluentialSelf-citationFoundation models in smart agriculture: Basics, opportunities, and challenges
Li, J., Xu, M., Xiang, L., Chen, D., Zhuang, W., Yin, X., & Li, Z. (2024) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2024.109032 · Cited by 159
Self-citationNoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers
Liu, Y., Yang, H., Dong, Z., Keutzer, K., Du, L., & Zhang, S. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01946 · Cited by 60
“FQ-ViT [19], EasyQuant [32], and PTQ4ViT [37], on ImageNet.”
Metadata
- Authors
- Lin, Y., Zhang, T., Sun, P., Li, Z., & Zhou, S.
- Affiliations
- Megvii (China) (CN)
- Venue
- Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
- Published
- 2022-07
- Pages
- 1173-1179
- OpenAlex
- W4285601701
- Cited by
- 198
- References
- 26
- Citations by year
- 2026: 49 · 2025: 77 · 2024: 47 · 2023: 25
- FWCI
- 13.51
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
- Keywords
- post-training quantization, 4-bit quantization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
