CoChat
Sign Up
Cited by 198Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningLow-power high-performance VLSI design

FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer

Open access
Lin, Yang, Zhang, Tianyu, Sun, Peiqin, Li, Zheng, Zhou, Shuchang·2022·Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
DOI 10.24963/ijcai.2022/164
Open this paper in CoChat — free accountOpen PDF

Abstract

Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers. In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers. In addition, observing an extreme non-uniform distribution in attention maps, we propose Log-Int-Softmax (LIS) to sustain that and simplify inference by using 4-bit quantization and the BitShift operator. Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bit-width on attention maps. For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our knowledge, we are the first to achieve lossless accuracy degradation (~1%) on fully quantized vision transformers. The code is available at https://github.com/megvii-research/FQ-ViT.

Integrity signals

Registry

Crossref · 10.24963/ijcai.2022/164

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

2 of 9 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured

  • Findingown data

    In ViT-L attention maps, about 98.8% of values are smaller than 1/16.

    “Averaging on all attention maps in ViT-L, about 98.8% of values are smaller than 1/16.”
    Quote in the paperSubstantial claim
  • Findingcitation

    FQ-ViT significantly exceeds PTQ for ViT, whose LayerNorm and Softmax are not quantized.

    “Meanwhile, our FQ-ViT significantly exceeds PTQ for ViT [Liu et al., 2021b], whose LayerNorm and Softmax are not quantized.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [Liu et al., 2021b]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    There is serious inter-channel variation in LayerNorm inputs of vision transformers, with some channel ranges exceeding 40× the median.

    “Firstly, we find a serious interchannel variation of LayerNorm inputs, which some channel ranges even exceed 40× of the median.”
    • Also stated in: § 3.2 Power-of-Two Factor for LayerNorm Quantization · plain
      “Looking into the inputs of LayerNorm layers, we find there is a serious inter-channel variation.”
    Quote in the paperSubstantial claim
  • Findingcitation

    Log2 quantization retains much order consistency between full precision and quantized attention maps.

    “Moreover, following the purpose of ranking-aware loss [Liu et al., 2021b], log2 quantization can retain much order consistency between full precision and quantized attention maps.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [Liu et al., 2021b]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    FQ-ViT achieves comparable performance with full precision models.

    “Experimental results show that our fully quantized vision transformers achieve comparable performance with the full precision models.”
    Quote in the paperSubstantial claim
  • May combine two claims. 27

  • May combine two claims.

    Premisereasoning

    LayerNorm cannot be folded into the previous layer due to its dynamic computational property, so it must be quantized separately.

    “Unlike BatchNorm [Ioffe and Szegedy, 2015], commonly used in CNNs, LayerNorm cannot be folded into the previous layer due to its dynamic computational property, so we have to quantize it separately.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [Ioffe and Szegedy, 2015] source

  • May combine two claims.

    Premisecitation

    Group-wise or channel-wise quantization assign different quantization parameters to different groups or channels but induce calculation of mean and variance in the floating-point domain, resulting in high hardware overhead.

    “A possible solution is using group-wise quantization [Shen et al., 2020] or channel-wise quantization [Li et al., 2019], which assign different quantization parameter to different group or channel.However, these will still induce the calculation of mean and variance in the floating-point domain, resulting in a high hardware overhead.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [Shen et al., 2020] source

      Section 2.3 (p. 3) says each subgroup can have its own quantization range, supporting the group-wise part of the claim. Section 3.2 (p. 5) warns that more groups require more lookup tables and can adversely affect hardware performance, but does not attribute that cost to floating-point calculations of mean and variance. The mean and variance discussed in Section 2.2 (p. 3) concern Hessian eigenvalues, not group-wise quantization overhead. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [Li et al., 2019]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Findingown data

    Log2 quantization allocates 12 bins for small values, whereas 4-bit uniform quantization assigns only 1 bin for such values.

    “Compared with 4-bit uniform quantization which only assigns 1 bin for such many values, log2 method can allocate 12 bins for them.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    FQ-ViT achieves 47.2 mAP on Mask R-CNN (Swin-S) and 50.8 mAP on Cascade Mask R-CNN (Swin-S) with 8-bit weights/activations and 4-bit attention maps.

    “Our FQ-ViT significantly improves the quantization accuracy and achieves 47.2 mAP on Mask R-CNN (Swin-S) and 50.8 mAP on Cascade Mask R-CNN (Swin-S) with 8-bit on weights/activations and 4-bit on attention maps.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    LIS can perform 4-bit quantization on attention maps, store attention maps on an aggressively low bit, and replace multiplication with BitShift operator.

    “We propose Log-Int-Softmax (LIS), a novel method that can perform 4-bit quantization on attention maps.With LIS, we can store attention maps on an aggressively lowbit and replace multiplication with BitShift operator.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    FQ-ViT outperforms previous works while using lower bitwidth on attention maps.

    “Comprehensive experiments on various transformer-based architectures and benchmarks show that our Fully Quantized Vision Transformer (FQ-ViT) outperforms previous works while even using lower bitwidth on attention maps.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    All current post-training quantization methods cannot capture fully quantized vision transformers, while FQ-ViT does and achieves nearly lossless quantization even with aggressively low-bit attention maps.

    “It is obvious that all current methods can't capture fully quantized vision transformers, while our FQ-ViT does it and achieves a nearly lossless quantization even with an aggressively low-bit on attention maps.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The channel-wise ranges fluctuate more wildly in vision transformers than in ResNets; e.g., maximum range/median range is 21.6/4.2 for ResNet152 and 622.5/15.5 for Swin-B.

    “It is observed that the channel-wise ranges fluctuate more wildly in vision transformers than those in ResNets.For instance, the maximum range/median range of ResNet152 is only 21.6/4.2,while it goes up to 622.5/15.5 in Swin-B.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    4-bit uniform quantization of attention maps causes severe performance drop; e.g., DeiT-T achieves only 8.69% top-1 accuracy on ImageNet, decreasing 63.05% from 8-bit case.

    “As experimenting on quantization of attention maps from 8-bit to 4-bit with uniform quantization, all vision transformers show severe performance drop.For example, DeiT-T only results in 8.69% top-1 accuracy on Im-ageNet with 4-bit uniform quantized attention maps, decreasing 63.05% from 8-bit case.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Log2 quantization saves the extreme degradation in 4-bit quantization of attention maps and achieves equal performance as 8-bit uniform quantization with 50% less memory footprint.

    “Consequently, we save the extreme degradation in 4-bit quantization of attention maps and achieve equal performance as 8-bit uniform quantization with 50% less memory footprint.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    FQ-ViT achieves 81.20% accuracy on DeiT-B with all modules quantized to 8-bit, and 80.85% when attention maps are compressed to 4-bit.

    “For instance, our FQ-ViT achieves 81.20% accuracy on DeiT-B in the case of all modules quantized to 8-bit, and it can still achieve 80.85% accuracy when the attention maps are compressed to 4-bit.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    PTF reduces the performance degradation and inference complexity of fully quantized vision transformers.

    “present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision transformers.”
    • Also stated in: § DeiT-T DeiT-S DeiT-B Swin-T Swin-S Swin · boosted
      “we propose Power-of-Two Factor (PTF) to quantize the inputs of the LayerNorm.In this way, the quantization error is greatly reduced, and the overall computational efficiency is the same as that of layer-wise quantization thanks to the BitShift operator.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When quantized to lower bit-width (6-bit and 4-bit), uniform quantization sharply degrades and even deactivates all attention area, while LIS still presents acceptable performance similar to 8-bit.

    “However, when quantized to lower bit-width, as the 6-bit and 4-bit cases show, uniform quantization sharply degrades and even deactivates all the attention area.On the contrary, LIS still presents acceptable performance similar to 8-bit.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The model with PTF and LIS outperforms the baseline and achieves almost lossless accuracy.

    “Firstly, the model with PTF and LIS outperforms the baseline and achieves almost lossless accuracy.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    PTF only introduces little amount of BitOPs, while LIS reduces that.

    “Secondly, thanks to the BitShift operator, PTF only introduces little amount of BitOPs, while LIS reduces that.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    FQ-ViT reaches 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO.

    “For instance, we reach 84.89% top-1 accuracy with ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When both using 8-bit, uniform quantization focuses on high activation area, while LIS keeps more texture in low activation area, retaining more relative rank of attention map.

    “When both using 8-bit, uniform quantization focuses on the high activation area, while LIS keeps more texture in the low activation area, which retains more relative rank of the attention map.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Model performance benefits from higher resolution and smaller patch size, making storage and computation of attention maps a bottleneck.

    “As model performance proved to benefit from higher resolution and smaller patch size [Dosovitskiy et al., 2021], when the increasing resolution and reducing patch size, the storage and computation of attention maps become the bottleneck which directly affect the throughput and latency of inference.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [Dosovitskiy et al., 2021]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data

    Layer-wise quantization applies the same quantization parameters to all channels and, given extreme inter-channel variation, leads to intolerable quantization error.

    “Based on such extreme inter-channel variation, layer-wise quantization, which applies the same quantization parameters to all channels, will lead to an intolerable quantization error.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    PTF equips different channels with different factors rather than different quantization parameters, achieving accurate quantization on LayerNorm inputs with only one layer-wise quantization scale.

    “We propose Power-of-Two Factor (PTF), a simple yet efficient post-training method that can achieve accurate quantization on LayerNorm inputs with only one layerwise quantization scale.”
    • Also stated in: § 3.2 Power-of-Two Factor for LayerNorm Quantization · plain
      “The core idea of PTF is to equip different channels with different factors, rather than different quantization parameters.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The fixed output range (0,1) of Softmax makes the log2 function calibration-free, so quantization of attention maps is not affected by fluctuation of calibration data.

    “Firstly, comparing to Equation ( 6), the fixed output range (0, 1) of Softmax makes the log2 function calibration-free: Attn Q = Q(Attn|b) = clip(⌊-log 2 (Attn)⌉, 0, 2 b -1).( 15) This ensures that the quantization of attention maps will not be affected by the fluctuation of calibration data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationcitation

    PTF computation of mean and variance can be done in the integer domain rather than floating-point, reducing energy and area costs.

    “During inference, layer-wise parameters s and zp can be extracted, so the computation of µ,σ could be done in the integer domain rather than floating-point, which reduces the energy and area costs [Lian et al., 2019].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [Lian et al., 2019] source

  • May combine two claims.

    Interpretationreasoning

    LIS achieves integer-only inference on Softmax modules, significantly reducing inference consumption.

    “We achieve integer-only inference on Softmax modules, significantly reducing the inference consumption.”
    • Also stated in: § Integer-only Inference · plain
      “we propose Log-Int-Softmax, an integer-only, faster, low consuming Softmax”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    FQ-ViT is the first work to achieve post-training quantization for fully quantized vision transformers and lossless accuracy degradation (~1%).

    “Combining these methods, we are the first work to achieve post-training quantization for fully quantized vision transformers.”
    • Also stated in: § Abstract · hedged
      “To our knowledge, we are the first to achieve lossless accuracy degradation (∼1%) on fully quantized vision transformers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    PTQ for ViT does not quantize Softmax and LayerNorm modules, resulting in incomplete quantization.

    “However, this work does not quantize Softmax and Layer-Norm modules, resulting in an incomplete quantization.”
    • Also stated in: § 4.2 Comparison with State-of-the-art Methods · plain
      “Note that PTQ for ViT [Liu et al., 2021b] is closest to our work, however it does not quantize the LayerNorm and Softmax, while we quantize all modules.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [Liu et al., 2021b]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisenone given

    Most existing quantization methods have been developed mainly on CNNs and suffer severe degradation when applied to fully quantized vision transformers.

    “However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    LIS provides higher quantization resolution for small values and presents a more efficient integer inference for Softmax.

    “In addition, we propose Log-Int-Softmax (LIS), which provides higher quantization resolution for small values and presents a more efficient integer inference for Softmax.”
    Quote in the paperSubstantial claim
27 of 27

  • The paper relies on this without support.

    The extreme non-uniform distribution of attention maps observed in ViT-L is representative of attention maps in other vision transformer architectures.

    “Inspired by the idea of sparse attention in Dynam-icViT [Rao et al., 2021], we probe into the distribution of attention maps, as Figure 3 shows.We observe a distribution centering at a fairly small value, while only a few outliers have larger values close to 1. Averaging on all attention maps in ViT-L, about 98.8% of values are smaller than 1/16.”
1 of 1

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • FQ-ViT preliminarily explores the feasibility of integer arithmetic for part of the operations such as Softmax, but it is built on I-BERT and ignores GELU, leaving a huge gap to integer-only inference.

    “For ViTs, FQ-ViT [30] preliminarily explores the feasibility of integer arithmetic for part of the operations ( e . g ., Softmax), but it is simply built on I-BERT [19] and ignores the notable GELUoperation, leaving a huge gap between it and integeronly inference.”

    From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]

    The Integer-only Inference section describes an integer-only Softmax and says it combines log2 quantization with i-exp from Kim et al. (2021), the I-BERT paper. It does not establish that FQ-ViT is simply built on I-BERT or that it ignores GELU and therefore remains far from integer-only inference. Page numbers are unavailable in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Although the three component approximations from prior work can potentially enable integer-only inference of ViTs, the computation of high-order polynomials is inefficient in inference, and they are developed for language models that do not fit the data distribution of ViTs, leading to mismatched approximations.

    “It is worth noting that although these three component approximations can potentially enable integeronly inference of ViTs (partly verified by FQ-ViT [30]), the computation of high-order polynomials is inefficient in inference, and they are developed for language models that do not fit the data distribution of ViTs, leading to mismatched approximations.”

    From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • LIS encounters the above problems because it is built on top of I-BERT.

    “LIS also encounters the above problems, since it is simply built on top of I-BERT.”

    From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]

    The Integer-only Inference section (page not available) says LIS combines log2 quantization with i-exp from Kim et al.’s I-BERT, supporting a connection between the methods. It does not say LIS is simply built on I-BERT or that LIS encounters the unspecified “above problems” because of it. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • LIS in FQ-ViT uses Log-Int-Softmax, which builds upon I-BERT and adds the logarithmic function.

    “LIS in FQ-ViT [30]: using Log-Int-Softmax, which builds opon I-BERT and adds the logarithmic function.”

    From I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference (2023) · cites this paper as [30]

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. [36]36. Touvron et al., 2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distilla- tion through attention. In International Conference on Machine Learning. PMLR, 2021.
  2. [38]38. Peisong Wang, Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantiza- tion via bit-split and stitching. In International Conference on Machine Learning. PMLR, 2020.

Cited by

Sorted by citations

Related

Authors
Lin, Y., Zhang, T., Sun, P., Li, Z., & Zhou, S.
Affiliations
Megvii (China) (CN)
ORCID
Venue
Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
Publisher
International Joint Conferences on Artificial Intelligence Organization
Published
2022-07
Pages
1173-1179
DOI
10.24963/ijcai.2022/164
OpenAlex
W4285601701
Cited by
198
References
26
Citations by year
2026: 49 · 2025: 77 · 2024: 47 · 2023: 25
FWCI
13.51
Type
conference paper
Language
en
Open access
Locations
  • International Joint Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
Keywords
post-training quantization, 4-bit quantization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review