Jumping through Local Minima: Quantization in the Loss Landscape of Vision Transformers
Open accessAbstract
Quantization scale and bit-width are the most important parameters when considering how to quantize a neural network. Prior work focuses on optimizing quantization scales in a global manner through gradient methods (gradient descent & Hessian analysis). Yet, when applying perturbations to quantization scales, we observe a very jagged, highly non-smooth test loss landscape. In fact, small perturbations in quantization scale can greatly affect accuracy, yielding a 0.5 – 0.8% accuracy boost in 4-bit quantized vision transformers (ViTs). In this regime, gradient methods break down, since they cannot reliably reach local minima. In our work, dubbed Evol-Q, we use evolutionary search to effectively traverse the non-smooth landscape. Additionally, we propose using an infoNCE loss, which not only helps combat overfitting on the small calibration dataset (1, 000 images) but also makes traversing such a highly non-smooth surface easier. Evol-Q improves the top-1 accuracy of a fully quantized ViT-Base by 10.30%, 0.78%, and 0.15% for 3-bit, 4-bit, and 8-bit weight quantization levels. Extensive experiments on a variety of CNN and ViT architectures further demonstrate its robustness in extreme quantization scenarios. Our code is available at https: // github.com/enyac-group/evol-q .
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Evol-Q employs a contrastive loss mechanism to determine optimal quantization scales, utilizing a block-wise evolutionary search strategy.
“Evol-Q [52] employs a contrastive loss mechanism, drawing inspiration from self-supervised learning paradigms, to determine optimal quantization scales. This method utilizes a block-wise evolutionary search strategy to iteratively identify the best scales for each layer.”
From Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey (2024) · cites this paper as [52]
Cites
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., & Kalenichenko, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00286 · Cited by 4,114 · see citing works
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.07733 · Cited by 3,846 · see citing works
An Empirical Study of Training Self-Supervised Vision Transformers
Chen, X., Xie, S., & He, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00950 · Cited by 1,648 · see citing works
Introduction to Evolutionary Computing
Eiben, A. E., & Smith, J. E. (2015) · Natural computing series · doi:10.1007/978-3-662-44874-8 · Cited by 1,244 · see citing works
Low-bit Quantization of Neural Networks for Efficient Inference
Choukroun, Y., Kravchik, E., Yang, F., & Kisilev, P. (2019) · 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) · doi:10.1109/iccvw.2019.00363 · Cited by 404 · see citing works
Post training 4-bit quantization of convolutional networks for rapid-deployment
Banner, R., Nahshan, Y., & Soudry, D. (2019) · neural information processing systems · Cited by 261 · see citing works
Up or Down? Adaptive Rounding for Post-Training Quantization
Nagel, M., Amjad, R. A., van Baalen, M., Louizos, C., & Blankevoort, T. (2020) · arXiv · doi:10.48550/arxiv.2004.10568 · Cited by 252 · see citing works
FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer
Lin, Y., Zhang, T., Sun, P., Li, Z., & Zhou, S. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/164 · Cited by 198 · cited 3× · see citing works
“In our framework, we learn these initial quantization parameters using a fast, layer-wise framework such as FQ-ViT [20], and then use evolutionary search to adjust the scales of each attention block’s projection layers.”
MethodPost-Training Quantization for Vision Transformer
Liu, Z., Wang, Y., Han, K., Ma, S., & Gao, W. (2021) · arXiv · doi:10.48550/arxiv.2106.14156 · Cited by 192 · see citing works
BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q., Yu, F., Wang, W., & Gu, S. (2021) · arXiv · doi:10.48550/arxiv.2102.05426 · Cited by 190 · see citing works
PTQ4ViT: Post-Training Quantization for Vision Transformers with Twin Uniform Quantization
Yuan, Z., Xue, C., Chen, Y., Wu, Q., & Sun, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19775-8_12 · Cited by 163 · see citing works
BinaryBERT: Pushing the Limit of BERT Quantization
Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q., Lyu, M. R.-T., & King, I. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.334 · Cited by 158 · see citing works
Loss aware post-training quantization
Nahshan, Y., Chmiel, B., Baskin, C., Zheltonozhskii, E., Banner, R., Bronstein, A. M., & Mendelson, A. (2021) · Machine Learning · doi:10.1007/s10994-021-06053-z · Cited by 155 · see citing works
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Li, Z., & Gu, Q. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01565 · Cited by 141 · cited 1× · see citing works
“Quantization-Aware Training (QAT) has shown impressive results on vision transformers [30, 19, 18, 17], yet these methods consider training with the entire dataset rather than an unlabeled calibration dataset.”
MethodData-Free Network Quantization With Adversarial Knowledge Distillation
Choi, Y., Choi, J., El-Khamy, M., & Lee, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00363 · Cited by 135 · see citing works
Li, Z., Sun, M., Lu, A., Ma, H., Yuan, G., Xie, Y., Tang, H., Li, Y., Leeser, M., Wang, Z., et al. (2022) · 2022 32nd International Conference on Field-Programmable Logic and Applications (FPL) · doi:10.1109/fpl57034.2022.00027 · Cited by 85 · see citing works
Patch Similarity Aware Data-Free Quantization for Vision Transformers
Li, Z., Ma, L., Chen, M., Xiao, J., & Gu, Q. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20083-0_10 · Cited by 61 · see citing works
EasyQuant: Post-training Quantization via Scale Optimization
Wu, D., Tang, Q., Zhao, Y., Zhang, M., Fu, Y., & Zhang, D. (2020) · arXiv · doi:10.48550/arxiv.2006.16669 · Cited by 44 · see citing works
A Deep Look into Logarithmic Quantization of Model Parameters in Neural Networks
Cai, J., Takemoto, M., & Nakajo, H. (2018) · Proceedings of the 10th International Conference on Advances in Information Technology · doi:10.1145/3291280.3291800 · Cited by 31 · see citing works
Neural Network Quantization with AI Model Efficiency Toolkit (AIMET)Preprint
Siddegowda, S., Fournarakis, M., Nagel, M., Blankevoort, T., Patel, C., & Khobare, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.08442 · Cited by 27 · see citing works
Network Binarization via Contrastive Learning
Shang, Y., Xu, D., Zong, Z., Nie, L., & Yan, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20083-0_35 · Cited by 26 · see citing works
Q-ViT: Fully Differentiable Quantization for Vision TransformerPreprint
Li, Z., Yang, T., Wang, P., & Cheng, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.07703 · Cited by 18 · see citing works
Synergistic Self-supervised and Quantization Learning
Cao, Y.-H., Sun, P., Huang, Y., Wu, J., & Zhou, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20056-4_34 · Cited by 11 · see citing works
TerViT: An Efficient Ternary Vision Transformer
Xu, S., Li, Y., Ma, T., Zeng, B., Zhang, B., Gao, P., & Lv, J. (2022) · arXiv · doi:10.48550/arxiv.2201.08050 · Cited by 5 · see citing works
Fu, Y., Yu, Q., Li, M., Ouyang, X., Chandra, V., & Lin, Y. (2022) · ACM/IEEE Design Automation Conference (DAC) · doi:10.1145/3489517.3530419 · Cited by 4 · see citing works
Flagged first, then by citations
3 of the 31 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
AdaLog: Post-training Quantization for Vision Transformers with Adaptive Logarithm Quantizer
Wu, Z., Chen, J., Zhong, H., Huang, D., & Wang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73383-3_24 · Cited by 20 · see citing works
GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers
Choi, D., & Kim, H. K. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i15.33759 · Cited by 18 · see citing works
Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction
Zhong, Y., You, H., Hu, J., Zhang, Y., & Ji, R. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3528042 · Cited by 19 · see citing works
CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-training Quantization of ViTs
Ramachandran, A., Kundu, S., & Krishna, T. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72855-6_18 · Cited by 14 · see citing works
Multi-Objective Convex Quantization for Efficient Model Compression
Fan, C., Guo, D., Wang, Z., & Wang, M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3521589 · Cited by 10 · see citing works
AHCPTQ: Accurate and Hardware-Compatible Post-Training Quantization for Segment Anything Model
Zhang, W., Zhong, Y., Ando, S., & Yoshioka, K. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02078 · Cited by 7 · see citing works
HAQ-ViT: A hardware-aware post-training quantization for efficient vision transformer inference
Zhao, S., Wang, Z., Sun, H., Wang, S., Chen, Z., Fu, C., Yang, J., & Sawan, M. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115830 · Cited by 5 · see citing works
Model Quantization and Hardware Acceleration for Vision Transformers: A Comprehensive Survey
Du, D., Gong, G., & Chu, X. (2024) · arXiv · doi:10.48550/arxiv.2405.00314 · Cited by 4 · see citing works
“3) Layer-by-layer Reconstruction error : Evol-Q [52] employs a contrastive loss mechanism, drawing inspiration from self-supervised learning paradigms, to determine optimal quantization scales.”
BackgroundAIRCHITECT v2: Learning the Hardware Accelerator Design Space Through Unified Representations
Seo, J., Ramachandran, A., Chuang, Y.-C., Itagi, A., & Krishna, T. (2025) · Design, Automation, and Test in Europe · doi:10.23919/date64628.2025.10993035 · Cited by 4 · see citing works
Evolutionary Machine Learning Meets Self-Supervised Learning: A Comprehensive Survey
Vinhas, A., Correia, J., & Machado, P. (2026) · IEEE Access · doi:10.1109/access.2026.3717264 · Cited by 2 · see citing works
I&S-ViT: An Inclusive & Stable Method for Post-Training ViTs Quantization
Zhong, Y., Hu, J., Lin, M., Chen, M., & Ji, R. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3610466 · Cited by 7 · see citing works
Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformers
Zhong, Y., Zhou, Y., Zhang, Y., Sui, W., Li, S., Li, Y., Chao, F., & Ji, R. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01160 · Cited by 2 · see citing works
Wang, G., Lu, Y., Xia, Z., Wei, X., Shi, L., & Zhang, B. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5693-9_9
MTLQ-ViT: Multi-granularity Tail-enhanced Logarithmic Quantization for Vision Transformers
Kang, Y., Xu, S., Lin, Q., He, K., Gao, Z., Chen, Z., & Pu, B. (2026) · Pattern Recognition · doi:10.1016/j.patcog.2026.113769
Tanveer, H., Ahmadilivani, M. H., Jenihhin, M., & Daneshtalab, M. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.104001
- Authors
- Frumkin, N., Gope, D., & Marculescu, D.
- Affiliations
- The University of Texas at Austin (US)
- Arm Inc.
- ORCID
- Marculescu, Diana
- Venue
- 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2023-10-01
- Pages
- 16932-16942
- OpenAlex
- W4390871684
- Cited by
- 14
- References
- 31
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 3
- FWCI
- 1.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2107085)
- Cockrell School of Engineering, University of Texas at Austin
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- neural network quantization, vision transformer, loss landscape, local minima, evolutionary search, InfoNCE loss, nonsmooth optimization, weight quantization, 4-bit quantization, calibration dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
