The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; 7 of the 13 sources its claims cite were checked for support.
Fine-Grained Fault Sensitivity Analysis of Vision Transformers Under Soft Errors
Open accessAbstract
Over the past decade, deep neural networks (DNNs) have revolutionized the fields of computer vision (CV) and natural language processing (NLP), achieving unprecedented performance across a variety of tasks. The Vision Transformer (ViT) has emerged as a powerful alternative to convolutional neural networks (CNNs), leveraging self-attention mechanisms to capture long-range dependencies and global context. Owing to their flexible architecture and scalability, ViTs have been widely adopted in safety-critical applications such as autonomous driving, where system reliability is paramount. However, ViTs’ reliability issues induced by soft errors in large-scale digital integrated circuits have generally been overlooked. In this paper, we present a fine-grained fault sensitivity analysis of ViT variants under bit-flip fault injections, focusing on different ViT models, transformer encoder layers, weight matrix types, and attention-head dimensions. Experimental results demonstrate that the first transformer encoder layer is susceptible to soft errors due to its essential role in local and global feature extraction. Moreover, in the middle and later layers, the Multi-Layer Perceptron (MLP) sub-blocks dominate the computational workload and significantly influence representation learning, making them critical points of vulnerability. These insights highlight key reliability bottlenecks in ViT architectures when deployed in error-prone environments.
Overview
CoChat Summary
TL;DR
This paper presents a fine-grained fault sensitivity analysis of Vision Transformer variants under bit-flip soft errors, using Int8 quantization and fault injection across model-, layer-, type-, and head-wise dimensions. It finds that the first transformer encoder layer and middle/later MLP sub-blocks are key vulnerability points, with BER thresholds varying by orders of magnitude across models.
Key findings
- Model-wise BER thresholds for 1%, 5%, and 10% accuracy loss varied widely: e.g., B16-224-augreg had thresholds of 4.062 × 10⁻⁶, 2.015 × 10⁻⁵, and 3.675 × 10⁻⁵, while L16-384-augreg had 5.484 × 10⁻⁵, 1.823 × 10⁻⁴, and 2.721 × 10⁻⁴; larger models with smaller patches tended to be more robust.
- Layer-wise, B16-384-augreg and B32-384-augreg (12 layers) showed high average vulnerability factors of 5.376 and 4.339, especially in early (Layers 1–2) and middle (Layers 3–9) stages; the first layer was highly sensitive, and mlp.fc1/mlp.fc2 dominated vulnerability in middle and later layers, with attn.proj important in the first layer.
- Type-wise, base models exhibited a notable V-matrix vulnerability peak at the first layers, while large models showed Q, K, and V vulnerability factors close to 1 across all layers.
- Head-wise, sensitivity varied across attention heads; for B16-384-augreg’s first layer, top sensitive heads included Head 7 (vulnerability factor 5.1096, MAD 157.72), Head 4 (3.9056, MAD 2.15), and Head 10 (2.6086, MAD 197.26), with top sensitive heads covering both large and small attention distances.
Methods
- Seven ViT models from the HuggingFace timm library were evaluated on ImageNet-1k validation batches; all were Original ViT variants with different patch sizes and layer counts (B16-224, B16-384, B32-224, B32-384, L16-384, L32-384, H14-224).
- Fault injection used post-training Int8 quantization of FP32 weights, injected bit-flip faults directly into quantized integer weights, then dequantized; model-wise experiments swept BER from 10⁻⁸ to 10⁻², while layer-, type-, and head-wise experiments used fixed BER values.
- Each experiment was repeated 10 times with different random seeds, reporting mean, standard deviation, worst-case accuracy, and 95% confidence intervals; experiments ran on an Intel Xeon Gold 6253CL CPU, 500GB RAM, two NVIDIA Tesla V100 GPUs, CUDA 12.4, PyTorch 2.5.1.
Limitations
- The paper does not state a dedicated limitations section. Evident from the text: experiments only cover Original ViT variants, despite the framework comparison mentioning support for Swin Transformer and DeepViT.
- Evident: the fault model focuses on bit-flip errors in memory/quantized weights, not all possible soft-error locations in hardware.
- Evident: MAD results in Figure 10 are computed from static model weights without repeated runs, so no error bars are shown; the authors also note future work is needed to develop theoretical models and mathematical formulations to explain the observed sensitivity patterns.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
5 Valid · 1 Not valid · 4 Not checked · 41 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Due to their large number of parameters, ViTs often suffer from high computational complexity [18] and an increased susceptibility to hardware-induced errors [19].”
What the sources say
- Supportedcites [19] Roquet, L., Fernandes dos Santos, F., Rech, P., Traiola, M., Sentieys, O., and Kritikakou, A. (2024, January 23–27). Cross-Layer Reliability Evaluation and Efficient Hardening of Large Vision Transformers Models. Proceedings of the DAC ’24: 61st ACM/IEEE Design Automation Conference, New York, NY, USA. source
The Abstract (p. 2) states that ViTs’ large size and complexity increase the expected error rate due to hardware faults. The Introduction (p. 2) also reports high neutron-induced fault rates when GPUs execute ViTs. (LLM, conf=0.94)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results demonstrate that the first transformer encoder layer is susceptible to soft errors due to its essential role in local and global feature extraction.”
- Also stated in: § 1. Introduction · plain
“At the layer-wise level, we identified the first transformer encoder layer as particularly susceptible to soft errors, owing to its foundational role in hierarchical feature extraction.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1. Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Different models' BER thresholds can span several orders of magnitude.”
- Also stated in: § 1. Introduction · plain
“From a model-wise perspective, we observed that architectural parameters such as model size and patch size substantially influence error resilience, with BER thresholds varying by several orders of magnitude across models.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1. Introduction · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the type-wise and head-wise analyses, we revealed heterogeneous vulnerability patterns across Q, K, and V projection matrices and attention heads.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, L16-384-augreg and L32-384-orig, with 24 layers, show much lower vulnerability factors.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
18 more claims
Create a free account to see all 51 claims and every source verdict.
- Probably not a substantial claim. 15
- It may state two separate claims as one. 14
- Borderline: one of the tests for a substantial claim is unclear. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.2.1. Model-Wise
“The x-axis represents the BER on a logarithmic scale, while the y-axis represents the classification error.”
§ 3.2.1. Model-Wise
“Figure 4. Average prediction accuracy across all layers for different models and BERs:”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.2.3. Type-Wise
“the size of attn.qkv is 3 ( N + 1 ) × D , where N + 1 represents the length of input patches plus a clk \_ token , and D stands for the dimension of each 1-D patch.”
§ 2.1. ViT Model
“| ViT-B/16 | patch embedding encoder layer1~12 classification head | 1 × 1 × 768 1 × 577 × 768 768 × 3 × 16 × 16 2304 × 768 768 × 768 3072 × 768 768 × 3072 768 × 1000”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.2.3. Type-Wise
“For the B16-384-augreg model with a 384-dimensional embedding and augmentation-based regularization”
§ 2.1. ViT Model
“| ViT-B/16 | patch embedding encoder layer1~12 classification head | 1 × 1 × 768 1 × 577 × 768 768 × 3 × 16 × 16 2304 × 768 768 × 768 3072 × 768 768 × 3072 768 × 1000”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Research works printed but not linked: 3
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.
- Nazari, N., Makrani, H.M., Fang, C., Sayadi, H., Rafatirad, S., Khasawneh, K.N., and Homayoun, H. (2024, January 14–16). Forget and Rewire: Enhancing the Resilience of Transformer-based Models against {Bit-Flip} Attacks. Proceedings of the 33rd USENIX Security Symposium (USENIX Security 24), Philadelphia, PA, USA.
- (2019). IEEE Standard for Floating-Point Arithmetic (Standard No. IEEE Std 754-2019).
Cited by
SENTRY: Statistical Reliability Analysis of Vision Transformers Under Soft Errors
Bhaduri, P. K., Taheri, M., Nazari, S., Jenihhin, M., Herglotz, C., & Hübner, M. (2026) · IEEE International Conference on Edge Computing and Communications (EDGE) · doi:10.1109/edge72783.2026.00016 · Cited by 1
“Subsequent work in [17] identifies that early layers are sensitive to attention projections, while deeper layers rely more heavily on MLP weights.”
Cherezova, N., Jutman, A., & Jenihhin, M. (2026) · IEEE International On-Line Testing Symposium (IOLTS) · doi:10.1109/iolts69666.2026.11633827
“[2] analyzes ViT sensitivity to the faults in weights.”
Kaan, A. R. I. K. (2026) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10702-8 · Cited by 4
Ogawa, K., Saikawa, Y., Tomioka, Y., & Saito, H. (2025) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc58881.2025.11342788
“Previous studies by Zhou et al. [21] performed a fault sensitivity analysis of ViTs at multiple abstraction levels, including layer-wise, type-wise (e.g., Q, K, V) and head-wise evaluations.”
Metadata
- Authors
- He, J., Liu, Y., Xu, C., Liao, X., & Yang, Y.
- Affiliations
- Xidian University (CN)
- Corresponding author
- Liu, Yi
- Xu, Changqing
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2025-06-13
- Volume
- 14
- Issue
- 12
- Pages
- 2418
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W4411282014
- Cited by
- 4
- References
- 31
- Citations by year
- 2026: 3 · 2025: 1
- FWCI
- 0.89
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- Funders
- Guangdong Basic and Applied Basic Research Foundation (2025A1515012058, KJZD20240903100506009, 2025JC-YBQN-822)
- Shenzhen Science and Technology Program (2025A1515012058, KJZD20240903100506009, 2025JC-YBQN-822)
- Natural Science Basic Research Program of Shaanxi (2025A1515012058, KJZD20240903100506009, 2025JC-YBQN-822)
- Topics
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Radiation Effects in Electronics
- Physical Sciences › Computer Science › Hardware and Architecture › VLSI and Analog Circuit Testing
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- vision transformer, soft errors, fault sensitivity analysis, multilayer perceptron, safety-critical systems, autonomous driving
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.