CoChat

Fine-Grained Fault Sensitivity Analysis of Vision Transformers Under Soft Errors

Open access

The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; 7 of the 13 sources its claims cite were checked for support.

Abstract

Over the past decade, deep neural networks (DNNs) have revolutionized the fields of computer vision (CV) and natural language processing (NLP), achieving unprecedented performance across a variety of tasks. The Vision Transformer (ViT) has emerged as a powerful alternative to convolutional neural networks (CNNs), leveraging self-attention mechanisms to capture long-range dependencies and global context. Owing to their flexible architecture and scalability, ViTs have been widely adopted in safety-critical applications such as autonomous driving, where system reliability is paramount. However, ViTs’ reliability issues induced by soft errors in large-scale digital integrated circuits have generally been overlooked. In this paper, we present a fine-grained fault sensitivity analysis of ViT variants under bit-flip fault injections, focusing on different ViT models, transformer encoder layers, weight matrix types, and attention-head dimensions. Experimental results demonstrate that the first transformer encoder layer is susceptible to soft errors due to its essential role in local and global feature extraction. Moreover, in the middle and later layers, the Multi-Layer Perceptron (MLP) sub-blocks dominate the computational workload and significantly influence representation learning, making them critical points of vulnerability. These insights highlight key reliability bottlenecks in ViT architectures when deployed in error-prone environments.

Overview

CoChat Summary

TL;DR

This paper presents a fine-grained fault sensitivity analysis of Vision Transformer variants under bit-flip soft errors, using Int8 quantization and fault injection across model-, layer-, type-, and head-wise dimensions. It finds that the first transformer encoder layer and middle/later MLP sub-blocks are key vulnerability points, with BER thresholds varying by orders of magnitude across models.

Key findings

  • Model-wise BER thresholds for 1%, 5%, and 10% accuracy loss varied widely: e.g., B16-224-augreg had thresholds of 4.062 × 10⁻⁶, 2.015 × 10⁻⁵, and 3.675 × 10⁻⁵, while L16-384-augreg had 5.484 × 10⁻⁵, 1.823 × 10⁻⁴, and 2.721 × 10⁻⁴; larger models with smaller patches tended to be more robust.
  • Layer-wise, B16-384-augreg and B32-384-augreg (12 layers) showed high average vulnerability factors of 5.376 and 4.339, especially in early (Layers 1–2) and middle (Layers 3–9) stages; the first layer was highly sensitive, and mlp.fc1/mlp.fc2 dominated vulnerability in middle and later layers, with attn.proj important in the first layer.
  • Type-wise, base models exhibited a notable V-matrix vulnerability peak at the first layers, while large models showed Q, K, and V vulnerability factors close to 1 across all layers.
  • Head-wise, sensitivity varied across attention heads; for B16-384-augreg’s first layer, top sensitive heads included Head 7 (vulnerability factor 5.1096, MAD 157.72), Head 4 (3.9056, MAD 2.15), and Head 10 (2.6086, MAD 197.26), with top sensitive heads covering both large and small attention distances.

Methods

  • Seven ViT models from the HuggingFace timm library were evaluated on ImageNet-1k validation batches; all were Original ViT variants with different patch sizes and layer counts (B16-224, B16-384, B32-224, B32-384, L16-384, L32-384, H14-224).
  • Fault injection used post-training Int8 quantization of FP32 weights, injected bit-flip faults directly into quantized integer weights, then dequantized; model-wise experiments swept BER from 10⁻⁸ to 10⁻², while layer-, type-, and head-wise experiments used fixed BER values.
  • Each experiment was repeated 10 times with different random seeds, reporting mean, standard deviation, worst-case accuracy, and 95% confidence intervals; experiments ran on an Intel Xeon Gold 6253CL CPU, 500GB RAM, two NVIDIA Tesla V100 GPUs, CUDA 12.4, PyTorch 2.5.1.

Limitations

  • The paper does not state a dedicated limitations section. Evident from the text: experiments only cover Original ViT variants, despite the framework comparison mentioning support for Swin Transformer and DeepViT.
  • Evident: the fault model focuses on bit-flip errors in memory/quantized weights, not all possible soft-error locations in hardware.
  • Evident: MAD results in Figure 10 are computed from static model weights without repeated runs, so no error bars are shown; the authors also note future work is needed to develop theoretical models and mathematical formulations to explain the observed sensitivity patterns.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/electronics14122418

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

5 Valid · 1 Not valid · 4 Not checked · 41 Own results and interpretations

18 more claims

Create a free account to see all 51 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 15
  • It may state two separate claims as one. 14
  • Borderline: one of the tests for a substantial claim is unclear. 2

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 46 references

Research works printed but not linked: 3

  1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.
  2. Nazari, N., Makrani, H.M., Fang, C., Sayadi, H., Rafatirad, S., Khasawneh, K.N., and Homayoun, H. (2024, January 14–16). Forget and Rewire: Enhancing the Resilience of Transformer-based Models against {Bit-Flip} Attacks. Proceedings of the 33rd USENIX Security Symposium (USENIX Security 24), Philadelphia, PA, USA.
  3. (2019). IEEE Standard for Floating-Point Arithmetic (Standard No. IEEE Std 754-2019).

Cited by

Metadata

Authors
He, J., Liu, Y., Xu, C., Liao, X., & Yang, Y.
Affiliations
Xidian University (CN)
Corresponding author
  • Liu, Yi
  • Xu, Changqing
Venue
Electronics
Publisher
MDPI AG
Published
2025-06-13
Volume
14
Issue
12
Pages
2418
ISSN
2079-9292 (ISSN-L)
OpenAlex
W4411282014
Cited by
4
References
31
Citations by year
2026: 3 · 2025: 1
FWCI
0.89
Type
journal article
Language
en
Open access
Locations
Electronics (publishedVersion, cc-by)
Funders
  • Guangdong Basic and Applied Basic Research Foundation (2025A1515012058, KJZD20240903100506009, 2025JC-YBQN-822)
  • Shenzhen Science and Technology Program (2025A1515012058, KJZD20240903100506009, 2025JC-YBQN-822)
  • Natural Science Basic Research Program of Shaanxi (2025A1515012058, KJZD20240903100506009, 2025JC-YBQN-822)
Topics
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Radiation Effects in Electronics
  • Physical Sciences › Computer Science › Hardware and Architecture › VLSI and Analog Circuit Testing
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
vision transformer, soft errors, fault sensitivity analysis, multilayer perceptron, safety-critical systems, autonomous driving

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.