CoChat

Soft Error Reliability Analysis of Vision Transformers

Open access
Xue, Xinghua, Liu, Cheng, Wang, Ying, Yang, Bing, Luo, Tao, Zhang, Lei, Li, Huawei, Li, Xiaowei · 2023 · IEEE Transactions on Very Large Scale Integration (VLSI) Systems

The DOI matches its Crossref record; 64 of the 67 references it cites were found in a registry or an index; 2 of the 17 sources its claims cite were checked for support.

Abstract

Vision transformers (ViTs) that leverage self-attention mechanism have shown superior performance on many classical vision tasks compared to convolutional neural networks (CNNs) and gain increasing popularity recently. Existing ViTs’ works mainly optimize performance and accuracy, but ViTs’ reliability issues induced by soft errors in large-scale VLSI designs have generally been overlooked. In this work, we mainly study the reliability of ViTs and investigate the vulnerability from different architecture granularities ranging from models, layers, modules, and patches for the first time. The investigation reveals that ViTs with the self-attention mechanism are generally more resilient on linear computing including general matrix–matrix multiplication (GEMM) and full connection (FC) and show a relatively even vulnerability distribution across the patches. ViTs involve more fragile non-linear computing such as softmax and GELU compared to typical CNNs. With the above observations, we propose a lightweight block-wise algorithm-based fault-tolerance (LB-ABFT) approach to protect the linear computing implemented with distinct sizes of GEMM and apply a range-based protection scheme to mitigate soft errors in non-linear computing. According to our experiments, the proposed fault-tolerant approaches enhance ViTs’ accuracy significantly with minor computing overhead in the presence of various soft errors.

Overview

CoChat Summary

TL;DR

This paper analyzes soft-error reliability of Vision Transformers at model, layer, module, and patch granularities, finding ViTs generally more resilient than CNNs but with distinct vulnerabilities in non-linear operations and patch distribution. It proposes lightweight block-wise ABFT for linear operations and range-based protection for non-linear operations, recovering accuracy with minor overhead.

Key findings

  • ViTs are generally more resilient than ResNet under the same bit error rate: ResNet18 starts to drop at BER 3E-10, while Swin-T and CaiT-XXS-24 start to drop around 9E-10; ResNet accuracy degradation curves are sharper, while ViT curves are smoother.
  • Module-wise, FF is the most fragile ViT component; MHA has much lower vulnerability because softmax limits outputs to 0–1 and suppresses errors. Non-linear functions in ViTs show non-trivial vulnerability despite taking only a fraction of overall operations, unlike ResNet.
  • Patch-wise vulnerability factors are relatively evenly distributed across ViT input patches, whereas ResNet vulnerability is concentrated in center patches because ImageNet objects are usually centered.
  • The proposed LB-ABFT plus range-based protection can almost fully recover accuracy even when unprotected accuracy drops by more than 50%; evaluation targets accuracy loss below 2% and maximum computing overhead of 2% normalized to total ViT computing. LB-ABFT achieves comparable accuracy to layer-wise TMR with much smaller overhead and higher accuracy than standard ABFT and intensity-guided ABFT under all tested BER setups.

Methods

  • Reliability evaluation uses ImageNet-1K with 10,000 randomly selected 256×256 images, int8-quantized models ViT-B, Swin-T, DeepViT-S, CaiT-XXS-24, and ResNet18/ResNet50, with BER from 1E-11 to 1E-7. Fault injection is operation-wise bit-flip injection on multiplication and addition outputs, covering linear and non-linear functions. Vulnerability factor is the accuracy difference between a model with only the target component perfectly protected and the model without protection.
  • Fault-tolerant design: LB-ABFT protects GEMM/FC linear operations by dividing large GEMMs into sub-GEMMs and using OptBlockSizeSelection to choose block size under different BERs and models; range-based protection suppresses out-of-range outputs for softmax and GELU. Evaluation compares against standard ABFT, intensity-guided ABFT, and layer-wise TMR.

Limitations

  • The paper does not state a dedicated limitations section.
  • Evident from the text: reliability evaluation is limited to ImageNet-1K with a 10,000-image subset, int8-quantized models, four ViT variants and two ResNets, and BER from 1E-11 to 1E-7; fault injection is an operation-wise approximate bit-flip model rather than hardware-level fault injection.
  • Evident from the text: LB-ABFT block-size optimization assumes an even error distribution across GEMMs, and the GELU range-based protection uses α = 0.02 from prior work with min/max obtained by sampling inputs.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tvlsi.2023.3317138

Cites

64 of 64 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

1 Valid · 4 Not valid · 8 Not checked · 79 Own results and interpretations

37 more claims

Create a free account to see all 92 claims and to verify the 5 references still open.

See all claims — free account
  • It may state two separate claims as one. 39
  • Probably not a substantial claim. 16

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 64 references

Cited by

See all 35

Metadata

Authors
Xue, X., Liu, C., Wang, Y., Yang, B., Luo, T., Zhang, L., Li, H., & Li, X.
Affiliations
  • Chinese Academy of Sciences (CN)
  • Department of Computer Science and Technology, Harbin University of Science and Technology, Harbin, China
  • Agency for Science, Technology and Research (SG)
Venue
IEEE Transactions on Very Large Scale Integration (VLSI) Systems
Published
2023-12
Volume
31
Issue
12
Pages
2126-2136
ISSN
  • 1063-8210 (ISSN-L)
  • 1557-9999
OpenAlex
W4387385773
Cited by
35
References
67
Citations by year
2026: 16 · 2025: 11 · 2024: 7 · 2023: 1
FWCI
2.59
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Very Large Scale Integration (VLSI) Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (62174162)
  • Space Trusted Computing and Electronic Information Technology Laboratory of BICE (OBCandETL-2022-07)
Topics
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Radiation Effects in Electronics
  • Physical Sciences › Computer Science › Hardware and Architecture › VLSI and Analog Circuit Testing
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Low-power high-performance VLSI design
Keywords
vision transformer, soft errors, fault tolerance, reliability analysis, self-attention, GELU, softmax, matrix-matrix multiplication, hardware fault tolerance, VLSI design

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Soft Error Reliability Analysis of Vision Transformers. https://app.cochat.ai/papers/10.1109/tvlsi.2023.3317138
The paper
Xue, X., Liu, C., Wang, Y., Yang, B., Luo, T., Zhang, L., Li, H., & Li, X. (2023). Soft Error Reliability Analysis of Vision Transformers. IEEE Transactions on Very Large Scale Integration (VLSI) Systems. https://doi.org/10.1109/tvlsi.2023.3317138

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.