CoChat

The DOI matches its Crossref record; 79 of the 86 references it cites were found in a registry or an index; 2 of the 15 sources its claims cite were checked for support.

EQ-ViT: Algorithm-Hardware Co-Design for End-to-End Acceleration of Real-Time Vision Transformer Inference on Versal ACAP Architecture

Open access
Dong, Peiyan, Zhuang, Jinming, Yang, Zhuoping, Ji, Shixin, Li, Yanyu, Xu, Dongkuan, Huang, Heng, Hu, Jingtong, Jones, Alex K., Shi, Yiyu, Wang, Yanzhi, Zhou, Peipei · 2024 · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems

Abstract

While Vision Transformers (ViTs) have shown consistent progress in computer vision, deploying them for real-time decision-making scenarios (< 1 ms) is challenging. Current computing platforms like CPUs, GPUs, or FPGA-based solutions struggle to meet this deterministic low-latency real-time requirement, even with quantized ViT models. Some approaches use pruning or sparsity to reduce model size and latency, but this often results in accuracy loss. To address the aforementioned constraints, in this work, we propose EQ-ViT, an end-to-end acceleration framework with novel algorithm and architecture co-design features to enable real-time ViT acceleration on AMD Versal Adaptive Compute Acceleration Platform (ACAP). The contributions are four-fold. First, we perform in-depth kernel-level performance profiling & analysis and explain the bottlenecks for existing acceleration solutions on GPU, FPGA, and ACAP. Second, on the hardware level, we introduce a new spatial and heterogeneous accelerator architecture, EQ-ViT architecture. This architecture leverages the heterogeneous features of ACAP, where both FPGA and artificial intelligence engines (AIEs) coexist on the same system-on-chip (SoC). Third, On the algorithm level, we create a comprehensive quantization-aware training strategy, EQ-ViT algorithm. This strategy concurrently quantizes both weights and activations into 8-bit integers, aiming to improve accuracy rather than compromise it during quantization. Notably, the method also quantizes nonlinear functions for efficient hardware implementation. Fourth, we design EQ-ViT automation framework to implement the EQ-ViT architecture for four different ViT applications on the AMD Versal ACAP VCK190 board, achieving accuracy improvement with 2.4%, and average speedups of 315.0x, 3.39x, 3.38x, 14.92x, 59.5x, 13.1x over computing solutions of Intel Xeon 8375C vCPU, Nvidia A10G, A100, Jetson AGX Orin GPUs, and AMD ZCU102, U250 FPGAs. The energy efficiency gains are 62.2x, 15.33x, 12.82x, 13.31x, 13.5x, 21.9x.

Overview

CoChat Summary

TL;DR

EQ-ViT is an algorithm-hardware co-design framework for real-time Vision Transformer inference on AMD Versal ACAP, combining INT8 quantization-aware training with heterogeneous AIE/FPGA spatial accelerators. It reports sub-1 ms latency on VCK190 for four ViT applications, with accuracy improvements over baselines and large speedups/energy-efficiency gains over CPU, GPU, and FPGA solutions.

Key findings

  • On AMD Versal ACAP VCK190, EQ-ViT achieves latencies of 0.56 ms, 0.46 ms, 0.89 ms, and 0.61 ms for four ViT applications under a 1 ms latency budget with batch size 6.
  • Average speedups are 315.0x, 3.39x, 3.38x, 14.93x, 59.5x, and 13.1x over Intel Xeon 8375C vCPU, Nvidia A10G, A100, Jetson AGX Orin GPUs, and AMD ZCU102, U250 FPGAs; energy-efficiency gains are 62.2x, 15.33x, 12.82x, 13.31x, 13.5x, and 21.9x.
  • EQ-ViT improves accuracy up to 2.4% over baseline, and over Q-ViT by 0.4%–0.9% on ImageNet, 0.3%–0.4% on Cifar-100, and 0.2% on Cifar-10; Cifar-100 and Cifar-10 gains over baseline are up to 1.4% and 0.8%, respectively.
  • Hardware-efficient nonlinear quantization saves Softmax DSP and LUT by 7.0x and 14.48x, LUTMem by 18834x, and total RAM by 9.19x; GeLU reduces REG and LUT by 137x and 142x. The MIP model predicts on-board latency with less than 4% error, and all four cases use over 98.5% AIE.

Methods

  • Algorithm-hardware co-design: activation-aware QAT quantizes weights and activations to INT8 and quantizes nonlinear functions; hardware maps MM/BMM kernels to AIEs and non-MM kernels to PL, with on-chip data forwarding, multiple spatial accelerators, and fine-grained pipelining.
  • Data and training: experiments use ImageNet-1k, Cifar-100, and Cifar-10 with DeiT variants and LV-ViT; training is in PyTorch on 4 NVIDIA V100 GPUs, with outlier-predictable training following Q-ViT with DGD.
  • Evaluation: implemented on AMD Versal ACAP VCK190 and compared against CPU, GPU, and FPGA solutions; analysis includes latency, energy efficiency, accuracy versus PTQ/QAT methods, resource utilization, and MIP model versus on-board implementation.

Limitations

  • The paper states that if a model cannot fit on a single board, partitioning it onto multiple devices is left as future work.
  • The A8W4 results on AMD Versal VEK280 are described as estimated, not measured on hardware.
  • Evident from the text: evaluation is limited to four ViT applications and primarily one ACAP board, and the accuracy advantage over full-precision models is partly attributed to DGD distillation integrated into the training setting.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tcad.2024.3443692

Cites

79 of 79 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 4 Not valid · 14 Not checked · 78 Own results and interpretations

40 more claims

Create a free account to see all 96 claims and to verify the 3 references still open.

See all claims — free account
  • It may state two separate claims as one. 45
  • The quote only partly states this claim. 4
  • Probably not a substantial claim. 7

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 86 references

Research works printed but not linked: 11

  1. Rafie, 2021, Autonomous vehicles drive AI advances for edge computing.
  2. 2023, Colliding Particles Not Cars: CERN’S Machine Learning Could Help Self-Driving Cars
  3. Vanholder, Efficient inference with tensorRT, 1, 1
  4. 2024, Nvidia Nsight systems
  5. Nvidia TensorRT documentation
  6. contributors, 2022, 68–95–99.7 rule
  7. System management interface SMI | NVIDIA developer
  8. ViT Github
  9. 2023, Board Evaluation and Management Tool
  10. NVIDIA H100 tensor core GPU architecture
  11. Ma, Rammer: Enabling holistic deep learning compiler optimizations with {rTasks}, 881

Cited by

See all 35

Metadata

Authors
Dong, P., Zhuang, J., Yang, Z., Ji, S., Li, Y., Xu, D., Huang, H., Hu, J., Jones, A. K., Shi, Y., Wang, Y., & Zhou, P.
Affiliations
  • Northeastern University (US)
  • Brown University (US)
  • North Carolina State University (US)
  • University of Maryland, College Park (US)
  • University of Pittsburgh (US)
  • Syracuse University (US)
  • University of Notre Dame (US)
Venue
IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
Published
2024-11
Volume
43
Issue
11
Pages
3949-3960
ISSN
  • 0278-0070 (ISSN-L)
  • 1937-4151
OpenAlex
W4404102506
Cited by
35
References
86
Citations by year
2026: 17 · 2025: 16 · 2024: 2
FWCI
4.88
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion)
Funders
  • NSF (2213701, 2217003, 2133267, 2122320, 2324864, 2324937, 2328972)
  • NIH (R01EB033387)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Hardware and Architecture › Embedded Systems Design Techniques
  • Physical Sciences › Engineering › Electrical and Electronic Engineering › Embedded Systems and FPGA Applications
Keywords
algorithm-hardware co-design, FPGA acceleration, energy efficiency, system-on-chip

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: EQ-ViT: Algorithm-Hardware Co-Design for End-to-End Acceleration of Real-Time Vision Transformer Inference on Versal ACAP Architecture. https://app.cochat.ai/papers/10.1109/tcad.2024.3443692
The paper
Dong, P., Zhuang, J., Yang, Z., Ji, S., Li, Y., Xu, D., Huang, H., Hu, J., Jones, A. K., Shi, Y., Wang, Y., & Zhou, P. (2024). EQ-ViT: Algorithm-Hardware Co-Design for End-to-End Acceleration of Real-Time Vision Transformer Inference on Versal ACAP Architecture. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems. https://doi.org/10.1109/tcad.2024.3443692

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.