CoChat
Sign Up
Cited by 3Parallel Computing and Optimization TechniquesAdvanced Neural Network ApplicationsTopic Modeling

A Slice and Dice Approach to Accelerate Compound Sparse Attention on GPU

Li, Hailong, Choi, Jaewan, Ahn, Jung Ho·2022·2022 IEEE International Symposium on Workload Characterization (IISWC)
DOI 10.1109/iiswc55918.2022.00019
Open this paper in CoChat — free account

Abstract

Transformer-based models are vital to various research domains, including NLP, computer vision, and recommendation systems. However, because the attention mechanism with quadratic complexity limits computation and memory footprint in long sequences, numerous sparse attention-based transformer models are proposed to alleviate these problems. To efficiently infer these models on GPUs, prior solutions such as Triton and Sputnik have accelerated sparse attention in the sparse transformers, including optimizations for sampled dense-dense matrix multiplication (SDDMM), sparse softmax (SpSoftmax), and sparse matrix-matrix multiplication (SpMM). Although the existing optimizations have achieved impressive improvement, they have shortcomings, including low data reuse, unnecessary computation, and unnecessary memory accesses in the latest sparse transformer models based on compound sparse attention.We propose Multigrain to accelerate sparse attention by applying compound sparse GPU kernels with multi-stream for compound sparse patterns. First, we take sparse patterns with high spatial locality and execute them on our customized coarse-grained kernels, which exploit data reuse and high-performance tensor core units. Second, we execute the sparse patterns with low spatial locality on the fine-grained kernels to reduce unnecessary computation and memory accesses. Last, we process the coarse-grained and fine-grained kernels in parallel using multi-stream for the SDDMM and SpMM. In the latest sparse transformer models such as Longformer and QDS-transformer, Multigrain achieved 2.07× and 1.55× end-to-end speedup, respectively, over Triton, which only uses the coarse-grained method. We also achieved 2.08× and 1.08× end-to-end speedup, respectively, compared to Sputnik, which only uses the fine-grained kernel.

Integrity signals

Registry

Crossref · 10.1109/iiswc55918.2022.00019

Cites

41 of 41 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Previous works store sparse matrices using either coarse-grain format or fine-grain format, which is inefficient for all sparse patterns.

    “Previous works store the sparse matrices using either coarsegrain format (e.g., block coordinate format) or fine-grain format, e.g., coordinate (COO) or compressed sparse row (CSR), which is inefficient for all sparse patterns.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [120]

  • Li et al. achieve higher performance than techniques using either coarse- or fine-grain methods.

    “They achieve higher performance than techniques using either coarse- or fine-grain methods.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [120]

  • CUTLASS kernels are more efficient than Sputnik kernels.

    “For fine-grain kernels, they adapt Sputnik kernels and use CUTLASS kernels, which are more efficient than Sputnik kernels.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [120]

  • The presence of overlapped coarse/fine-grain patterns in the same row leads to inaccuracies in sparse softmax.

    “Since softmax reads all the row elements, the presence of overlapped coarse/fine-grain patterns in the same row leads to inaccuracies.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [120]

  • Transformers that use composite sparse attention, such as Longformer, are inefficient on GPUs since different sparse patterns have different locality patterns.

    “Li et al. [120] note that transformers that use composite sparse attention, such as Longformer, are inefficient on GPUs since different sparse patterns have different locality patterns.”

    From A Survey of Techniques for Optimizing Transformer Inference (2023) · cites this paper as [120]

Cites

Flagged first, then by citations

4 of the 45 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Related

Authors
Li, H., Choi, J., & Ahn, J. H.
Affiliations
Seoul National University (KR)
ORCID
Venue
2022 IEEE International Symposium on Workload Characterization (IISWC)
Publisher
IEEE
Published
2022-11
Pages
104-116
DOI
10.1109/iiswc55918.2022.00019
OpenAlex
W4311263234
Cited by
3
References
45
Citations by year
2023: 2 · 2022: 1
FWCI
0.60
Type
conference paper
Language
en
Open access
closed
Licence
Locations
IEEE International Symposium on Workload Characterization (publishedVersion)
Funders
Samsung
Topics
  • Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
sparse attention, sparse transformer, GPU kernel optimization, sparse matrix-matrix multiplication, tensor cores, Longformer, inference acceleration, data reuse

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review