The DOI matches its DataCite record; 88 of the 88 references it cites were found in a registry or an index; 40 of the 48 sources its claims cite were checked for support.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Open accessAbstract
Machine learning's explosive growth has produced a proliferation of accelerator architectures, yet no framework maps a workload's computational character to the hardware that serves it best. We build that framework: decomposing ML workloads — transformers, CNNs, GNNs, diffusion and state-space models — into core compute primitives characterized by arithmetic intensity, memory-access pattern, and parallelism, and evaluating how GPUs, TPUs, systolic arrays, dataflow processors, wafer-scale engines, and neuromorphic chips serve them. Using hierarchical roofline and utilization modelling, we show the binding constraint has shifted from peak FLOPS to memory bandwidth and data-movement energy — which quantization only sharpens — and consolidate eight persistent, unsolved problems. These motivate the central contribution: the Fused Memory-Compute Tile (FMCT, codename Habanero), an SRAM-first, HBM-free accelerator combining a three-mode (GEMM / bandwidth / fused) phase-adaptive tile switched by compiler directive, a non-linear unit fused into the systolic datapath, compute-in-memory as a selectable mode rather than the whole architecture, and an inline KV-cache compression engine. A validated component-level energy model shows ~4-7× batch-1 efficiency gains over HBM GPUs. We position FMCT against its closest SRAM-first, no-HBM analogs — d-Matrix Corsair and Tenstorrent — and OpenAI's HBM-retaining Jalapeño ASIC, closing with principles for memory-centric, workload-adaptive accelerators.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
FlashAttention is an IO-aware exact attention algorithm that uses tiling and recomputation to avoid materializing the large \(N \times N\) attention matrix in GPU HBM, reducing memory reads/writes while computing exact attention. It speeds up Transformer training and enables longer contexts, including first better-than-chance Transformer results on Path-X and Path-256.
Key findings
- FlashAttention requires fewer HBM accesses than standard attention: \(\Theta(N^2 d^2 M^{-1})\) vs. \(\Theta(Nd + N^2)\), and is up to 9× fewer HBM accesses for typical \(d\) and \(M\).
- It trains models faster: 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) over the MLPerf 1.1 record, up to 3× speedup on GPT-2 (seq. length 1K), and 2.4× on Long-Range Arena (seq. length 1K–4K).
- Longer context improves quality: 0.7 better perplexity on GPT-2 and 6.4 points of lift on long-document classification; FlashAttention achieves 61.4% accuracy on Path-X (16K), and block-sparse FlashAttention achieves 63.1% on Path-256 (64K).
- Attention benchmarking: FlashAttention is up to 3× faster than standard attention for common sequence lengths and up to 20× more memory efficient than exact attention baselines; block-sparse FlashAttention is faster than all existing approximate attention methods tested.
Methods
- Algorithm design: exact attention computed by tiling Q, K, V blocks into on-chip SRAM, incrementally performing softmax reduction, and recomputing attention in the backward pass from stored softmax statistics instead of storing the \(N \times N\) attention matrix.
- Analysis: proves IO complexity for forward and backward passes, plus a lower bound showing no exact attention algorithm can asymptotically improve HBM accesses over all SRAM sizes; extends to block-sparse FlashAttention with IO complexity scaled by sparsity.
- Experiments: CUDA implementation benchmarked on A100, RTX 3090, and T4 GPUs; trained BERT-large, GPT-2, Long-Range Arena, MIMIC-III, ECtHR, Path-X, and Path-256, comparing runtime, memory, and accuracy against exact, approximate, and sparse attention baselines.
Limitations
- The paper states that building IO-aware attention implementations currently requires writing a new CUDA kernel for each new attention implementation, using a lower-level language than PyTorch, with significant engineering effort, and implementations may not transfer across GPU architectures.
- The paper states its IO-aware implementation is optimal within constants for single-GPU attention, but multi-GPU attention adds another memory hierarchy and is left as future work.
- The paper states the lower bound is proved over a subrange of SRAM sizes \(M\), and parameterized complexity lower bounds in terms of \(M\) are left as future work; the text also makes evident that some approximate attention methods become faster than FlashAttention beyond sequence length 1K, though block-sparse FlashAttention remains faster than all tested approximate methods.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
38 Own result · 35 No basis given · 17 Own interpretation · 14 Partly supported · 7 Supported · 3 Couldn’t confirm · 3 Own method · 1 Mixed support · 1 Sources not found · 1 Not checked yet · accuracy not yet measured
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“We analyze the IO complexity of FlashAttention , showing that it requires fewer HBM accesses than standard attention, and is optimal for a range of SRAM sizes.”
- Also stated in: § 1 Introduction · boosted
“We analyze the IO complexity [1] of FlashAttention , proving that it requires 𝑂 𝑁 2 𝑑 2 𝑀 1 HBM accesses where 𝑑 is the head dimension and 𝑀 is the size of SRAM, as compared to Ω 𝑁𝑑 𝑁 2 of standard attention.”
What the sources say
- Could not verifycites [1] [1] Alok Aggarwal and S Vitter, Jeffrey. The input/output complexity of sorting and related problems. Communications of the ACM , 31(9):1116-1127, 1988. - sourceExperimental — accuracy not yet measured
The Introduction and Section 3 describe I/O bounds for sorting-related problems and transfers between disk and internal memory. A search of our copy for FlashAttention, HBM, SRAM, and attention found no passage comparing FlashAttention with standard attention, so it cannot verify the claimed reduction in HBM accesses. (LLM, conf=0.78)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · boosted
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FlashAttention trains Transformers faster than existing baselines: 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) compared to the MLPerf 1.1 training speed record, 3 speedup on GPT-2 (seq. length 1K), and 2.4 speedup on long-range arena (seq. length 1K-4K).”
- Also stated in: § 1 Introduction · plain
“We train BERT-large (seq. length 512) 15% faster than the training speed record in MLPerf 1.1 [58]”
What the sources say
- Partly supportedcites [58] [58] Peter Mattson, Christine Cheng, Gregory Diamos, Cody Coleman, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, et al. Mlperf training benchmark. Proceedings of Machine Learning and Systems , 2:336-349, 2020. - sourceExperimental — accuracy not yet measured
Section 3.2 (p. 6) describes MLPerf’s end-to-end time-to-train metric, supporting the identification of MLPerf as a training benchmark. But Section 5 (p. 9) reports only rounds v0.5 and v0.6; our copy does not establish a v1.1 BERT-large speed record or a 15% FlashAttention speedup against it. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“3 speedup on GPT-2 (seq. length 1K)”
- Also stated in: § 1 Introduction · plain
“GPT2 (seq. length 1K) 3 faster than baseline implementations from HuggingFace [87] and Megatron-LM [77]”
What the sources say
- Could not verifycites [87] [87] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics. URL https://www sourceExperimental — accuracy not yet measured
The Deployment section reports speedups from ONNX optimization of models including GPT-2, but does not discuss FlashAttention. Searches for FlashAttention and a 3x speedup at sequence length 1K found no passage addressing the claimed result; the ONNX result does not verify or contradict it. Page numbers were unavailable in our copy. (LLM, conf=0.76)
- Partly supportedcites [77] [77] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 , 2019. - sourceExperimental — accuracy not yet measured
Appendix D.2 (p. 14) reports a 2.98-fold training speedup for a 1.2-billion-parameter model using eight GPUs rather than one. That result concerns model parallelism, not FlashAttention; the passage does not specify a 1K sequence length or the claimed baseline comparison. (LLM, conf=0.83)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“We analyze the IO complexity [1] of FlashAttention , proving that it requires 𝑂 𝑁 2 𝑑 2 𝑀 1 HBM accesses where 𝑑 is the head dimension and 𝑀 is the size of SRAM, as compared to Ω 𝑁𝑑 𝑁 2 of standard attention.”
- Also stated in: § 3.2 Analysis: IO Complexity of FlashAttention · plain
“Theorem 2. Let 𝑁 be the sequence length, 𝑑 be the head dimension, and 𝑀 be size of SRAM with 𝑑 𝑀 𝑁𝑑 . Standard attention (Algorithm 0) requires Θ 𝑁𝑑 𝑁 2 HBM accesses, while FlashAttention (Algorithm 1) requires Θ 𝑁 2 𝑑 2 𝑀 1 HBM accesses.”
What the sources say
- Supportedcites [1] [1] Alok Aggarwal and S Vitter, Jeffrey. The input/output complexity of sorting and related problems. Communications of the ACM , 31(9):1116-1127, 1988. - source
The citation attaches to “IO complexity,” not to the FlashAttention bounds that the citing authors say they prove. The source’s Introduction defines memory and block-transfer parameters, and Section 3 describes its I/O model, supporting the citation as an identification of that framework. (LLM, conf=0.89)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.2 Analysis: IO Complexity of FlashAttention · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the first Transformers to achieve better-than-chance performance on the Path-X challenge (seq. length 16K, 61.4% accuracy)”
- Also stated in: § 1 Introduction · plain
“FlashAttention enables the first Transformer that can achieve better-than-chance performance on the Path-X [80] challenge, solely from using a longer sequence length (16K).”
What the sources say
- Partly supportedcites [80] [80] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations , 2020. - sourceExperimental — accuracy not yet measured
Section 2.2.6 (p. 4) identifies Path-X as a 16K-input benchmark, and Section 3.3 (p. 7) reports that the models tested achieved at best 50%. The paper does not report FlashAttention achieving 61.4% or establish that it was the first Transformer to exceed chance on Path-X. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We argue that a missing principle is making attention algorithms IOaware -accounting for reads and writes between levels of GPU memory.”
- Also stated in: § 1 Introduction · plain
“In this paper, we argue that a missing principle is making attention algorithms IO-aware [1]-that is, carefully accounting for reads and writes to different levels of fast and slow memory (e.g., between fast GPU on-chip SRAM and relatively slow GPU high bandwidth memory, or HBM [45], Figure 1 left).”
What the sources say
- Partly supportedcites [1] [1] Alok Aggarwal and S Vitter, Jeffrey. The input/output complexity of sorting and related problems. Communications of the ACM , 31(9):1116-1127, 1988. - sourceExperimental — accuracy not yet measured
The Introduction describes studying algorithms by the number of I/Os, and Section 3 models transfers between disk and internal memory. That supports the general principle of accounting for data movement, but the paper does not address attention algorithms or transfers between levels of GPU memory. (LLM, conf=0.85)
- Not checked — no full text of the sourcecites [45] [45] Zhe Jia, Marco Maggioni, Benjamin Staiger, and Daniele P Scarpazza. Dissecting the nvidia Volta GPU architecture via microbenchmarking. arXiv preprint arXiv:1804.06826 , 2018. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our reproduced baselines perform better than as reported in the original comparison [80].”
What the sources say
- Supportedcites [80] [80] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations , 2020. - source
The cited paper is the original comparison: its benchmark table in §3.2 (p. 6) reports baseline results across the Long Range Arena tasks. The citing authors’ statement about their own reproduced baselines is their comparison with those reported results, not a finding attributed to this source. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This technique [60] has been used in the literature [51, 66] to show that attention computation does not need quadratic extra memory (though the number of HBM accesses is still quadratic, resulting in slow run-time).”
- Also stated in: § B Algorithm Details · plain
“Though they reduce the amount of extra memory required, naively they still incur quadratic HBM accesses, resulting in slower execution speed.”
What the sources say
- Could not verifycites [60] [60] Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867 , 2018. - sourceExperimental — accuracy not yet measured
Section 5 (Benchmarking) links memory accesses to Softmax performance, but the paper does not address memory-efficient attention or whether its HBM accesses are quadratic. I searched our copy for attention, quadratic accesses, and the claimed run-time consequence; the passages concern Softmax and Softmax+TopK instead. (LLM, conf=0.78)
- Partly supportedcites [51] [51] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In The International Conference on Machine Learning (ICML) , 2020. - sourceExperimental — accuracy not yet measured
The memory-efficient attention discussion (Section 2, p. 2) says attention can use memory proportional to sequence length but may be less efficient. Table 1 (Section 2, pp. 4–5) gives memory-efficient attention quadratic time complexity. The paper does not identify quadratic HBM accesses as the reason for slower execution. (LLM, conf=0.87)
- Partly supportedcites [66] [66] Markus N Rabe and Charles Staats. Self-attention does not need 𝑂 𝑛 2 memory. arXiv preprint arXiv:2112.05682 , 2021. - sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says the memory-efficient algorithm avoids quadratic memory use but still has quadratic time complexity. The Inference section (p. 4) reports roughly similar runtime to standard attention; it does not establish quadratic HBM accesses or attribute slow runtime to them. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § B Algorithm Details · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The butterfly pattern we use for our block-sparse attention is motivated by the fact that butterfly matrices [15, 64] and their products have been shown to be able to express any structured matrices with almost optimal runtime and number of parameters [16, 20].”
What the sources say
- Partly supportedcites [15] [15] Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, and Christopher Ré. Learning fast algorithms for linear transforms using butterfly factorizations. In International Conference on Machine Learning (ICML) , 2019. - sourceExperimental — accuracy not yet measured
Section 3.3 (pp. 6–7) says the butterfly-product hierarchy represents several important transforms at low depth and can represent all square matrices at linear depth. The Introduction (p. 1) gives O(N) parameters and O(N log N) multiplication for a class of structured matrices. These passages do not establish that *any* structured matrix has a butterfly representation with almost-optimal runtime and parameter count; §4.1 (p. 7) says some structured matrices are not perfectly captured by the BPBP class. (LLM, conf=0.86)
- Not matchedcites [64] [64] D Stott Parker. Random butterfly transformations with applications in computational linear algebra. 1995. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [16] [16] Tri Dao, Nimit Sohoni, Albert Gu, Matthew Eichhorn, Amit Blonder, Megan Leszczynski, Atri Rudra, and Christopher Ré. Kaleidoscope: An efficient, learnable representation for all structured linear maps. In International Conference on Learning Representations (ICLR) , 2020. - source
Section 2 (p. 3) states that kaleidoscope matrices, composed of products of butterfly matrices, represent all structured matrices with almost optimal parameter count and runtime. The Introduction (p. 2) specifies that this means structured linear maps with fast arithmetic algorithms, with complexity nearly tight up to logarithmic factors. (LLM, conf=0.94)
- Not matchedcites [20] [20] Christopher De Sa, Albert Gu, Rohan Puttagunta, Christopher Ré, and Atri Rudra. A two-pronged progress in structured dense matrix vector multiplication. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1060-1079. SIAM, 2018. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Training Transformers with longer sequences with FlashAttention improves performance on the MIMIC-III [47] and ECtHR [6, 7] datasets.”
What the sources say
- Supportedcites [47] [47] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016. - source
The Methods section identifies MIMIC-III as the third iteration of the MIMIC critical care database. Citation [47] is attached to the name of that dataset, not to the claim that longer Transformer sequences improve performance. (LLM, conf=0.93)
- Supportedcites [6] [6] Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. Neural legal judgment prediction in English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4317-4323, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1424. URL https://www.aclweb.org/anthology/P19-1424 . - source
The citation is attached to the ECtHR dataset, and this paper introduces a dataset of European Court of Human Rights cases. The Introduction and §2 (“ECHR Dataset”; page not specified in our copy) identify it as a dataset of approximately 11,500 cases. (LLM, conf=0.91)
- Supportedcites [7] [7] Ilias Chalkidis, Manos Fergadiotis, Dimitrios Tsarapatsanis, Nikolaos Aletras, Ion Androutsopoulos, and Prodromos Malakasiotis. Paragraph-level rationale extraction through regularization: A case study on european court of human rights cases. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics , Mexico City, Mexico, 2021. Association for Computational Linguistics. - source
The citation is attached to “ECtHR datasets,” identifying the dataset rather than attributing the FlashAttention performance finding to this source. The Introduction describes the paper’s dataset as comprising 11,000 ECtHR cases. No page number is available in our copy. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
- It may state two separate claims as one. 48
- Probably not a substantial claim. 19
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“For large sequence lengths 𝑁 , 𝑠 is often set to 𝑁 1 2 [11] or 𝑁 1 log 𝑁 [3, 17, 92], resulting in Θ 𝑁 𝑁 or Θ 𝑁 log 𝑁 IO complexity.”
What the sources say
- Disputedcites [11] [11] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. - sourceExperimental — accuracy not yet measured
The citation to Child et al. is attached to the logarithmic-complexity option. Instead, Section 4.3 (p. 4) chooses a stride close to √N, and Section 4.2 (p. 4) gives O(N√N) computation. The paper does not give the attributed Θ(N log N) IO complexity. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
- Could not verifycites [3] [3] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020. - sourceExperimental — accuracy not yet measured
Section 3.1 (p. 3) gives sliding-window attention a computation complexity of O(n × w) for a fixed window size. Our copy does not say that sparsity s is set to N⁻¹ log N or give a Θ(N log N) IO complexity; a search for those terms found no supporting passage. (LLM, conf=0.73)
- Partly supportedcites [17] [17] Tri Dao, Beidi Chen, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models. In International Conference on Learning Representations (ICLR) , 2022. - sourceExperimental — accuracy not yet measured
Section 3.2 (p. 5) says flat butterfly matrices have O(n log k) nonzeros; with maximum stride k = n, this supports an O(N log N) sparsity pattern. The Problem Formulation (p. 17) relates memory accesses to nonzero blocks. The paper does not state that s is often set to either claimed value, or establish the stated Θ IO complexities, including the Θ(N√N) case. (LLM, conf=0.78)
- Partly supportedcites [92] [92] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems , 33, 2020. - sourceExperimental — accuracy not yet measured
Section 2 (pp. 3–4) describes sparse attention and its reduced complexity, including attention to a limited number of random keys. It does not specify a sparsity fraction of (log N)/N or establish Θ(N log N) IO complexity; the Θ(N√N) alternative is attached to a different citation in the citing sentence. (LLM, conf=0.76)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FlashAttention and block-sparse FlashAttention enable longer context in Transformers, yielding higher quality models (0.7 better perplexity on GPT-2 and 6.4 points of lift on long-document classification) and entirely new capabilities: the first”
- Also stated in: § 1 Introduction · plain
“We observe a 0.7 improvement in perplexity on GPT-2 and 6.4 points of lift from modeling longer sequences on long-document classification [13].”
- Also stated in: § 4 Experiments · plain
“Modeling longer sequences yields 6.4 points of lift on two longdocument classification tasks.”
What the sources say
- Partly supportedcites [13] [13] Xiang Dai, Ilias Chalkidis, Sune Darkner, and Desmond Elliott. Revisiting transformer-based models for long document classification. arXiv preprint arXiv:2204.06683 , 2022. - sourceExperimental — accuracy not yet measured
The Introduction and Conclusions report that processing more text can improve long-document classification, and §3.1 describes Longformer as a way to process longer sequences. Our copy does not attribute those gains to FlashAttention or block-sparse FlashAttention, and it does not report the claimed 0.7 GPT-2 perplexity improvement or 6.4-point classification lift (searched those terms and figures). Page numbers were not available in our copy. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup.”
- Also stated in: § 1 Introduction · plain
“Although these methods reduce the compute requirements to linear or near-linear in sequence length, many of them do not display wall-clock speedup against standard attention and have not gained wide adoption.”
What the sources say
- Partly supportedcites [51] [51] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In The International Conference on Machine Learning (ICML) , 2020. - sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says LSH attention reduces complexity from O(L²) to O(L log L), and the synthetic-task analysis (p. 5) reports accuracy losses in some settings. But the paper does not support the claim that approximate attention methods often fail to achieve wall-clock speedups; its Experiments section (p. 9) reports that LSH attention remains fast as sequence length increases while regular attention slows. (LLM, conf=0.87)
- Partly supportedcites [74] [74] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics , 9: 53-68, 2021. - sourceExperimental — accuracy not yet measured
The Abstract says sparse attention reduces compute complexity, and §6.1 reports that local attention was slightly worse than full attention on CIFAR-10. Section 6.3 supports a qualified wall-clock concern: Routing Transformer was slower than local attention on the reported TPU tests. But local attention was faster than full attention, and some Routing Transformer configurations were faster too, so the paper does not establish that approximate attention methods generally fail to achieve wall-clock speedups. (LLM, conf=0.84)
- Partly supportedcites [12] [12] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations (ICLR) , 2020. - sourceExperimental — accuracy not yet measured
The Introduction and Related Work (p. 1) describes more tractable attention methods and limitations that can affect their representation power or accuracy. Section 2.2 (p. 3) shows reduced time complexity for the paper’s approximate attention mechanism. But the paper does not say that approximate methods often fail to achieve wall-clock speedups; Section 4.1 (p. 6) instead reports lower wall-clock time for its own method. (LLM, conf=0.84)
- Partly supportedcites [50] [50] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning , pages 5156-5165. PMLR, 2020. - sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says sparse attention and locality-sensitive hashing reduce computational complexity but do not speed up autoregressive inference. That supports the speed claim only for that setting: the paper also says these methods can be trained efficiently on long sequences, and it does not establish a general trade-off in model quality. (LLM, conf=0.86)
- Partly supportedcites [84] [84] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020. - sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says sparse attention reduces complexity but incurs a performance drop, while Reformer’s efficiency gains appear only for sequences longer than 2048 tokens. The claim overstates the wall-clock point for sparse attention: the paper reports a limited but real 20% speedup, not no speedup. (LLM, conf=0.85)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“The first major difference is that Rabe and Staats [66] focuses on the reducing the total memory footprint (maximum amount of GPU memory required) while FlashAttention focuses on reducing memory accesses (the number of memory reads/writes).”
What the sources say
- Supportedcites Rabe and Staats [66] source
The Introduction (p. 1) presents algorithms designed to reduce the memory required for attention, and the Conclusion (p. 6) explicitly describes dramatically reducing its memory requirement. The citation is attached to the characterization of Rabe and Staats’s work; the comparison with FlashAttention is the citing authors’ framing. (LLM, conf=0.93)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In prior work, all transformer models have either run out of memory, or only achieved random performance [80].”
What the sources say
- Partly supportedcites [80] [80] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations , 2020. - sourceExperimental — accuracy not yet measured
The Quantitative Results (p. 7) say that all models evaluated on Path-X failed, achieving at best 50% accuracy, consistent with chance performance. The paper does not establish this for all transformer models or for Path-256, and the cited passages do not say that any Path-X model ran out of memory. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“3 LRA accuracy results are known to be highly dependent on the tuning procedure [90].”
What the sources say
- Could not verifycites [90] [90] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nystöm-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence , volume 35, page 14138, 2021. - sourceExperimental — accuracy not yet measured
The Long Range Arena Benchmark section (p. 8) describes the evaluation setup and reports accuracy, but does not say that accuracy is highly dependent on the tuning procedure. Searches for LRA tuning and hyperparameter sensitivity found a sensitivity discussion only for MRPC, a different task (Fine-tuning on Downstream NLP tasks, p. 7). (LLM, conf=0.76)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“As a result, FlashAttention is faster than standard attention (2-4 ) while Rabe and Staats [66] is around the same speed or slightly slower than standard attention.”
What the sources say
- Supportedcites Rabe and Staats [66] source
The citation to Rabe and Staats supports the comparison of their implementation with standard attention, not the separate FlashAttention speed figure. The Abstract (p. 1) says their implementation is within a few percent of standard attention’s runtime, and §5.1 (p. 4) describes its runtime performance as roughly similar in the measured TPU experiment. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, even though structured matrices are efficient in theory, they have not seen wide adoption since it is hard to translate their efficiency to wall-clock speedup since dense unconstrained matrix multiply has very optimize implementation, a phenomenon known as the hardware lottery [41].”
What the sources say
- Partly supportedcites [41] [41] Sara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489 , 2020. - sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FlashAttention instead simplifies the backward pass analytically (Appendices B.2 and B.4). It only recomputes the attention matrix and does not recompute the temporary output of each block. This reduces the memory requirement for the backward pass and yields speedup.”
What the sources say
- Partly supportedcites Rabe and Staats [66] sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Table 2 shows up to 3 endto-end speedup compared to Huggingface and 1.7 speedup compared to Megatron-LM.”
- Also stated in: § 4.1 Faster Models with FlashAttention · plain
“GPT-2 small and medium using FlashAttention achieve up to 3 speed up compared to Huggingface implementation and up to 1.7 compared to Megatron-LM.”
What the sources say
- Could not verifycites [87] [87] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics. URL https://www sourceExperimental — accuracy not yet measured
The Deployment section reports a nearly 4× speedup using ONNX, not FlashAttention. Searches for the claimed 3× and 1.7× end-to-end comparisons found no such results in our copy; Related Work mentions Megatron-LM but does not benchmark against it. (LLM, conf=0.77)
- Supportedcites [77] [77] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 , 2019. - source
The citation identifies Megatron-LM as the comparison baseline, rather than attributing the citing paper’s Table 2 speedup measurements to it. The source describes its own model-parallel implementation in “Model Parallel Transformers” (p. 4) and evaluates its scaling in “Scaling Analysis” (p. 6). (LLM, conf=0.89)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.1 Faster Models with FlashAttention · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FlashAttention instead incrementally updates the output (Algorithm 1 line 12) after processing each block, so only one copy of the output is needed (instead of 𝐾 copies for 𝐾 blocks). This means that FlashAttention has smaller total memory requirement compared to Rabe and Staats [66].”
What the sources say
- Partly supportedcites Rabe and Staats [66] sourceExperimental — accuracy not yet measured
Section 4 (p. 4) says the practical implementation keeps one summary per key/value chunk before rescaling them, producing √n summaries and O(√n) memory overhead. That supports the comparison with storing K intermediate results, but the paper does not establish the claimed smaller total memory requirement for FlashAttention. It also describes an O(log n)-memory self-attention algorithm in the Abstract (p. 1), so the comparison cannot be applied to all of its methods. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“FlashAttention yields the fastest single-node BERT training speed that we know of. We train a BERT-large [22] model with FlashAttention on Wikipedia. Table 1 compares our training time to the implementation from Nvidia that set the training speed record for MLPerf 1.1 [58]. Our implementation is 15% faster.”
What the sources say
- Not checked — no full text of the sourcecites [22] [22] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. 2019. - source
- Could not verifycites [58] [58] Peter Mattson, Christine Cheng, Gregory Diamos, Cody Coleman, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, et al. Mlperf training benchmark. Proceedings of Machine Learning and Systems , 2:336-349, 2020. - sourceExperimental — accuracy not yet measured
The citation is attached to the claim that an NVIDIA implementation set the MLPerf 1.1 training-speed record, not to the citing authors’ 15% comparison. The Results (p. 9) discuss only MLPerf v0.5 and v0.6; searches for MLPerf 1.1, NVIDIA, and BERT found no passage establishing the cited record. (LLM, conf=0.76)
hedgeddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Rabe and Staats [66] suggests that the backward pass can be done without quadratic extra memory by applying gradient checkpointing to the memory-efficient forward pass.”
What the sources say
- Supportedcites Rabe and Staats [66] source
Section 5.2 (p. 5) says the authors apply checkpointing to the function that summarizes attention chunks, allowing intermediate results to be forgotten during the forward pass and recomputed during backpropagation. It also explains that their algorithm never forms the full attention matrix, preserving its memory advantage during differentiation. (LLM, conf=0.94)
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a proof of concept, we implement block-sparse FlashAttention , a sparse attention algorithm that is 2-4 faster than even FlashAttention , scaling up to sequence length of 64k.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Theorem 1. Algorithm 1 returns O softmax QK V with 𝑂 𝑁 2 𝑑 FLOPs and requires 𝑂 𝑁 additional memory beyond inputs and output.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and is optimal for a range of SRAM sizes.”
- Also stated in: § 1 Introduction · plain
“Moreover, we provide a lower bound, showing that no exact attention algorithm can asymptotically improve on the number of HBM accesses over all SRAM sizes.”
- Also stated in: § 3.2 Analysis: IO Complexity of FlashAttention · plain
“Proposition 3. Let 𝑁 be the sequence length, 𝑑 be the head dimension, and 𝑀 be size of SRAM with 𝑑 𝑀 𝑁𝑑 . There does not exist an algorithm to compute exact attention with 𝑜 𝑁 2 𝑑 2 𝑀 1 HBM accesses for all 𝑀 in the range 𝑑 𝑁𝑑 .”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
In the paper
“Even with the increased FLOPs due to recomputation, our algorithm both runs faster (up to 7.6x on GPT-2 [67], Figure 1 right) and uses less memory -linear in sequence length-than standard attention, thanks to the massively reduced amount of HBM access.”
What the sources say
- Not matchedcites [67] [67] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: All gradient checkpointing implementations known to the authors trade speed for memory.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FlashAttention is up to 3 faster than the standard attention implementation across common sequence lengths from 128 to 2K and scales up to 64K.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Up to sequence length of 512, FlashAttention is both faster and more memory-efficient than any existing attention method, whereas for sequence length beyond 1K, some approximate attention methods (e.g., Linformer) start to become faster.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FlashAttention is up to 20 more memory efficient than exact attention baselines, and is more memory-efficient than the approximate attention baselines.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Theorem 5. Let 𝑁 be the sequence length, 𝑑 be the head dimension, and 𝑀 be size of SRAM with 𝑑 𝑀 𝑁𝑑 . Standard attention (Algorithm 0) backward pass requires Θ 𝑁𝑑 𝑁 2 HBM accesses, while FlashAttention backward pass (Algorithm 4) requires Θ 𝑁 2 𝑑 2 𝑀 1 HBM accesses.”
- Also stated in: § C Proofs · plain
“At each step of the standard attention backward pass, one needs to load inputs of size 𝑁𝑑 or 𝑁 2 from HBM, and needs to write the outputs of size 𝑁 2 or 𝑁𝑑 to HBM. This incurs Θ 𝑁𝑑 𝑁 2 HBM accesses.”
- Also stated in: § C Proofs · plain
“Similar to Theorem 2, we see that each element of K and V is loaded from HBM once. Each element of dK and dV is only written to HBM once. We make 𝑇 𝑐 passes over Q O dO , each pass loading all of Q O dO to HBM. We also make 𝑇 𝑐 passes over dQ , each pass reading/writing all of dQ from/to HBM. Therefore the number of HBM accesses is Θ 𝑁𝑑 𝑁𝑑𝑇 𝑐 Θ 𝑁𝑑𝑇 𝑐 .”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § C Proofs · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our IO-aware implementation of attention is optimal within constants for computing attention on a single GPU.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see that similar to the forward pass, the backward pass performs 𝑂 𝑁 2 FLOPs and only requires 𝑂 𝑁 extra memory beyond inputs, output, output gradient, and input gradients.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The runtimes of many approximate/sparse attention mechanisms grow linearly with sequence length, but FlashAttention still runs faster than approximate and sparse attention for short sequences due to fewer memory accesses.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We do not need to store the dropout mask of size 𝑂 𝑁 2 from the forward pass. Instead, we can save the pseudo-random number generator states from the forward pass and re-generate the dropout mask in the backward pass. This allows us to only use 𝑂 𝑁 extra memory.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 4 shows that that GPT-2 with FlashAttention and context length 4K is still 30% faster than GPT-2 from Megatron with context length 1K, while achieving 0.7 better perplexity.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, block-sparse FlashAttention enables the Transformers to scale to sequence length 64K, achieving 63.1 accuracy 4 on Path-256.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All other algorithms except for Linformer run out of memory on an A100 GPU before 64K, and FlashAttention is still 2 more efficient than Linformer.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FlashAttention and block-sparse FlashAttention have the same memory footprint, which grows linearly with sequence length.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the LRA benchmark, block-sparse FlashAttention achieves 2.8 speedup, while performing on par with standard attention (Section 4).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Appendix E includes plots of the validation perplexity throughout training, confirming that FlashAttention is as numerically stable as the baselines and produces the same training / validation curves.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Both of these datasets contain very long text documents; the average number of tokens in MIMIC is 2,395 tokens, and the longest document contains 14,562 tokens, while the average and longest numbers in ECtHR are 2,197 and 49,392, respectively.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We confirm that runtime of block-sparse FlashAttention scales linearly in seq. length and is faster than all existing approximate attention baselines.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We first derive the forward and backward passes of attention and show that they can be computed in a memory-efficient manner (requiring extra memory linear instead of quadratic in the sequence length).”
- Also stated in: § B.1 Memory-efficient forward pass · plain
“Therefore the forward pass can be computed with 𝑂 𝑛 extra memory:”
- Also stated in: § B.2 Memory-efficient backward pass · plain
“Therefore the backward pass can also be computed with 𝑂 𝑛 extra memory:”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § B.1 Memory-efficient forward pass · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We confirm that the memory footprint of FlashAttention scales linearly with seq. length and is up to 3 faster than standard attention for common seq. lengths (up to 2K).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe slightly higher speedups on the RTX 3090 (between 2.5-4.5 ), since the memory bandwidth on an RTX 3090 is lower than on an A100 (roughly 900 GB/s vs. 1.5 TB/s).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As a result, we can support much longer sequences (e.g., up to length 64K). We also support more head dimensions (16, 32, 64, 128) and broader GPU types (all Turing and Ampere GPUs at the time of writing).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We train the model on 8 A100-40GB GPUs, and we measure the wall-clock training time. Training GPT-2 small takes between 2.7-9.5 days, and training GPT-2 medium takes between 6.9-21.0 days (Table 2).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For the block-sparse case, notice that we only need to load blocks corresponding to nonzero blocks. As a result, the number of HBM accesses are scaled by 𝑠 , the fraction of nonzero blocks in the block-sparsity mask. However, for small values of 𝑠 , we would still need to write the result O 𝑁 𝑑 . Therefore the number of HBM accesses is”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Longformer and BigBird do not support sequence lengths longer than 8092.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We use FP16 for all measurements, except for Local Attention, whose implementation only supports FP32.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“After hyperparameter tuning, almost all of the attention methods achieve similar accuracy on all of the five LRA tasks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Block-Sparse (OpenAI) does not support sequence lengths longer than 4096.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to FMHA, the overall runtime of FlashAttention is about 4% slower for sequence length 128, 8% faster for sequence length 256, and 5% faster for sequence length 512.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The Megatron implementation does not support sequence lengths longer than 2048.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Generally FlashAttention is slightly faster than FMHA in the forward pass and slightly slower than FMHA in the backward pass.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In the forward pass, it stores the attention matrix softmax mask QK 𝑇 to HBM to be used in gradient computation. As a result, it does not offer substantial memory saving (though for shorter sequences memory footprint is often not a primary concern).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see that FlashAttention behaves the same as the baseline implementation and the validation perplexity curves of the two implementations almost lie on top of each other.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For Path-X, we take the best performing checkpoint (according to val accuracy), and additionally fine-tune it for 200 epochs with the same warmup and learning rate (this adds roughly 4 points of accuracy to FlashAttention for Path-X, but the model starts overfitting afterwards).”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length.”
- Also stated in: § 1 Introduction · plain
“since the self-attention module at their heart has time and memory complexity quadratic in sequence length.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
None of the sources cited for this claim have been checked yet.
In the paper
“IO-aware algorithms have been critical for similar memory-bound operations, when reading and writing data can account for a large portion of the runtime-such as database joins [71], image processing [70], numerical linear algebra [4], and more [40, 85].”
What the sources say
- Not matchedcites [71] [71] Raghu Ramakrishnan, Johannes Gehrke, and Johannes Gehrke. Database management systems , volume 3. McGraw-Hill New York, 2003. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [70] [70] Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. Acm Sigplan Notices , 48(6):519-530, 2013. - source
- Not matchedcites [4] [4] L Susan Blackford, Antoine Petitet, Roldan Pozo, Karin Remington, R Clint Whaley, James Demmel, Jack Dongarra, Iain Duff, Sven Hammarling, Greg Henry, et al. An updated set of basic linear algebra subprograms (blas). ACM Transactions on Mathematical Software , 28(2):135-151, 2002. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We implement FlashAttention in CUDA to achieve fine-grained control over memory access and fuse all the attention operations into one GPU kernel.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Common Python interfaces to deep learning such as PyTorch and Tensorflow do not allow fine-grained control of memory access.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This requires writing the attention algorithm in a considerably lower-level language than PyTorch, and requires significant engineering effort.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Implementations may also not be transferrable across GPU architectures.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our current approach to building IO-aware implementations of attention requires writing a new CUDA kernel for each new attention implementation.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The discrepancies may be due to subtle distribution shifts: MIMIC-III contains specialized medical text and thus may be more susceptible to a distribution shift in the document length, whereas ECtHR contains general language.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Path-256 requires longer sequences but has relatively shorter paths than Path-X, so it is easier to obtain a higher accuracy.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We describe the FlashAttention algorithm to implement both the forward and the backward passes on GPUs that reduces HBM accesses, leading to both faster runtime and smaller memory footprint.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Attention time and memory scale quadratically in the sequence length.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Each block requires more memory, so we need to use smaller block sizes to fit into SRAM.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, we can repeatedly load the inputs Q K and recompute the block of the attention matrix that we need, significantly reducing HBM access.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods did not successfully run the backward pass with masking due to a bug in external libraries, so we measured them without masking to be generous.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We only measure memory footprint once, since it does not vary between runs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
Common Python interfaces to deep learning such as PyTorch and Tensorflow do not allow fine-grained control of memory access.
“However, common Python interfaces to deep learning such as PyTorch and Tensorflow do not allow fine-grained control of memory access.”
The paper relies on this without support.
All gradient checkpointing implementations known to the authors trade speed for memory.
“While gradient checkpointing has been suggested to reduce the maximum amount of memory required [66], all implementations (that we know off) have to trade speed for memory.”
The paper relies on this without support.
Attention time and memory scale quadratically in the sequence length.
“However, one of their computational bottlenecks is that their time and memory scales quadratic in the sequence length.”
The paper relies on this without support.
RTX 3090 memory bandwidth is roughly 900 GB/s and A100 memory bandwidth is roughly 1.5 TB/s, and the RTX 3090's is lower.
“since the memory bandwidth on an RTX 3090 is lower than on an A100 (roughly 900 GB/s vs. 1.5 TB/s).”
The paper relies on this without support.
T4 SRAM is smaller than A100 SRAM.
“T4 SRAM is smaller than A100, so we need to make the block sizes smaller in FlashAttention .”
The paper relies on this without support.
T4 GPUs are commonly used for inference.
“T4 GPUs are commonly used for inference, so we also report speedup on the forward pass only.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, FlashAttention [166-167] uses CUDA kernels that fuse block-wise matrix multiplication with an online softmax computation, to reduce the quadratic time and memory complexity in the sequence length of vanilla self-attention.”
From A Survey of Large Language Models (2026) · cites this paper as [166]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The additive pre-softmax form is compatible with FlashAttention [2,13,23] .”
From Inference-Time Attention Steering for Vision-Language-Action Driving Models (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent works extend context windows to 100K+ tokens [18], [19], [23].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“unlimited context remains computationally infeasible and doesn't address selective retention or consistency.”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Specifically, in LLMs, each decoding step involves generating a single token, with the transfer of weights between High Bandwidth Memory (HBM) and cache contributing to significant overhead [33, 34].”
From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [33]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“FlashAttention thoroughly researched and implemented various tiling techniques, accelerating selfattention by 2-4 × while cutting memory operations by 10-20 × .”
From ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition (2024) · cites this paper as Dao et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”
From SGLang: Efficient Execution of Structured Language Model Programs (2023) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chunked-prefill schedulers such as Sarathi-Serve [1], memory-efficient attention kernels [8, 14], paged memory management [2], and tensor/pipeline parallelism layouts [10, 31] mitigate but do not eliminate this asymmetry, and they leave the cross-pool transport question largely open.”
From Janus: Joint Prefill/Decode Disaggregation with KV-Cache-Aware Multi-Cloud Routing for Edge-Adjacent LLM Serving (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“FlashAttention [5] exploits the asymmetric GPU memory hierarchy to bring significant memory saving (linear instead of quadratic) and runtime speedup (2-4 × compared to optimized baselines), with no approximation.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By significantly reducing the amount of memory reads/writes, FlashAttention yields 2-4 × wall-clock speedup over optimized baseline attention implementations.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The backward pass also achieves 2-4 × wall-clock speedup due to reduce memory reads/writes.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the backward pass, by re-computing the values of the attention matrices S and P once blocks of inputs Q , K , V are already loaded to SRAM, FlashAttention avoids having to store large intermediate values. By not having to save the large matrices S and P of size 𝑁 × 𝑁 , FlashAttention yields 10-20 × memory saving depending on sequence length”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“FlashAttention [13] applies tiling and kernel optimizations to reduce the peak memory of attention computation and reduce I/O costs.”
From Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“Recent improvements in hardware (e.g., faster GPUs with more memory) and algorithms (Dai et al., 2019;Dao et al., 2022;Poli et al., 2023;Rubin and Berant, 2023, inter alia) have resulted in language models with larger context windows (e.g., 4096, 32K, and even 100K tokens)”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Compactor scoring incurs minimal overhead compared to the cost of a single Flash Attention (Dao et al., 2022) pass for large contexts”
From Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores (2025) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“On modern hardware accelerators (GPUs) most operations (except matrix multiply) are bounded by memorybandwidth (Dao, Fu, Ermon, et al. 2022; Ivanov et al. 2021; Williams, Waterman, and Patterson 2009).”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Dao, Fu, Ermon, et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“A primary focus has been on accelerating kernel execution by minimizing memory movement during attention computation, with FlashAttention being a prominent example [166].”
From A Survey of Large Language Models (2026) · cites this paper as [166]
The cited source says what the claim attributes to it.
In the citing paper
“Partition strategies in ChunkAttention are built on online softmax (Milakov and Gimelshein, 2018) and inspired by FlashAttention (Dao et al., 2022; Dao, 2023), which adopted the same algorithm.”
From ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition (2024) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“FlashAttention [3] shows that attention performance depends heavily on data movement.”
From TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing (2026) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“FlashAttention exploits asymmetric GPU memory hierarchy to achieve linear memory scaling instead of quadratic requirements”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [200]
The cited source says what the claim attributes to it.
In the citing paper
“Despite this growing importance, LLMs consistently face challenges in processing long contexts, primarily due to the substantial computational resources required for long sequence training (Dao et al., 2022; Dao, 2023) as well as the apparent inability to generalize to sequences longer than those encountered during training (Chen et al., 2023; Peng et al., 2023b).”
From ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“While performance monotonically improves, longer prompts require much more compute costs (Dao et al., 2022).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“FlashAttention (Dao et al., 2022) achieves 2-4× speedup over standard attention while reducing memory from O(N 2 ) to O(N ).”
From KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch? (2026) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“As with FlashAttention , Algorithm 1 returns the correct output O = softmax ( QK ⊤ ) V (with no approximation), using 𝑂 ( 𝑁 2 𝑑 ) FLOPs and requires 𝑂 ( 𝑁 ) additional memory beyond inputs and output (to store the logsumexp 𝐿 ).”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“up to 10-20 × memory saving”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Dao et al. [5] proposed to reorder the attention computation and leverages classical techniques (tiling, recomputation) to significantly speed it up and reduce memory usage from quadratic to linear in sequence length.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“FlashAttention applies the classical technique of tiling to reduce memory IOs, by (1) loading blocks of inputs from HBM to SRAM, (2) computing attention with respect to that block, and then (3) updating the output without writing the large intermediate matrices S and P to HBM.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Our version is on par with or better than the original on all cases considered, while covering more use-cases and hardware.”
From DINOv2: Learning Robust Visual Features without Supervision (2023) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“While mLSTM is parallelizable analog to FlashAttention (Dao et al., 2022;Dao, 2024) or GLA (Yang et al., 2023), sLSTM is not parallelizable due to the memory mixing (hidden-hidden connections).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“To avoid text truncation, researchers try to extend the context length of LLMs via training from scratch, fine-tuning, or specific algorithms (e.g., FlashAttention; Dao et al., 2022).”
From Recursively summarizing enables long-term dialogue memory in large language models (2025) · cites this paper as Dao et al., 2022
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“FlashAttention-2 providing approximately twice the speed through reduced non-matrix multiplication operations and optimized work distribution [200, 199].”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [200]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The first version of FlashAttention parallelizes over batch size and number of heads. We use 1 thread block to process one attention head, and there are overall batch size · number of heads thread blocks.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredIn the citing paper
“Secondly, the practicality of fine-tuning poses challenges for many labs. Even minimally fine-tuning foundation models can require extensive GPU resources, given that, for example, scGPT's architecture relies on the use of FlashAttention [16] not available for older and smaller graphics cards 3 .”
From Assessing the limits of zero-shot foundation models in single-cell biology (2023) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ E.6 Full Benchmarking Results
“Longformer and BigBird do not support sequence lengths longer than 8092.”
§ E.6 Full Benchmarking Results
“| Longformer | 1.27 | 1.31 | 1.29 | 2.04 | 5.24 | 10.74 | 25.95 | - | - | - | | BigBird | 1.33 | 1.28 | 1.32 | 1.81 | 5.55 | 11.44 | 27.45 | - | - | - |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Faster Models with FlashAttention
“Table 2 shows up to 3 endto-end speedup compared to Huggingface and 1.7 speedup compared to Megatron-LM.”
§ 4.1 Faster Models with FlashAttention
“| GPT-2 small - FlashAttention | 18.2 | 2.7 days (3.5 ) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.3 Benchmarking Attention
“On the other hand, block-sparse FlashAttention is faster than all implementations of exact, sparse, and approximate attention that we know of, across all sequence lengths.”
§ E.6 Full Benchmarking Results
“| FlashAttention | 0.35 | 0.36 | 0.80 | 2.52 | 9.16 | 36.70 | 146.13 | 583.45 | 2332.01 | 9323.63 | | Block-Sparse FlashAttention | 0.91 | 0.83 | 0.94 | 0.92 | 1.83 | 3.50 | 7.02 | 13.56 | 26.71 | 53.92 |”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 9
- [4][4] L Susan Blackford, Antoine Petitet, Roldan Pozo, Karin Remington, R Clint Whaley, James Demmel, Jack Dongarra, Iain Duff, Sven Hammarling, Greg Henry, et al. An updated set of basic linear algebra subprograms (blas). ACM Transactions on Mathematical Software , 28(2):135-151, 2002. -
- [26][26] Jean Feydy, Joan Glaunès, Benjamin Charlier, and Michael Bronstein. Fast geometric learning with symbolic matrices. Advances in Neural Information Processing Systems , 33, 2020. -
- [40][40] John Hennessy and David Patterson. Memory hierarchy design. Computer Architecture: A Quantitative Approach , pages 390-525, 2003. -
- [44][44] Zhe Jia and Peter Van Sandt. Dissecting the Ampere GPU architecture via microbenchmarking. GPU Technology Conference, 2021. -
- [55][55] Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. -
- [59][59] Frank McSherry, Michael Isard, and Derek G Murray. Scalability! but at what COST ? In 15th Workshop on Hot Topics in Operating Systems (HotOS XV) , 2015. -
- [65][65] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, highperformance deep learning library. Advances in neural information processing systems , 32, 2019. -
- [67][67] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
- [88][88] David P Woodruff. Optimal space lower bounds for all frequency moments. In SODA , volume 4, pages 167-175. Citeseer, 2004. -
Cited by
Sorted by citations
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., & Stoica, I. (2023) · Proceedings of the 29th Symposium on Operating Systems Principles · doi:10.1145/3600006.3613165 · Cited by 1,817
Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023) · arXiv · doi:10.48550/arxiv.2309.06180 · Cited by 1,802
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
“Dao et al. (2022) instead provide a faster exact attention by a carefully-crafted IO-aware CUDA kernel.”
BackgroundMamba: Linear-Time Sequence Modeling with Selective State Spaces
Gu, A., & Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2312.00752 · Cited by 1,579
Self-citationscGPT: Towards Building a Foundation Model for Single-Cell Multi-omics Using Generative AI
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., & Wang, B. (2023) · doi:10.1101/2023.04.30.538439 · Cited by 1,694
“To address this challenge and ensure efficient self-attention mechanisms, we leverage the accelerated self-attention implementation by Flash-Attention [65].”
MethodA Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
“We discuss both in sections II-C and II-D, respectively.”
DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078
“We implemented our own version of FlashAttention (Dao et al., 2022) to improve memory usage and speed on the self-attention layers.”
MethodMultimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention
Liu, X., Peng, H., Zheng, N., Yang, Y., Hu, H., & Yuan, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01386 · Cited by 956
“The activation functions are element-wise operations that occupy a considerable amount of processing time on GPU/CPU [15,48,72], thus utilizing ReLU instead of more complicated activation functions is of better efficiency.”
BackgroundMethodInfluentialPoint Transformer V3: Simpler, Faster, Stronger
Wu, X., Jiang, L., Wang, P., Liu, Z., Liu, X., Qiao, Y., Ouyang, W., He, T., & Zhao, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00463 · Cited by 863
“Our experimental results demonstrate that xCPE fully unleashes the performance with a slight increase in latency of a few milliseconds compared to the standard CPE, the performance gains justify this minor trade-off. tion [18, 19], which can be advantageous for our PTv3.”
BackgroundResultBoltz-1 Democratizing Biomolecular Interaction ModelingPreprint
Wohlwend, J., Corso, G., Passaro, S., Getz, N., Reveiz, M., Leidal, K., Swiderski, W., Atkinson, L., Portnoi, T., Chinn, I., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.11.19.624167 · Cited by 592
“This is an extension of FlashAttention that allows for the inner bias term [Dao et al., 2022].”
Regulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466
“and, in particular, transformers, more efficient [48, 49], the energy required to train models this large has triggered concerns from a climate policy perspective [25, 50-54] (see also Part 6).”
BackgroundGQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023) · arXiv · doi:10.48550/arxiv.2305.13245 · Cited by 447
“Recently, decoder-only models are extremely popular, and since these models do not have separate self-attention and cross-attention, we expect GQA to have a stronger advantage over MQA.”
BackgroundAhdritz, G., Bouatta, N., Floristean, C., Kadyan, S., Xia, Q., Gerecke, W., O’Donnell, T. J., Berenberg, D., Fisk, I., Zanichelli, N., et al. (2024) · Nature Methods · doi:10.1038/s41592-024-02272-z · Cited by 439
“As such, OpenFold can be readily used by the widest community of developers and interfaces with a rich ecosystem of existing machine learning software 26–29 .”
InfluentialGQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebron, F., & Sanghai, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.298 · Cited by 454
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340
Is Mamba effective for time series forecasting?
Wang, Z., Kong, F., Feng, S., Wang, M., Yang, X., Zhao, H., Wang, D., & Zhang, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129178 · Cited by 362
xLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · arXiv · doi:10.48550/arxiv.2405.04517 · Cited by 299
Splitwise: Efficient Generative LLM Inference Using Phase Splitting
Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, Í., Maleki, S., & Bianchini, R. (2024) · 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca59077.2024.00019 · Cited by 298
“LLM inference serving is a rapidly developing field, with several recent works optimizing batching [23], [25], [51], [53], [81], scheduling [22], [42], [51], [66], [73], [79], and memory usage [32], [35], [51], [74].”
MethodJia, D., Yuan, Y., He, H., Wu, X., Yu, H., Lin, W., Sun, L., Zhang, C., & Hu, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01887 · Cited by 333
“, Flash Attention [10] can decrease the memory to linear complexity, and the sequential attention technique in SwinV2 can also greatly reduce the memory consumption for both self/cross attention.”
BackgroundxLSTM: Extended Long Short-Term Memory
Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-3417 · Cited by 286
Efficient Deformable ConvNets: Rethinking Dynamic and Sparse Operator for Vision Applications
Xiong, Y., Li, Z., Chen, Y., Wang, F., Zhu, X., Luo, J., Wang, W., Lü, T., Li, H., Qiao, Y., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00540 · Cited by 310
“Flash-Attention [9], by reducing the number of accesses to High Bandwidth Memory (HBM), achieves a significantly faster speed in practice despite having higher FLOPs compared to vanilla attention.”
Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment
Chen, Q., Chen, X., Wang, J., Zhang, S., Yao, K., Feng, H., Han, J., Ding, E., Zeng, G., & Wang, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00610 · Cited by 304
“With a memoryefficient implementation [12], Group DETR does not bring much memory burden during training, only requires 1.”
BackgroundMethodEPlus-LLM: A large language model-based computing platform for automated building energy modeling
Jiang, G., Ma, Z., Zhang, L., & Chen, J. (2024) · Applied Energy · doi:10.1016/j.apenergy.2024.123431 · Cited by 204
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 242
A Survey of AI-Generated Content (AIGC)
Cao, Y., Li, S., Liu, Y., Yan, Z., Dai, Y., Yu, P. S., & Sun, L. (2024) · ACM Computing Surveys · doi:10.1145/3704262 · Cited by 1,457
A Cookbook of Self-Supervised Learning
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12210 · Cited by 182
“Another approach, is to resort to IO-aware optimizations [Ivanov et al., 2021], the most known one perhaps being FlashAttention [Dao et al., 2022].”
BackgroundMentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models
Yang, K., Zhang, T., Kuang, Z., Xie, Q., Huang, J., & Ananiadou, S. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3648137 · Cited by 275
Jaume, G., Vaidya, A. J., Chen, R. J., Williamson, D. F. K., Liang, P. P., & Mahmood, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01100 · Cited by 184
“…reduce memory and computational requirements [4, 12, 14, 15, 29, 31, 80, 82], for instance, by approximating self-attention with a low-rank decomposition [44, 82], using latent bottleneck distillation [31, 32, 50], by optimizing GPU reads/writes [14, 15] or using sparse attention patterns [4, 53].”
BackgroundYour Diffusion Model is Secretly a Zero-Shot Classifier
Li, A. C., Prabhudesai, M., Duggal, S., Brown, E. V., & Pathak, D. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00210 · Cited by 242
Legalbench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D., et al. (2023) · doi:10.2139/ssrn.4583531 · Cited by 154
Self-citationGenerative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145
“Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Dao, T. (2023) · arXiv · doi:10.48550/arxiv.2307.08691 · Cited by 143
“In the case of long sequences (which usually means small batch sizes or small number of heads), to make better use of the multiprocessors on the GPU, we now additionally parallelize over the sequence length dimension.”
BackgroundMethodInfluentialSelf-citationSparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detection
Xie, Y., Xu, C., Rakotosaona, M., Rim, P., Tombari, F., Keutzer, K., Tomizuka, M., & Zhan, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01613 · Cited by 167
“† : CMT [49] adopts flash-attention [7] for transformer acceleration.”
BackgroundGunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.11644 · Cited by 138
FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformer
Liu, Z., Yang, X., Tang, H., Shang, Y., & Han, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00122 · Cited by 135
When Embodied AI Meets Industry 5.0: Human-Centered Smart Manufacturing
Xu, J., Sun, Q., Han, Q., & Tang, Y. S. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2025.125327 · Cited by 141
GFNet: Global Filter Networks for Visual Recognition
Rao, Y., Zhao, W., Zhu, Z. D., Zhou, J., & Lu, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3263824 · Cited by 571
“There are also many promising methods to improve the efficiency of Transformers by modifying the self-attention operation for lower time complexity and memory access like Linformer [68], Performer [11], FlashAttention [14] and SOFT [49].”
MethodInfluentialToken Merging for Fast Stable Diffusion
Bolya, D., & Hoffman, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00484 · Cited by 115
“Flash Attention [2] computes attention efficiently by cleverly accounting for memory bandwidth.”
BackgroundDynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
Stojkovic, J., Zhang, C., Goiri, Í., Torrellas, J., & Choukse, E. (2025) · 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA) · doi:10.1109/hpca61900.2025.00102 · Cited by 112
Deep learning for cross-domain data fusion in urban computing: Taxonomy, advances, and outlook
Zou, X., Yan, Y., Hao, X., Hu, Y., Wen, H., Liu, E., Zhang, J., Li, Y., Li, T., Zheng, Y., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102606 · Cited by 127
Efficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 156
Wang, F., Zhang, Z., Zhang, X., Wu, Z., Mo, T., Lu, Q., Wang, W., Li, R., Xu, J., Tang, X., et al. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3768165 · Cited by 106
Evolution and Prospects of Foundation Models: From Large Language Models to Large Multimodal Models
Chen, Z., Xu, L., Zheng, H., Chen, L., Tolba, A. K. R., Zhao, L., Yu, K., & Feng, H. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.052618 · Cited by 126
Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.11511 · Cited by 101
“FlashAttention (Dao et al., 2022) is used to make the long-context training more efficient.”
MethodDeepSeek LLM: Scaling Open-Source Language Models with Longtermism
DeepSeek-AI, :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2401.02954 · Cited by 99
“We also leverage the flash attention (Dao, 2023; Dao et al., 2022) technique to improve hardware utilization.”
MethodLarge Language Models for Next Point-of-Interest Recommendation
Li, P., de Rijke, M., Xue, H., Ao, S., Song, Y., & Salim, F. D. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657840 · Cited by 122
Reducing hallucination in structured outputs via Retrieval-Augmented Generation
Ayala, O., & Bechard, P. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track) · doi:10.18653/v1/2024.naacl-industry.19 · Cited by 97
“All models were trained with flash-attention (Dao et al., 2022) on a single A100 80GB GPU.”
MethodEvaluating Very Long-Term Conversational Memory of LLM Agents
Maharana, A., Lee, D.-H., Tulyakov, S., Bansal, M., Barbieri, F., & Fang, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.747 · Cited by 96
Transformers and genome language models
Consens, M. E., Dufault, C., Wainberg, M., Forster, D., Karimzadeh, M., Goodarzi, H., Theis, F. J., Moses, A. M., & Wang, B. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01007-9 · Cited by 170
“Transformer-based gLMs that do not use downsampling techniques to reduce dimensionality before calculating self-attention, or do not apply a more efficiently implemented attention method 97 , will be limited by their context window.”
Large Language Models for Telecom: Forthcoming Impact on the Industry
Maatouk, A., Piovesan, N., Ayed, F., Domenico, A. D., & Debbah, M. A. (2024) · IEEE Communications Magazine · doi:10.1109/mcom.001.2300473 · Cited by 97
Understanding the Potential of FPGA-based Spatial Acceleration for Large Language Model Inference
Chen, H., Zhang, J., Du, Y., Xiang, S., Yue, Z., Zhang, N., Cai, Y., & Zhang, Z. (2024) · ACM Transactions on Reconfigurable Technology and Systems · doi:10.1145/3656177 · Cited by 98
“The rapid evolution of LLM architectures [5, 54, 70] contrasts with the comparatively slow pace of hardware development.”
On Efficient Training of Large-Scale Deep Learning Models
Shen, L., Sun, Y., Yu, Z., Ding, L., Tian, X., & Tao, D. (2024) · ACM Computing Surveys · doi:10.1145/3700439 · Cited by 156
BOLT: Privacy-Preserving, Accurate and Efficient Inference for Transformers
Pang, Q., Zhu, J., Möllering, H., Zheng, W., & Schneider, T. H. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00130 · Cited by 102
“Algorithms from Flash Attention [15] are also compatible with BOLT but do not significantly speed up operations since they optimize GPU memory access during attention computation, whereas BOLT runs on the CPU and its bottlenecks are communication and HE computations rather than memory access.”
MethodLarge models for intelligent transportation systems and autonomous vehicles: A survey
Gan, L., Chu, W., Li, G., Tang, X., & Li, K. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102786 · Cited by 81
A Survey on Efficient Training of Transformers
Zhuang, B., Liu, J., Pan, Z., He, H., Weng, Y., & Shen, C. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/764 · Cited by 78
Chen, K., Zhou, Y., Ding, M., Wang, Y., Ren, Z., & Yang, Y. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae163 · Cited by 84
“24.0) and Flash attention library [29].”
InfluentialSnapKV: LLM Knows What You are Looking for Before Generation
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., & Chen, D. (2024) · arXiv · doi:10.48550/arxiv.2404.14469 · Cited by 74
“Specifically, in LLMs, each decoding step involves generating a single token, with the transfer of weights between High Bandwidth Memory (HBM) and cache contributing to significant overhead [33, 34].”
MethodEvans, Z., Parker, J. D., Carr, C., Zukowski, Z., Taylor, J., & Pons, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888461 · Cited by 236
Token Fusion: Bridging the Gap between Token Pruning and Token Merging
Kim, M., Gao, S., Hsu, Y.-C., Shen, Y., & Jin, H. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00141 · Cited by 103
“Furthermore, our technique can be seamlessly integrated with works like [7] that aim to expedite the computation speed of the attention module.”
MethodDPNet: Dual-Path Network for Real-Time Object Detection With Lightweight Attention
Zhou, Q., Shi, H., Xiang, W., Kang, B., & Latecki, L. J. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3376563 · Cited by 75
“Besides taking running speed into account, FlashAttention [42] also considers the memory constraint in designing efficient Transformers.”
EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices
Yi, R., Guo, L., Wei, S., Zhou, A., Wang, S., & Xu, M. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3546466 · Cited by 194
Comprehensive review of Transformer‐based models in neuroscience, neurology, and psychiatry
Cong, S., Wang, H., Zhou, Y., Wang, Z., Yao, X., & Yang, C. (2024) · Brain‐X · doi:10.1002/brx2.57 · Cited by 75
Understanding LLMs: A comprehensive overview from training to inference
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129190 · Cited by 81
Token Merging: Your ViT But Faster
Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., & Hoffman, J. (2022) · arXiv · doi:10.48550/arxiv.2210.09461 · Cited by 140
“Some focus on faster attention (Choromanski et al., 2020; Shen et al., 2021; Dao et al., 2022; Wang et al., 2020; Bolya et al., 2022), some attempt to prune heads or features (Meng et al.”
BackgroundHan, S., Wang, M., Zhang, J., Li, D., & Duan, J. (2024) · Electronics · doi:10.3390/electronics13245040 · Cited by 81
Assessing the limits of zero-shot foundation models in single-cell biology
Kedzierska, K. Z., Crawford, L., Amini, A. P., & Lu, A. X. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.16.561085 · Cited by 61
“Even minimally fine-tuning foundation models can require extensive GPU resources, given that, for example, scGPT’s architecture relies on the use of FlashAttention [16] not available for older and smaller graphics cards 3 .”
MethodLamaakal, I., Maleh, Y., El Makkaoui, K., Ouahbi, I., Pławiak, P., Alfarraj, O., Almousa, M., & Abd El-Latif, A. A. (2025) · Sensors · doi:10.3390/s25051318 · Cited by 60
“Additionally, Flash Attention [79,80] accelerates the computation of attention by minimizing memory usage, which is particularly important when working with longer input sequences.”
Progress and opportunities of foundation models in bioinformatics
Li, Q., Hu, Z., Wang, Y., Li, L., Fan, Y., King, I., Jia, G., Wang, S., Song, L., & Li, Y. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae548 · Cited by 72
“These include leveraging attention mechanisms such as FlashAttention [129] and Multi-Query Attention [130], kernels [131], sparse activation [65], and other advanced mechanisms to reduce both the model’s training time and detection time.”
MethodLLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference
Zhang, H., Ning, A., Prabhakar, R. B., & Wentzlaff, D. (2024) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca59077.2024.00082 · Cited by 105
“Figure 13 indicates our proposed latency-oriented design and throughput-oriented design are around the sweet point.”
Evaluating the representational power of pre-trained DNA language models for regulatory genomics
Tang, Z., Somia, N., Yu, Y., & Koo, P. K. (2025) · Genome biology · doi:10.1186/s13059-025-03674-8 · Cited by 71
Adapting Language Models to Compress Contexts
Chevalier, A., Wettig, A., Ajith, A., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.232 · Cited by 59
Large language models in plant biology
Lam, H. Y. I., Ong, X. E., & Mutwil, M. (2024) · Trends in Plant Science · doi:10.1016/j.tplants.2024.04.013 · Cited by 70
UNI-RNA: UNIVERSAL PRE-TRAINED MODELS REVOLUTIONIZE RNA RESEARCHPreprint
Wang, X., Gu, R., Chen, Z., Li, Y., Ji, X.-H., Ke, G., & Wen, H. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.07.11.548588 · Cited by 65
“Implementation Details In the pursuit of enhancing the efficacy of our proposed model, we have employed the IO-aware Flash Attention mechanism[73] in place of the traditional MultiHeadAttention.”
MethodSpecies-aware DNA language models capture regulatory elements and their evolution
Karollus, A., Hingerl, J., Gankin, D., Grosshauser, M., Klemon, K., & Gagneur, J. (2024) · Genome biology · doi:10.1186/s13059-024-03221-x · Cited by 66
Chen, B., Cheng, X., Li, P., Geng, Y., Gong, J., Li, S., Bei, Z., Tan, X., Wang, B., Zeng, X., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02636-z · Cited by 216
“To mitigate this, more advanced efficient compression technologies in terms of parameters or memory, such as quantization 32 , kernel fusion 28 and other accelerate technologies 33–35 , could be applied.”
ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputs
Zhai, Y., Jiang, C., Wang, L., Jia, X., Zhang, S., Chen, Z., Liu, X., & Zhu, Y. (2023) · IEEE International Parallel and Distributed Processing Symposium (IPDPS) · doi:10.1109/ipdps54959.2023.00042 · Cited by 68
“This is because FlashAttention maps a whole attention unit to a threadblock, which, although allows for the complete preservation of the intermediate matrix of an attention unit within shared-memory for any sequence length, results in performance degradation when there are insufficient tasks assigned.”
MethodInfluentialSpeed Is All You Need: On-Device Acceleration of Large Diffusion Models via GPU-Aware Optimizations
Chen, Y., Sarokin, R., Lee, J., Tang, J., Chang, C.-L., Kulik, A., & Grundmann, M. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00490 · Cited by 66
“Although FlashAttention seeks to enhance latency and reduce global memory read/write, its kernel is highly register-intensive.”
BackgroundMethodInfluentialEvaluation of LLM-based chatbots for OSINT-based Cyber Threat Awareness
Shafee, S., Bessani, A., & Ferreira, P. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125509 · Cited by 101
“In Vicuna, gradient checkpointing [56] and flash attention [57] alleviate the memory demand.”
Recursively summarizing enables long-term dialogue memory in large language models
Wang, Q., Fu, Y., Cao, Y., Wang, S., Tian, Z., & Ding, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130193 · Cited by 55
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2310.06770 · Cited by 72
“We used DeepSpeed Ulysses (Jacobs et al., 2023) and Flash Attention (Dao et al., 2022) to enable long context training.”
BackgroundChen, Z., Chen, H., Imani, M., Chen, R., & Imani, F. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125769 · Cited by 61
BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems
Wang, Y., Chen, Y., Li, Z., Kang, X., Fang, Y., Zhou, Y., Zheng, Y., Tang, Z., He, X. J., Guo, R., et al. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737413 · Cited by 172
Chain-of-Thought in Neural Code Generation: From and for Lightweight Language Models
Yang, G., Zhou, Y., Chen, X., Zhang, X., Zhuo, T. Y., & Chen, T. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3440503 · Cited by 75
Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology
Kraus, O. Z., Kenyon-Dean, K., Saberian, S., Fallah, M., McLean, P., Leung, J., Sharma, V., Khan, A., Balakrishnan, J., Çelik, S., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01117 · Cited by 61
“All MAE-ViTs were trained with stochastic depth [3], LayerScale [3], flash attention [18], parallel scaling blocks [19], QK-normalization [19], and no QK-bias [19].”
MethodTelecomGPT: A Framework to Build Telecom-Specific Large Language Models
Zou, H., Zhao, Q., Tian, Y., Bariah, L., Bader, F., Lestable, T., & Debbah, M. A. (2025) · IEEE Transactions on Machine Learning in Communications and Networking · doi:10.1109/tmlcn.2025.3593184 · Cited by 109
Scalable Swin Transformer network for brain tumor segmentation from incomplete MRI modalities
Zhang, D., Wang, C., Chen, T., Chen, W., & Shen, Y. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102788 · Cited by 50
Zhu, H., Gu, J., Wang, H., Jiang, Z., Zhang, Z., Tang, R., Feng, C., Han, S., Chen, R. T., & Pan, D. Z. (2024) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca57654.2024.00059 · Cited by 53
“L system.”
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., & Jia, J. (2023) · arXiv · doi:10.48550/arxiv.2309.12307 · Cited by 50
Wang, Z., Yi, J., Chen, A., Chen, L., Hui, L., & Xu, K. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.01.017 · Cited by 48
Wang, R., Gao, Z., Zhang, L., Yue, S., & Gao, Z. (2025) · Computer Science Review · doi:10.1016/j.cosrev.2025.100755 · Cited by 61
Large language model-based evolutionary optimizer: Reasoning with elitism
Brahmachary, S., Joshi, S. M., Panda, A., Koneripalli, K., Sagotra, A. K., Patel, H., Sharma, A., Jagtap, A. D., & Kalyanaraman, K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129272 · Cited by 66
Cross-Modal Learning for Anomaly Detection in Complex Industrial Process: Methodology and Benchmark
Wu, G., Zhang, Y., Deng, L., Zhang, J., & Chai, T. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3491865 · Cited by 54
Nie, T., Sun, J., & Ma, W. (2025) · Artificial Intelligence for Transportation · doi:10.1016/j.ait.2025.100003 · Cited by 70
“…x i based on the preceding tokens x i in this sequence: • Autoencoding pretraining: (e.g., BERT (Devlin et al., 2019) and BART (Lewis et al., 2019)) masks or corrupts random tokens with replaced spans or masks and trains models to reconstruct them, enhancing bidirectional context understanding.”
CAT-LM Training Language Models on Aligned Code And Tests
Rao, N., Jain, K., Alon, U., Goues, C. L., & Hellendoorn, V. J. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00193 · Cited by 70
“We use the GPT-NeoX toolkit [20] enhanced with Flash Attention [21] to pretrain CAT-LM using an auto-regressive (standard left-to-right) pre-training objective that captures the mapping between code and test files, while learning general code and test structure.”
Metadata
- Authors
- Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 16344-16359
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2205.14135
- OpenAlex
- W7133227460
- Cited by
- 1,214
- References
- 0
- Citations by year
- 2026: 555 · 2025: 316 · 2024: 256 · 2023: 86 · 2022: 1
- FWCI
- 132.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763315, 1563078, 1937301)
- U.S. Department of Defense (N000141712266)
- Accenture
- Nvidia
- VMware
- National Institutes of Health (N000141712266, CCF1563078, CCF1763315)
- National Defense Science and Engineering Graduate
- Office of Naval Research (N000141712266, ONR-N00014, N00014-20, N00014)
- National Institute of Biomedical Imaging and Bioengineering (U54EB020405)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- FlashAttention, memory efficiency, transformer attention, large language models, GPU memory optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
