The DOI matches its DataCite record; 88 of the 88 references it cites were found in a registry or an index; 40 of the 48 sources its claims cite were checked for support.
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Open accessAbstract
Machine learning's explosive growth has produced a proliferation of accelerator architectures, yet no framework maps a workload's computational character to the hardware that serves it best. We build that framework: decomposing ML workloads — transformers, CNNs, GNNs, diffusion and state-space models — into core compute primitives characterized by arithmetic intensity, memory-access pattern, and parallelism, and evaluating how GPUs, TPUs, systolic arrays, dataflow processors, wafer-scale engines, and neuromorphic chips serve them. Using hierarchical roofline and utilization modelling, we show the binding constraint has shifted from peak FLOPS to memory bandwidth and data-movement energy — which quantization only sharpens — and consolidate eight persistent, unsolved problems. These motivate the central contribution: the Fused Memory-Compute Tile (FMCT, codename Habanero), an SRAM-first, HBM-free accelerator combining a three-mode (GEMM / bandwidth / fused) phase-adaptive tile switched by compiler directive, a non-linear unit fused into the systolic datapath, compute-in-memory as a selectable mode rather than the whole architecture, and an inline KV-cache compression engine. A validated component-level energy model shows ~4-7× batch-1 efficiency gains over HBM GPUs. We position FMCT against its closest SRAM-first, no-HBM analogs — d-Matrix Corsair and Tenstorrent — and OpenAI's HBM-retaining Jalapeño ASIC, closing with principles for memory-centric, workload-adaptive accelerators.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
FlashAttention is an IO-aware exact attention algorithm that uses tiling and recomputation to avoid materializing the large \(N \times N\) attention matrix in GPU HBM, reducing memory reads/writes while computing exact attention. It speeds up Transformer training and enables longer contexts, including first better-than-chance Transformer results on Path-X and Path-256.
Key findings
- FlashAttention requires fewer HBM accesses than standard attention: \(\Theta(N^2 d^2 M^{-1})\) vs. \(\Theta(Nd + N^2)\), and is up to 9× fewer HBM accesses for typical \(d\) and \(M\).
- It trains models faster: 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) over the MLPerf 1.1 record, up to 3× speedup on GPT-2 (seq. length 1K), and 2.4× on Long-Range Arena (seq. length 1K–4K).
- Longer context improves quality: 0.7 better perplexity on GPT-2 and 6.4 points of lift on long-document classification; FlashAttention achieves 61.4% accuracy on Path-X (16K), and block-sparse FlashAttention achieves 63.1% on Path-256 (64K).
- Attention benchmarking: FlashAttention is up to 3× faster than standard attention for common sequence lengths and up to 20× more memory efficient than exact attention baselines; block-sparse FlashAttention is faster than all existing approximate attention methods tested.
Methods
- Algorithm design: exact attention computed by tiling Q, K, V blocks into on-chip SRAM, incrementally performing softmax reduction, and recomputing attention in the backward pass from stored softmax statistics instead of storing the \(N \times N\) attention matrix.
- Analysis: proves IO complexity for forward and backward passes, plus a lower bound showing no exact attention algorithm can asymptotically improve HBM accesses over all SRAM sizes; extends to block-sparse FlashAttention with IO complexity scaled by sparsity.
- Experiments: CUDA implementation benchmarked on A100, RTX 3090, and T4 GPUs; trained BERT-large, GPT-2, Long-Range Arena, MIMIC-III, ECtHR, Path-X, and Path-256, comparing runtime, memory, and accuracy against exact, approximate, and sparse attention baselines.
Limitations
- The paper states that building IO-aware attention implementations currently requires writing a new CUDA kernel for each new attention implementation, using a lower-level language than PyTorch, with significant engineering effort, and implementations may not transfer across GPU architectures.
- The paper states its IO-aware implementation is optimal within constants for single-GPU attention, but multi-GPU attention adds another memory hierarchy and is left as future work.
- The paper states the lower bound is proved over a subrange of SRAM sizes \(M\), and parameterized complexity lower bounds in terms of \(M\) are left as future work; the text also makes evident that some approximate attention methods become faster than FlashAttention beyond sequence length 1K, though block-sparse FlashAttention remains faster than all tested approximate methods.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
38 Own result · 35 No basis given · 17 Own interpretation · 14 Partly supported · 7 Supported · 3 Couldn’t confirm · 3 Own method · 1 Mixed support · 1 Sources not found · 1 Not checked yet · accuracy not yet measured
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“We analyze the IO complexity of FlashAttention , showing that it requires fewer HBM accesses than standard attention, and is optimal for a range of SRAM sizes.”
- Also stated in: § 1 Introduction · boosted
“We analyze the IO complexity [1] of FlashAttention , proving that it requires 𝑂 𝑁 2 𝑑 2 𝑀 1 HBM accesses where 𝑑 is the head dimension and 𝑀 is the size of SRAM, as compared to Ω 𝑁𝑑 𝑁 2 of standard attention.”
What the sources say
- Could not verifycites [1] [1] Alok Aggarwal and S Vitter, Jeffrey. The input/output complexity of sorting and related problems. Communications of the ACM , 31(9):1116-1127, 1988. - sourceExperimental — accuracy not yet measured
The Introduction and Section 3 describe I/O bounds for sorting-related problems and transfers between disk and internal memory. A search of our copy for FlashAttention, HBM, SRAM, and attention found no passage comparing FlashAttention with standard attention, so it cannot verify the claimed reduction in HBM accesses. (LLM, conf=0.78)
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · boosted
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FlashAttention trains Transformers faster than existing baselines: 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) compared to the MLPerf 1.1 training speed record, 3 speedup on GPT-2 (seq. length 1K), and 2.4 speedup on long-range arena (seq. length 1K-4K).”
- Also stated in: § 1 Introduction · plain
“We train BERT-large (seq. length 512) 15% faster than the training speed record in MLPerf 1.1 [58]”
What the sources say
- Partly supportedcites [58] [58] Peter Mattson, Christine Cheng, Gregory Diamos, Cody Coleman, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, et al. Mlperf training benchmark. Proceedings of Machine Learning and Systems , 2:336-349, 2020. - sourceExperimental — accuracy not yet measured
Section 3.2 (p. 6) describes MLPerf’s end-to-end time-to-train metric, supporting the identification of MLPerf as a training benchmark. But Section 5 (p. 9) reports only rounds v0.5 and v0.6; our copy does not establish a v1.1 BERT-large speed record or a 15% FlashAttention speedup against it. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“3 speedup on GPT-2 (seq. length 1K)”
- Also stated in: § 1 Introduction · plain
“GPT2 (seq. length 1K) 3 faster than baseline implementations from HuggingFace [87] and Megatron-LM [77]”
What the sources say
- Could not verifycites [87] [87] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics. URL https://www sourceExperimental — accuracy not yet measured
The Deployment section reports speedups from ONNX optimization of models including GPT-2, but does not discuss FlashAttention. Searches for FlashAttention and a 3x speedup at sequence length 1K found no passage addressing the claimed result; the ONNX result does not verify or contradict it. Page numbers were unavailable in our copy. (LLM, conf=0.76)
- Partly supportedcites [77] [77] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 , 2019. - sourceExperimental — accuracy not yet measured
Appendix D.2 (p. 14) reports a 2.98-fold training speedup for a 1.2-billion-parameter model using eight GPUs rather than one. That result concerns model parallelism, not FlashAttention; the passage does not specify a 1K sequence length or the claimed baseline comparison. (LLM, conf=0.83)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“We analyze the IO complexity [1] of FlashAttention , proving that it requires 𝑂 𝑁 2 𝑑 2 𝑀 1 HBM accesses where 𝑑 is the head dimension and 𝑀 is the size of SRAM, as compared to Ω 𝑁𝑑 𝑁 2 of standard attention.”
- Also stated in: § 3.2 Analysis: IO Complexity of FlashAttention · plain
“Theorem 2. Let 𝑁 be the sequence length, 𝑑 be the head dimension, and 𝑀 be size of SRAM with 𝑑 𝑀 𝑁𝑑 . Standard attention (Algorithm 0) requires Θ 𝑁𝑑 𝑁 2 HBM accesses, while FlashAttention (Algorithm 1) requires Θ 𝑁 2 𝑑 2 𝑀 1 HBM accesses.”
What the sources say
- Supportedcites [1] [1] Alok Aggarwal and S Vitter, Jeffrey. The input/output complexity of sorting and related problems. Communications of the ACM , 31(9):1116-1127, 1988. - source
The citation attaches to “IO complexity,” not to the FlashAttention bounds that the citing authors say they prove. The source’s Introduction defines memory and block-transfer parameters, and Section 3 describes its I/O model, supporting the citation as an identification of that framework. (LLM, conf=0.89)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.2 Analysis: IO Complexity of FlashAttention · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the first Transformers to achieve better-than-chance performance on the Path-X challenge (seq. length 16K, 61.4% accuracy)”
- Also stated in: § 1 Introduction · plain
“FlashAttention enables the first Transformer that can achieve better-than-chance performance on the Path-X [80] challenge, solely from using a longer sequence length (16K).”
What the sources say
- Partly supportedcites [80] [80] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations , 2020. - sourceExperimental — accuracy not yet measured
Section 2.2.6 (p. 4) identifies Path-X as a 16K-input benchmark, and Section 3.3 (p. 7) reports that the models tested achieved at best 50%. The paper does not report FlashAttention achieving 61.4% or establish that it was the first Transformer to exceed chance on Path-X. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We argue that a missing principle is making attention algorithms IOaware -accounting for reads and writes between levels of GPU memory.”
- Also stated in: § 1 Introduction · plain
“In this paper, we argue that a missing principle is making attention algorithms IO-aware [1]-that is, carefully accounting for reads and writes to different levels of fast and slow memory (e.g., between fast GPU on-chip SRAM and relatively slow GPU high bandwidth memory, or HBM [45], Figure 1 left).”
What the sources say
- Partly supportedcites [1] [1] Alok Aggarwal and S Vitter, Jeffrey. The input/output complexity of sorting and related problems. Communications of the ACM , 31(9):1116-1127, 1988. - sourceExperimental — accuracy not yet measured
The Introduction describes studying algorithms by the number of I/Os, and Section 3 models transfers between disk and internal memory. That supports the general principle of accounting for data movement, but the paper does not address attention algorithms or transfers between levels of GPU memory. (LLM, conf=0.85)
- Not checked — no full text of the sourcecites [45] [45] Zhe Jia, Marco Maggioni, Benjamin Staiger, and Daniele P Scarpazza. Dissecting the nvidia Volta GPU architecture via microbenchmarking. arXiv preprint arXiv:1804.06826 , 2018. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“Our reproduced baselines perform better than as reported in the original comparison [80].”
What the sources say
- Supportedcites [80] [80] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations , 2020. - source
The cited paper is the original comparison: its benchmark table in §3.2 (p. 6) reports baseline results across the Long Range Arena tasks. The citing authors’ statement about their own reproduced baselines is their comparison with those reported results, not a finding attributed to this source. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This technique [60] has been used in the literature [51, 66] to show that attention computation does not need quadratic extra memory (though the number of HBM accesses is still quadratic, resulting in slow run-time).”
- Also stated in: § B Algorithm Details · plain
“Though they reduce the amount of extra memory required, naively they still incur quadratic HBM accesses, resulting in slower execution speed.”
What the sources say
- Could not verifycites [60] [60] Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867 , 2018. - sourceExperimental — accuracy not yet measured
Section 5 (Benchmarking) links memory accesses to Softmax performance, but the paper does not address memory-efficient attention or whether its HBM accesses are quadratic. I searched our copy for attention, quadratic accesses, and the claimed run-time consequence; the passages concern Softmax and Softmax+TopK instead. (LLM, conf=0.78)
- Partly supportedcites [51] [51] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In The International Conference on Machine Learning (ICML) , 2020. - sourceExperimental — accuracy not yet measured
The memory-efficient attention discussion (Section 2, p. 2) says attention can use memory proportional to sequence length but may be less efficient. Table 1 (Section 2, pp. 4–5) gives memory-efficient attention quadratic time complexity. The paper does not identify quadratic HBM accesses as the reason for slower execution. (LLM, conf=0.87)
- Partly supportedcites [66] [66] Markus N Rabe and Charles Staats. Self-attention does not need 𝑂 𝑛 2 memory. arXiv preprint arXiv:2112.05682 , 2021. - sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says the memory-efficient algorithm avoids quadratic memory use but still has quadratic time complexity. The Inference section (p. 4) reports roughly similar runtime to standard attention; it does not establish quadratic HBM accesses or attribute slow runtime to them. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § B Algorithm Details · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The butterfly pattern we use for our block-sparse attention is motivated by the fact that butterfly matrices [15, 64] and their products have been shown to be able to express any structured matrices with almost optimal runtime and number of parameters [16, 20].”
What the sources say
- Partly supportedcites [15] [15] Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, and Christopher Ré. Learning fast algorithms for linear transforms using butterfly factorizations. In International Conference on Machine Learning (ICML) , 2019. - sourceExperimental — accuracy not yet measured
Section 3.3 (pp. 6–7) says the butterfly-product hierarchy represents several important transforms at low depth and can represent all square matrices at linear depth. The Introduction (p. 1) gives O(N) parameters and O(N log N) multiplication for a class of structured matrices. These passages do not establish that *any* structured matrix has a butterfly representation with almost-optimal runtime and parameter count; §4.1 (p. 7) says some structured matrices are not perfectly captured by the BPBP class. (LLM, conf=0.86)
- Not matchedcites [64] [64] D Stott Parker. Random butterfly transformations with applications in computational linear algebra. 1995. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [16] [16] Tri Dao, Nimit Sohoni, Albert Gu, Matthew Eichhorn, Amit Blonder, Megan Leszczynski, Atri Rudra, and Christopher Ré. Kaleidoscope: An efficient, learnable representation for all structured linear maps. In International Conference on Learning Representations (ICLR) , 2020. - source
Section 2 (p. 3) states that kaleidoscope matrices, composed of products of butterfly matrices, represent all structured matrices with almost optimal parameter count and runtime. The Introduction (p. 2) specifies that this means structured linear maps with fast arithmetic algorithms, with complexity nearly tight up to logarithmic factors. (LLM, conf=0.94)
- Not matchedcites [20] [20] Christopher De Sa, Albert Gu, Rohan Puttagunta, Christopher Ré, and Atri Rudra. A two-pronged progress in structured dense matrix vector multiplication. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1060-1079. SIAM, 2018. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Training Transformers with longer sequences with FlashAttention improves performance on the MIMIC-III [47] and ECtHR [6, 7] datasets.”
What the sources say
- Supportedcites [47] [47] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016. - source
The Methods section identifies MIMIC-III as the third iteration of the MIMIC critical care database. Citation [47] is attached to the name of that dataset, not to the claim that longer Transformer sequences improve performance. (LLM, conf=0.93)
- Supportedcites [6] [6] Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. Neural legal judgment prediction in English. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 4317-4323, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1424. URL https://www.aclweb.org/anthology/P19-1424 . - source
The citation is attached to the ECtHR dataset, and this paper introduces a dataset of European Court of Human Rights cases. The Introduction and §2 (“ECHR Dataset”; page not specified in our copy) identify it as a dataset of approximately 11,500 cases. (LLM, conf=0.91)
- Supportedcites [7] [7] Ilias Chalkidis, Manos Fergadiotis, Dimitrios Tsarapatsanis, Nikolaos Aletras, Ion Androutsopoulos, and Prodromos Malakasiotis. Paragraph-level rationale extraction through regularization: A case study on european court of human rights cases. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics , Mexico City, Mexico, 2021. Association for Computational Linguistics. - source
The citation is attached to “ECtHR datasets,” identifying the dataset rather than attributing the FlashAttention performance finding to this source. The Introduction describes the paper’s dataset as comprising 11,000 ECtHR cases. No page number is available in our copy. (LLM, conf=0.90)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
- It may state two separate claims as one. 48
- Probably not a substantial claim. 19
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“For large sequence lengths 𝑁 , 𝑠 is often set to 𝑁 1 2 [11] or 𝑁 1 log 𝑁 [3, 17, 92], resulting in Θ 𝑁 𝑁 or Θ 𝑁 log 𝑁 IO complexity.”
What the sources say
- Disputedcites [11] [11] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019. - sourceExperimental — accuracy not yet measured
The citation to Child et al. is attached to the logarithmic-complexity option. Instead, Section 4.3 (p. 4) chooses a stride close to √N, and Section 4.2 (p. 4) gives O(N√N) computation. The paper does not give the attributed Θ(N log N) IO complexity. (LLM, conf=0.85)
Checked by a reasoning model — may be wrong.
- Could not verifycites [3] [3] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020. - sourceExperimental — accuracy not yet measured
Section 3.1 (p. 3) gives sliding-window attention a computation complexity of O(n × w) for a fixed window size. Our copy does not say that sparsity s is set to N⁻¹ log N or give a Θ(N log N) IO complexity; a search for those terms found no supporting passage. (LLM, conf=0.73)
- Partly supportedcites [17] [17] Tri Dao, Beidi Chen, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Ré. Pixelated butterfly: Simple and efficient sparse training for neural network models. In International Conference on Learning Representations (ICLR) , 2022. - sourceExperimental — accuracy not yet measured
Section 3.2 (p. 5) says flat butterfly matrices have O(n log k) nonzeros; with maximum stride k = n, this supports an O(N log N) sparsity pattern. The Problem Formulation (p. 17) relates memory accesses to nonzero blocks. The paper does not state that s is often set to either claimed value, or establish the stated Θ IO complexities, including the Θ(N√N) case. (LLM, conf=0.78)
- Partly supportedcites [92] [92] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems , 33, 2020. - sourceExperimental — accuracy not yet measured
Section 2 (pp. 3–4) describes sparse attention and its reduced complexity, including attention to a limited number of random keys. It does not specify a sparsity fraction of (log N)/N or establish Θ(N log N) IO complexity; the Θ(N√N) alternative is attached to a different citation in the citing sentence. (LLM, conf=0.76)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FlashAttention and block-sparse FlashAttention enable longer context in Transformers, yielding higher quality models (0.7 better perplexity on GPT-2 and 6.4 points of lift on long-document classification) and entirely new capabilities: the first”
- Also stated in: § 1 Introduction · plain
“We observe a 0.7 improvement in perplexity on GPT-2 and 6.4 points of lift from modeling longer sequences on long-document classification [13].”
- Also stated in: § 4 Experiments · plain
“Modeling longer sequences yields 6.4 points of lift on two longdocument classification tasks.”
What the sources say
- Partly supportedcites [13] [13] Xiang Dai, Ilias Chalkidis, Sune Darkner, and Desmond Elliott. Revisiting transformer-based models for long document classification. arXiv preprint arXiv:2204.06683 , 2022. - sourceExperimental — accuracy not yet measured
The Introduction and Conclusions report that processing more text can improve long-document classification, and §3.1 describes Longformer as a way to process longer sequences. Our copy does not attribute those gains to FlashAttention or block-sparse FlashAttention, and it does not report the claimed 0.7 GPT-2 perplexity improvement or 6.4-point classification lift (searched those terms and figures). Page numbers were not available in our copy. (LLM, conf=0.86)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup.”
- Also stated in: § 1 Introduction · plain
“Although these methods reduce the compute requirements to linear or near-linear in sequence length, many of them do not display wall-clock speedup against standard attention and have not gained wide adoption.”
What the sources say
- Partly supportedcites [51] [51] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In The International Conference on Machine Learning (ICML) , 2020. - sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says LSH attention reduces complexity from O(L²) to O(L log L), and the synthetic-task analysis (p. 5) reports accuracy losses in some settings. But the paper does not support the claim that approximate attention methods often fail to achieve wall-clock speedups; its Experiments section (p. 9) reports that LSH attention remains fast as sequence length increases while regular attention slows. (LLM, conf=0.87)
- Partly supportedcites [74] [74] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics , 9: 53-68, 2021. - sourceExperimental — accuracy not yet measured
The Abstract says sparse attention reduces compute complexity, and §6.1 reports that local attention was slightly worse than full attention on CIFAR-10. Section 6.3 supports a qualified wall-clock concern: Routing Transformer was slower than local attention on the reported TPU tests. But local attention was faster than full attention, and some Routing Transformer configurations were faster too, so the paper does not establish that approximate attention methods generally fail to achieve wall-clock speedups. (LLM, conf=0.84)
- Partly supportedcites [12] [12] Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference on Learning Representations (ICLR) , 2020. - sourceExperimental — accuracy not yet measured
The Introduction and Related Work (p. 1) describes more tractable attention methods and limitations that can affect their representation power or accuracy. Section 2.2 (p. 3) shows reduced time complexity for the paper’s approximate attention mechanism. But the paper does not say that approximate methods often fail to achieve wall-clock speedups; Section 4.1 (p. 6) instead reports lower wall-clock time for its own method. (LLM, conf=0.84)
- Partly supportedcites [50] [50] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning , pages 5156-5165. PMLR, 2020. - sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says sparse attention and locality-sensitive hashing reduce computational complexity but do not speed up autoregressive inference. That supports the speed claim only for that setting: the paper also says these methods can be trained efficiently on long sequences, and it does not establish a general trade-off in model quality. (LLM, conf=0.86)
- Partly supportedcites [84] [84] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768 , 2020. - sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says sparse attention reduces complexity but incurs a performance drop, while Reformer’s efficiency gains appear only for sequences longer than 2048 tokens. The claim overstates the wall-clock point for sparse attention: the paper reports a limited but real 20% speedup, not no speedup. (LLM, conf=0.85)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“The first major difference is that Rabe and Staats [66] focuses on the reducing the total memory footprint (maximum amount of GPU memory required) while FlashAttention focuses on reducing memory accesses (the number of memory reads/writes).”
What the sources say
- Supportedcites Rabe and Staats [66] source
The Introduction (p. 1) presents algorithms designed to reduce the memory required for attention, and the Conclusion (p. 6) explicitly describes dramatically reducing its memory requirement. The citation is attached to the characterization of Rabe and Staats’s work; the comparison with FlashAttention is the citing authors’ framing. (LLM, conf=0.93)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In prior work, all transformer models have either run out of memory, or only achieved random performance [80].”
What the sources say
- Partly supportedcites [80] [80] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations , 2020. - sourceExperimental — accuracy not yet measured
The Quantitative Results (p. 7) say that all models evaluated on Path-X failed, achieving at best 50% accuracy, consistent with chance performance. The paper does not establish this for all transformer models or for Path-256, and the cited passages do not say that any Path-X model ran out of memory. (LLM, conf=0.88)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“3 LRA accuracy results are known to be highly dependent on the tuning procedure [90].”
What the sources say
- Could not verifycites [90] [90] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nystöm-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence , volume 35, page 14138, 2021. - sourceExperimental — accuracy not yet measured
The Long Range Arena Benchmark section (p. 8) describes the evaluation setup and reports accuracy, but does not say that accuracy is highly dependent on the tuning procedure. Searches for LRA tuning and hyperparameter sensitivity found a sensitivity discussion only for MRPC, a different task (Fine-tuning on Downstream NLP tasks, p. 7). (LLM, conf=0.76)
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Could not verify
Every source the paper cites for this claim was checked and supports it.
In the paper
“As a result, FlashAttention is faster than standard attention (2-4 ) while Rabe and Staats [66] is around the same speed or slightly slower than standard attention.”
What the sources say
- Supportedcites Rabe and Staats [66] source
The citation to Rabe and Staats supports the comparison of their implementation with standard attention, not the separate FlashAttention speed figure. The Abstract (p. 1) says their implementation is within a few percent of standard attention’s runtime, and §5.1 (p. 4) describes its runtime performance as roughly similar in the measured TPU experiment. (LLM, conf=0.92)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, even though structured matrices are efficient in theory, they have not seen wide adoption since it is hard to translate their efficiency to wall-clock speedup since dense unconstrained matrix multiply has very optimize implementation, a phenomenon known as the hardware lottery [41].”
What the sources say
- Partly supportedcites [41] [41] Sara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489 , 2020. - sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FlashAttention instead simplifies the backward pass analytically (Appendices B.2 and B.4). It only recomputes the attention matrix and does not recompute the temporary output of each block. This reduces the memory requirement for the backward pass and yields speedup.”
What the sources say
- Partly supportedcites Rabe and Staats [66] sourceExperimental — accuracy not yet measured
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Table 2 shows up to 3 endto-end speedup compared to Huggingface and 1.7 speedup compared to Megatron-LM.”
- Also stated in: § 4.1 Faster Models with FlashAttention · plain
“GPT-2 small and medium using FlashAttention achieve up to 3 speed up compared to Huggingface implementation and up to 1.7 compared to Megatron-LM.”
What the sources say
- Could not verifycites [87] [87] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics. URL https://www sourceExperimental — accuracy not yet measured
The Deployment section reports a nearly 4× speedup using ONNX, not FlashAttention. Searches for the claimed 3× and 1.7× end-to-end comparisons found no such results in our copy; Related Work mentions Megatron-LM but does not benchmark against it. (LLM, conf=0.77)
- Supportedcites [77] [77] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 , 2019. - source
The citation identifies Megatron-LM as the comparison baseline, rather than attributing the citing paper’s Table 2 speedup measurements to it. The source describes its own model-parallel implementation in “Model Parallel Transformers” (p. 4) and evaluates its scaling in “Scaling Analysis” (p. 6). (LLM, conf=0.89)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 4.1 Faster Models with FlashAttention · plain
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“FlashAttention instead incrementally updates the output (Algorithm 1 line 12) after processing each block, so only one copy of the output is needed (instead of 𝐾 copies for 𝐾 blocks). This means that FlashAttention has smaller total memory requirement compared to Rabe and Staats [66].”
What the sources say
- Partly supportedcites Rabe and Staats [66] sourceExperimental — accuracy not yet measured
Section 4 (p. 4) says the practical implementation keeps one summary per key/value chunk before rescaling them, producing √n summaries and O(√n) memory overhead. That supports the comparison with storing K intermediate results, but the paper does not establish the claimed smaller total memory requirement for FlashAttention. It also describes an O(log n)-memory self-attention algorithm in the Abstract (p. 1), so the comparison cannot be applied to all of its methods. (LLM, conf=0.84)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“FlashAttention yields the fastest single-node BERT training speed that we know of. We train a BERT-large [22] model with FlashAttention on Wikipedia. Table 1 compares our training time to the implementation from Nvidia that set the training speed record for MLPerf 1.1 [58]. Our implementation is 15% faster.”
What the sources say
- Not checked — no full text of the sourcecites [22] [22] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. 2019. - source
- Could not verifycites [58] [58] Peter Mattson, Christine Cheng, Gregory Diamos, Cody Coleman, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, et al. Mlperf training benchmark. Proceedings of Machine Learning and Systems , 2:336-349, 2020. - sourceExperimental — accuracy not yet measured
The citation is attached to the claim that an NVIDIA implementation set the MLPerf 1.1 training-speed record, not to the citing authors’ 15% comparison. The Results (p. 9) discuss only MLPerf v0.5 and v0.6; searches for MLPerf 1.1, NVIDIA, and BERT found no passage establishing the cited record. (LLM, conf=0.76)
hedgeddirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Rabe and Staats [66] suggests that the backward pass can be done without quadratic extra memory by applying gradient checkpointing to the memory-efficient forward pass.”
What the sources say
- Supportedcites Rabe and Staats [66] source
Section 5.2 (p. 5) says the authors apply checkpointing to the function that summarizes attention chunks, allowing intermediate results to be forgotten during the forward pass and recomputed during backpropagation. It also explains that their algorithm never forms the full attention matrix, preserving its memory advantage during differentiation. (LLM, conf=0.94)
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a proof of concept, we implement block-sparse FlashAttention , a sparse attention algorithm that is 2-4 faster than even FlashAttention , scaling up to sequence length of 64k.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Theorem 1. Algorithm 1 returns O softmax QK V with 𝑂 𝑁 2 𝑑 FLOPs and requires 𝑂 𝑁 additional memory beyond inputs and output.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and is optimal for a range of SRAM sizes.”
- Also stated in: § 1 Introduction · plain
“Moreover, we provide a lower bound, showing that no exact attention algorithm can asymptotically improve on the number of HBM accesses over all SRAM sizes.”
- Also stated in: § 3.2 Analysis: IO Complexity of FlashAttention · plain
“Proposition 3. Let 𝑁 be the sequence length, 𝑑 be the head dimension, and 𝑀 be size of SRAM with 𝑑 𝑀 𝑁𝑑 . There does not exist an algorithm to compute exact attention with 𝑜 𝑁 2 𝑑 2 𝑀 1 HBM accesses for all 𝑀 in the range 𝑑 𝑁𝑑 .”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
In the paper
“Even with the increased FLOPs due to recomputation, our algorithm both runs faster (up to 7.6x on GPT-2 [67], Figure 1 right) and uses less memory -linear in sequence length-than standard attention, thanks to the massively reduced amount of HBM access.”
What the sources say
- Not matchedcites [67] [67] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: All gradient checkpointing implementations known to the authors trade speed for memory.
- Not matched
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FlashAttention is up to 3 faster than the standard attention implementation across common sequence lengths from 128 to 2K and scales up to 64K.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Up to sequence length of 512, FlashAttention is both faster and more memory-efficient than any existing attention method, whereas for sequence length beyond 1K, some approximate attention methods (e.g., Linformer) start to become faster.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FlashAttention is up to 20 more memory efficient than exact attention baselines, and is more memory-efficient than the approximate attention baselines.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Theorem 5. Let 𝑁 be the sequence length, 𝑑 be the head dimension, and 𝑀 be size of SRAM with 𝑑 𝑀 𝑁𝑑 . Standard attention (Algorithm 0) backward pass requires Θ 𝑁𝑑 𝑁 2 HBM accesses, while FlashAttention backward pass (Algorithm 4) requires Θ 𝑁 2 𝑑 2 𝑀 1 HBM accesses.”
- Also stated in: § C Proofs · plain
“At each step of the standard attention backward pass, one needs to load inputs of size 𝑁𝑑 or 𝑁 2 from HBM, and needs to write the outputs of size 𝑁 2 or 𝑁𝑑 to HBM. This incurs Θ 𝑁𝑑 𝑁 2 HBM accesses.”
- Also stated in: § C Proofs · plain
“Similar to Theorem 2, we see that each element of K and V is loaded from HBM once. Each element of dK and dV is only written to HBM once. We make 𝑇 𝑐 passes over Q O dO , each pass loading all of Q O dO to HBM. We also make 𝑇 𝑐 passes over dQ , each pass reading/writing all of dQ from/to HBM. Therefore the number of HBM accesses is Θ 𝑁𝑑 𝑁𝑑𝑇 𝑐 Θ 𝑁𝑑𝑇 𝑐 .”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § C Proofs · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our IO-aware implementation of attention is optimal within constants for computing attention on a single GPU.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see that similar to the forward pass, the backward pass performs 𝑂 𝑁 2 FLOPs and only requires 𝑂 𝑁 extra memory beyond inputs, output, output gradient, and input gradients.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The runtimes of many approximate/sparse attention mechanisms grow linearly with sequence length, but FlashAttention still runs faster than approximate and sparse attention for short sequences due to fewer memory accesses.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We do not need to store the dropout mask of size 𝑂 𝑁 2 from the forward pass. Instead, we can save the pseudo-random number generator states from the forward pass and re-generate the dropout mask in the backward pass. This allows us to only use 𝑂 𝑁 extra memory.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 4 shows that that GPT-2 with FlashAttention and context length 4K is still 30% faster than GPT-2 from Megatron with context length 1K, while achieving 0.7 better perplexity.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, block-sparse FlashAttention enables the Transformers to scale to sequence length 64K, achieving 63.1 accuracy 4 on Path-256.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All other algorithms except for Linformer run out of memory on an A100 GPU before 64K, and FlashAttention is still 2 more efficient than Linformer.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“FlashAttention and block-sparse FlashAttention have the same memory footprint, which grows linearly with sequence length.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the LRA benchmark, block-sparse FlashAttention achieves 2.8 speedup, while performing on par with standard attention (Section 4).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Appendix E includes plots of the validation perplexity throughout training, confirming that FlashAttention is as numerically stable as the baselines and produces the same training / validation curves.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Both of these datasets contain very long text documents; the average number of tokens in MIMIC is 2,395 tokens, and the longest document contains 14,562 tokens, while the average and longest numbers in ECtHR are 2,197 and 49,392, respectively.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We confirm that runtime of block-sparse FlashAttention scales linearly in seq. length and is faster than all existing approximate attention baselines.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We first derive the forward and backward passes of attention and show that they can be computed in a memory-efficient manner (requiring extra memory linear instead of quadratic in the sequence length).”
- Also stated in: § B.1 Memory-efficient forward pass · plain
“Therefore the forward pass can be computed with 𝑂 𝑛 extra memory:”
- Also stated in: § B.2 Memory-efficient backward pass · plain
“Therefore the backward pass can also be computed with 𝑂 𝑛 extra memory:”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § B.1 Memory-efficient forward pass · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We confirm that the memory footprint of FlashAttention scales linearly with seq. length and is up to 3 faster than standard attention for common seq. lengths (up to 2K).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe slightly higher speedups on the RTX 3090 (between 2.5-4.5 ), since the memory bandwidth on an RTX 3090 is lower than on an A100 (roughly 900 GB/s vs. 1.5 TB/s).”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As a result, we can support much longer sequences (e.g., up to length 64K). We also support more head dimensions (16, 32, 64, 128) and broader GPU types (all Turing and Ampere GPUs at the time of writing).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We train the model on 8 A100-40GB GPUs, and we measure the wall-clock training time. Training GPT-2 small takes between 2.7-9.5 days, and training GPT-2 medium takes between 6.9-21.0 days (Table 2).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“For the block-sparse case, notice that we only need to load blocks corresponding to nonzero blocks. As a result, the number of HBM accesses are scaled by 𝑠 , the fraction of nonzero blocks in the block-sparsity mask. However, for small values of 𝑠 , we would still need to write the result O 𝑁 𝑑 . Therefore the number of HBM accesses is”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Longformer and BigBird do not support sequence lengths longer than 8092.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We use FP16 for all measurements, except for Local Attention, whose implementation only supports FP32.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“After hyperparameter tuning, almost all of the attention methods achieve similar accuracy on all of the five LRA tasks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Block-Sparse (OpenAI) does not support sequence lengths longer than 4096.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to FMHA, the overall runtime of FlashAttention is about 4% slower for sequence length 128, 8% faster for sequence length 256, and 5% faster for sequence length 512.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The Megatron implementation does not support sequence lengths longer than 2048.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Generally FlashAttention is slightly faster than FMHA in the forward pass and slightly slower than FMHA in the backward pass.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In the forward pass, it stores the attention matrix softmax mask QK 𝑇 to HBM to be used in gradient computation. As a result, it does not offer substantial memory saving (though for shorter sequences memory footprint is often not a primary concern).”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We see that FlashAttention behaves the same as the baseline implementation and the validation perplexity curves of the two implementations almost lie on top of each other.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For Path-X, we take the best performing checkpoint (according to val accuracy), and additionally fine-tune it for 200 epochs with the same warmup and learning rate (this adds roughly 4 points of accuracy to FlashAttention for Path-X, but the model starts overfitting afterwards).”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length.”
- Also stated in: § 1 Introduction · plain
“since the self-attention module at their heart has time and memory complexity quadratic in sequence length.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 Introduction · plain
None of the sources cited for this claim have been checked yet.
In the paper
“IO-aware algorithms have been critical for similar memory-bound operations, when reading and writing data can account for a large portion of the runtime-such as database joins [71], image processing [70], numerical linear algebra [4], and more [40, 85].”
What the sources say
- Not matchedcites [71] [71] Raghu Ramakrishnan, Johannes Gehrke, and Johannes Gehrke. Database management systems , volume 3. McGraw-Hill New York, 2003. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites [70] [70] Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. Acm Sigplan Notices , 48(6):519-530, 2013. - source
- Not matchedcites [4] [4] L Susan Blackford, Antoine Petitet, Roldan Pozo, Karin Remington, R Clint Whaley, James Demmel, Jack Dongarra, Iain Duff, Sven Hammarling, Greg Henry, et al. An updated set of basic linear algebra subprograms (blas). ACM Transactions on Mathematical Software , 28(2):135-151, 2002. -
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We implement FlashAttention in CUDA to achieve fine-grained control over memory access and fuse all the attention operations into one GPU kernel.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Common Python interfaces to deep learning such as PyTorch and Tensorflow do not allow fine-grained control of memory access.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This requires writing the attention algorithm in a considerably lower-level language than PyTorch, and requires significant engineering effort.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Implementations may also not be transferrable across GPU architectures.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our current approach to building IO-aware implementations of attention requires writing a new CUDA kernel for each new attention implementation.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The discrepancies may be due to subtle distribution shifts: MIMIC-III contains specialized medical text and thus may be more susceptible to a distribution shift in the document length, whereas ECtHR contains general language.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Path-256 requires longer sequences but has relatively shorter paths than Path-X, so it is easier to obtain a higher accuracy.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We describe the FlashAttention algorithm to implement both the forward and the backward passes on GPUs that reduces HBM accesses, leading to both faster runtime and smaller memory footprint.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Attention time and memory scale quadratically in the sequence length.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Each block requires more memory, so we need to use smaller block sizes to fit into SRAM.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, we can repeatedly load the inputs Q K and recompute the block of the attention matrix that we need, significantly reducing HBM access.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“These methods did not successfully run the backward pass with masking due to a bug in external libraries, so we measured them without masking to be generous.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We only measure memory footprint once, since it does not vary between runs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
Common Python interfaces to deep learning such as PyTorch and Tensorflow do not allow fine-grained control of memory access.
“However, common Python interfaces to deep learning such as PyTorch and Tensorflow do not allow fine-grained control of memory access.”
The paper relies on this without support.
All gradient checkpointing implementations known to the authors trade speed for memory.
“While gradient checkpointing has been suggested to reduce the maximum amount of memory required [66], all implementations (that we know off) have to trade speed for memory.”
The paper relies on this without support.
Attention time and memory scale quadratically in the sequence length.
“However, one of their computational bottlenecks is that their time and memory scales quadratic in the sequence length.”
The paper relies on this without support.
RTX 3090 memory bandwidth is roughly 900 GB/s and A100 memory bandwidth is roughly 1.5 TB/s, and the RTX 3090's is lower.
“since the memory bandwidth on an RTX 3090 is lower than on an A100 (roughly 900 GB/s vs. 1.5 TB/s).”
The paper relies on this without support.
T4 SRAM is smaller than A100 SRAM.
“T4 SRAM is smaller than A100, so we need to make the block sizes smaller in FlashAttention .”
The paper relies on this without support.
T4 GPUs are commonly used for inference.
“T4 GPUs are commonly used for inference, so we also report speedup on the forward pass only.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, FlashAttention [166-167] uses CUDA kernels that fuse block-wise matrix multiplication with an online softmax computation, to reduce the quadratic time and memory complexity in the sequence length of vanilla self-attention.”
From A Survey of Large Language Models (2026) · cites this paper as [166]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The additive pre-softmax form is compatible with FlashAttention [2,13,23] .”
From Inference-Time Attention Steering for Vision-Language-Action Driving Models (2026) · cites this paper as [2]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent works extend context windows to 100K+ tokens [18], [19], [23].”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“unlimited context remains computationally infeasible and doesn't address selective retention or consistency.”
From From Goldfish to Elephant: Long-Term Memory for Large Language Models via Selective Storage and Hierarchical Compression (2026) · cites this paper as [19]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Specifically, in LLMs, each decoding step involves generating a single token, with the transfer of weights between High Bandwidth Memory (HBM) and cache contributing to significant overhead [33, 34].”
From SnapKV: LLM Knows What You are Looking for Before Generation (2024) · cites this paper as [33]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“FlashAttention thoroughly researched and implemented various tiling techniques, accelerating selfattention by 2-4 × while cutting memory operations by 10-20 × .”
From ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition (2024) · cites this paper as Dao et al., 2022
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, SGLang is compatible with many other common inference optimizations [60, 39, 3, 23, 59, 10, 26, 15, 19, 32, 31, 11].”
From SGLang: Efficient Execution of Structured Language Model Programs (2023) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Chunked-prefill schedulers such as Sarathi-Serve [1], memory-efficient attention kernels [8, 14], paged memory management [2], and tensor/pipeline parallelism layouts [10, 31] mitigate but do not eliminate this asymmetry, and they leave the cross-pool transport question largely open.”
From Janus: Joint Prefill/Decode Disaggregation with KV-Cache-Aware Multi-Cloud Routing for Edge-Adjacent LLM Serving (2026) · cites this paper as [8]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“FlashAttention [5] exploits the asymmetric GPU memory hierarchy to bring significant memory saving (linear instead of quadratic) and runtime speedup (2-4 × compared to optimized baselines), with no approximation.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By significantly reducing the amount of memory reads/writes, FlashAttention yields 2-4 × wall-clock speedup over optimized baseline attention implementations.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The backward pass also achieves 2-4 × wall-clock speedup due to reduce memory reads/writes.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the backward pass, by re-computing the values of the attention matrices S and P once blocks of inputs Q , K , V are already loaded to SRAM, FlashAttention avoids having to store large intermediate values. By not having to save the large matrices S and P of size 𝑁 × 𝑁 , FlashAttention yields 10-20 × memory saving depending on sequence length”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“FlashAttention [13] applies tiling and kernel optimizations to reduce the peak memory of attention computation and reduce I/O costs.”
From Efficient Memory Management for Large Language Model Serving with PagedAttention (2023) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“Recent improvements in hardware (e.g., faster GPUs with more memory) and algorithms (Dai et al., 2019;Dao et al., 2022;Poli et al., 2023;Rubin and Berant, 2023, inter alia) have resulted in language models with larger context windows (e.g., 4096, 32K, and even 100K tokens)”
From Lost in the Middle: How Language Models Use Long Contexts (2024) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“Compactor scoring incurs minimal overhead compared to the cost of a single Flash Attention (Dao et al., 2022) pass for large contexts”
From Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores (2025) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“On modern hardware accelerators (GPUs) most operations (except matrix multiply) are bounded by memorybandwidth (Dao, Fu, Ermon, et al. 2022; Ivanov et al. 2021; Williams, Waterman, and Patterson 2009).”
From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Dao, Fu, Ermon, et al. 2022
The cited source says what the claim attributes to it.
In the citing paper
“A primary focus has been on accelerating kernel execution by minimizing memory movement during attention computation, with FlashAttention being a prominent example [166].”
From A Survey of Large Language Models (2026) · cites this paper as [166]
The cited source says what the claim attributes to it.
In the citing paper
“Partition strategies in ChunkAttention are built on online softmax (Milakov and Gimelshein, 2018) and inspired by FlashAttention (Dao et al., 2022; Dao, 2023), which adopted the same algorithm.”
From ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition (2024) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“FlashAttention [3] shows that attention performance depends heavily on data movement.”
From TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing (2026) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“FlashAttention exploits asymmetric GPU memory hierarchy to achieve linear memory scaling instead of quadratic requirements”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [200]
The cited source says what the claim attributes to it.
In the citing paper
“Despite this growing importance, LLMs consistently face challenges in processing long contexts, primarily due to the substantial computational resources required for long sequence training (Dao et al., 2022; Dao, 2023) as well as the apparent inability to generalize to sequences longer than those encountered during training (Chen et al., 2023; Peng et al., 2023b).”
From ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“While performance monotonically improves, longer prompts require much more compute costs (Dao et al., 2022).”
From Does Few-Shot Learning Help LLM Performance in Code Synthesis? (2024) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“FlashAttention (Dao et al., 2022) achieves 2-4× speedup over standard attention while reducing memory from O(N 2 ) to O(N ).”
From KernelBench-Verified: Do LLM-Generated Kernels Actually Beat PyTorch? (2026) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“As with FlashAttention , Algorithm 1 returns the correct output O = softmax ( QK ⊤ ) V (with no approximation), using 𝑂 ( 𝑁 2 𝑑 ) FLOPs and requires 𝑂 ( 𝑁 ) additional memory beyond inputs and output (to store the logsumexp 𝐿 ).”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“up to 10-20 × memory saving”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Dao et al. [5] proposed to reorder the attention computation and leverages classical techniques (tiling, recomputation) to significantly speed it up and reduce memory usage from quadratic to linear in sequence length.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“FlashAttention applies the classical technique of tiling to reduce memory IOs, by (1) loading blocks of inputs from HBM to SRAM, (2) computing attention with respect to that block, and then (3) updating the output without writing the large intermediate matrices S and P to HBM.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Our version is on par with or better than the original on all cases considered, while covering more use-cases and hardware.”
From DINOv2: Learning Robust Visual Features without Supervision (2023) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“While mLSTM is parallelizable analog to FlashAttention (Dao et al., 2022;Dao, 2024) or GLA (Yang et al., 2023), sLSTM is not parallelizable due to the memory mixing (hidden-hidden connections).”
From xLSTM: Extended Long Short-Term Memory (2024) · cites this paper as Dao et al., 2022
The cited source says what the claim attributes to it.
In the citing paper
“To avoid text truncation, researchers try to extend the context length of LLMs via training from scratch, fine-tuning, or specific algorithms (e.g., FlashAttention; Dao et al., 2022).”
From Recursively summarizing enables long-term dialogue memory in large language models (2025) · cites this paper as Dao et al., 2022
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“FlashAttention-2 providing approximately twice the speed through reduced non-matrix multiplication operations and optimized work distribution [200, 199].”
From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [200]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The first version of FlashAttention parallelizes over batch size and number of heads. We use 1 thread block to process one attention head, and there are overall batch size · number of heads thread blocks.”
From FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) · cites this paper as [5]
Experimental — accuracy not yet measuredIn the citing paper
“Secondly, the practicality of fine-tuning poses challenges for many labs. Even minimally fine-tuning foundation models can require extensive GPU resources, given that, for example, scGPT's architecture relies on the use of FlashAttention [16] not available for older and smaller graphics cards 3 .”
From Assessing the limits of zero-shot foundation models in single-cell biology (2023) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ E.6 Full Benchmarking Results
“Longformer and BigBird do not support sequence lengths longer than 8092.”
§ E.6 Full Benchmarking Results
“| Longformer | 1.27 | 1.31 | 1.29 | 2.04 | 5.24 | 10.74 | 25.95 | - | - | - | | BigBird | 1.33 | 1.28 | 1.32 | 1.81 | 5.55 | 11.44 | 27.45 | - | - | - |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1 Faster Models with FlashAttention
“Table 2 shows up to 3 endto-end speedup compared to Huggingface and 1.7 speedup compared to Megatron-LM.”
§ 4.1 Faster Models with FlashAttention
“| GPT-2 small - FlashAttention | 18.2 | 2.7 days (3.5 ) |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.3 Benchmarking Attention
“On the other hand, block-sparse FlashAttention is faster than all implementations of exact, sparse, and approximate attention that we know of, across all sequence lengths.”
§ E.6 Full Benchmarking Results
“| FlashAttention | 0.35 | 0.36 | 0.80 | 2.52 | 9.16 | 36.70 | 146.13 | 583.45 | 2332.01 | 9323.63 | | Block-Sparse FlashAttention | 0.91 | 0.83 | 0.94 | 0.92 | 1.83 | 3.50 | 7.02 | 13.56 | 26.71 | 53.92 |”
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 9
- [4][4] L Susan Blackford, Antoine Petitet, Roldan Pozo, Karin Remington, R Clint Whaley, James Demmel, Jack Dongarra, Iain Duff, Sven Hammarling, Greg Henry, et al. An updated set of basic linear algebra subprograms (blas). ACM Transactions on Mathematical Software , 28(2):135-151, 2002. -
- [26][26] Jean Feydy, Joan Glaunès, Benjamin Charlier, and Michael Bronstein. Fast geometric learning with symbolic matrices. Advances in Neural Information Processing Systems , 33, 2020. -
- [40][40] John Hennessy and David Patterson. Memory hierarchy design. Computer Architecture: A Quantitative Approach , pages 390-525, 2003. -
- [44][44] Zhe Jia and Peter Van Sandt. Dissecting the Ampere GPU architecture via microbenchmarking. GPU Technology Conference, 2021. -
- [55][55] Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. -
- [59][59] Frank McSherry, Michael Isard, and Derek G Murray. Scalability! but at what COST ? In 15th Workshop on Hot Topics in Operating Systems (HotOS XV) , 2015. -
- [65][65] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, highperformance deep learning library. Advances in neural information processing systems , 32, 2019. -
- [67][67] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. -
- [88][88] David P Woodruff. Optimal space lower bounds for all frequency moments. In SODA , volume 4, pages 167-175. Citeseer, 2004. -
Cited by
Sorted by citations
Items 801–900 of 1,286
Cai, J., Kong, D., Huang, H., Jiang, Z., Ma, Z., Guo, Q., Zhang, Z., Shi, G., Gao, M., Ma, K., et al. (2026) · IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca68181.2026.11408526
An Efficient Swin Transformer Architecture Based on Grouped Query Attention
Wang, L., & Ma, Z. (2025) · International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · doi:10.1109/prai67447.2025.11412576
Lim, S., Kim, J.-H., Moon, S., Cha, J., Seo, D., Kim, J., Lee, H., Lee, J., & Kim, J. (2026) · IEEE Journal of Solid-State Circuits · doi:10.1109/jssc.2026.3663603
Towards Efficient Acceleration of Hyena and Kolmogorov–Arnold Networks on NPUs
Das, A., Kim, H., Raha, A., Kundu, S., Ghosh, S. K., Mathaikutty, D. A., & Raghunathan, V. (2025) · International Conference on Intelligent Computing and Systems at the Edge (ICEdge) · doi:10.1109/icedge67252.2025.11412934
Low-Rank SVD Compression for Memory-Efficient Transformer Attention
Shah, M. A., Khan, R. Q., & Masood, M. (2025) · International Conference on Innovative Computing (ICIC) · doi:10.1109/icic68258.2025.11413215
DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving
Yang, M., Choi, J.-Y., Moon, K., Jang, M., & Jeon, E. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11400767
SwiftSpec: Disaggregated Speculative Decoding and Fused Kernels for Low-Latency LLM Inference
Zhang, Z., Jiang, Z., Jiang, C., Yu, M., Zheng, S., Lin, H., Liu, X., & Hoffmann, H. (2026) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3779212.3790246 · Cited by 3
PipeViT: Accelerating Vision Transformers via Intra-Layer Pipelining
Zhou, X., Xu, Y. E., Lu, H., Yu, J., & Wang, K. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420796
M3DKV: Monolithic 3D Gain Cell Memory Enabled Efficient KV Cache & Processing
Yang, J., Su, Y., Fu, Y., Tang, J., & Yan, B. Y. (2026) · Asia and South Pacific Design Automation Conference (ASP-DAC) · doi:10.1109/asp-dac66049.2026.11420572
Ruf, T., & Rekabdar, B. (2025) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi67134.2025.00016
InfoST: An Information-Theoretic Framework for Evaluating Spatio-Temporal Prediction
Khanal, N., Li, H., & Zhuang, Y. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00035
Tang, W., Ma, C., Zhou, S., ZHANG, S., Wang, Q., Wu, K., Hong, H., Lin, J., Yang, Y., & Yu, K. (2026) · Journal of Earth Science · doi:10.1007/s12583-025-0329-8
ChannelFormer: A transformer-based model for fNIRS missing channel reconstruction
Zhao, Y., Li, K., Cao, Y., & Chen, Y. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.115032
Di, L., Zhang, B., & Wang, Y. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3673920
LightMotion: A Lightweight Motion Prediction Model with Spatial Attention and Progressive Learning
Aditya, W., Chen, W.-H., Lin, Y.-C., & Yang, T. J. (2025) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc60802.2025.11423658
Contrast & Compress: Learning Lightweight Embeddings for Short Trajectories
Vivekanandan, A., Hubschneider, C., & Zöllner, J. M. (2025) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc60802.2025.11423089
Liu, C., Yuan, C., Sun, J., Zhang, Z., Huang, Z., Guo, S., Ang, M. H., & Tay, E. H. (2025) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc60802.2025.11423650
Differential attention for retinal image captioning via cross-modal integration
Hieu, N. H., Thanh, L. V., Lan, L. T. H., & Pham, H. V. (2026) · Signal Image and Video Processing · doi:10.1007/s11760-026-05208-6
AccelAes: Accelerating Diffusion Transformers for Training-Free Aesthetic-Enhanced Image Generation
Yin, X., Xu, C., Zhou, H., Wei, B., & Cai, W. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37517-9_23 · Cited by 4
TokenFlow: Token-Level GPU Sharing and Adaptive Scheduling for Multi-Model Concurrent LLM Inference
Zeng, K., Huang, Z., Yang, Y., Zhang, X., Huang, S.-Y., & Meng, R. (2026) · International Conference on Electronic Technology and Information Science (ICETIS) · doi:10.1109/icetis70504.2026.11633493
End-to-End Contextual Isolation for Large-Scale Multi-Tenant AI Platforms
Katta, R. R., & Dorai, V. (2025) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca66035.2025.11431065
Zhang, K., Zhang, Y., Zheyang, L. I., Tan, L., Tan, W., Re, Y., Hu, J., & Xu, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6434892
Patra, A., Kundu, S., Dey, A., Saha, T., Karmakar, S., & Bishnu, S. K. (2026) · International Conference on Electronics, Materials Engineering & Nano-Technology (IEMENTech) · doi:10.1109/iementech202669403.2026.11434122
YOLO-AF: An Improved Lightweight Architecture for Real-Time Cell Detection in Imaging Flow Cytometry
Liu, Z., Yong, Q., Han, Y., Chen, L., Lu, Y., Mu, R., Yang, M., (15105980), H. H. Z., Yang, S., Yang, Q., et al. (2026) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001426570065
3-D 1T-nC-1T Ferroelectric-Based KV Cache for Efficient Long-Context LLM Inference
Xu, W., Luo, D., Deng, M., Cao, S., Zhong, S., Li, M. J., Huang, Q., & Huang, R. (2026) · IEEE Transactions on Electron Devices · doi:10.1109/ted.2026.3670094
IMBPS - Iterative MLP Blocks with Parameter Splits for Improving LLM Inference
Nagaraja, G. P., Ramachandran, A. C., Sharma, S., Govindarajan, R., & Raghavendra, P. (2025) · IEEE International Conference on High Performance Computing, Data, and Analytics · doi:10.1109/hipc66333.2025.00027
SpaceFusion++: An operator fusion scheduler for neural language model inference
Zhu, L., Yao, J., & Guan, H. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.103708
Adaptive cooperation in large-scale MARL via dynamic grouping and cooperation switching
Wang, B., Zhang, L., Zhou, S., Chen, H., Huang, J. X., & Gong, J. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132180
Li, H., Fan, J., & Cai, W. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.108890
ThermoFormer: Predicting Protein Melting Temperature Through Large-Scale Pretraining
Li, J., & Li, M. (2026) · Catalysts · doi:10.3390/catal16040288
CardioDiT: Latent Diffusion Transformers for 4D Cardiac MRI Synthesis
Seyfarth, M., Müller, S. K., Ghanaat, A., Ayx, I., Fastenrath, F., Wild, P. S., Hertel, A., Papavassiliu, T., Dar, S. U. H., & Engelhardt, S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38189-7_7
Investigating the Fundamental Limit: A Feasibility Study of Hybrid-Neural Archival
Armstrong, M., Qiu, Z., Vo, H. Q., & Mukherjee, A. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-32729-1_2
Wang, Y., Wu, Z., Chong, Y. J., & Fong, X. (2026) · IEEE Journal on Emerging and Selected Topics in Circuits and Systems · doi:10.1109/jetcas.2026.3679018
Multimodal Fusion of Imaging and Multi-omics Data for Enhanced Breast Cancer Detection
Kantha, D., & Loew, M. H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-18474-0_36
Dual Fine-Tuning and Dual Alignment Training for Medical Language Models
Lu, J., & Chen, Z. (2025) · International Conference on Cyberworlds (CW) · doi:10.1109/cw68232.2025.00066
EFlow: Fast Few-Step Video Generator Training from Scratch via Efficient Solution Flow
Park, D., Li, Y., Tulyakov, S., & Kag, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37314-4_17 · Cited by 1
Zhang, H., Guo, B., Jin, L., Xu, J., He, Y., Tian, T., & Yang, X. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132290
Han, Y., Wu, Y., Jiao, Z., Wang, Y., Liu, X., Wang, S., Xu, H., Hu, X., & Zhang, L. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37041-9_16 · Cited by 3
Open Full-duplex Voice Agent with Speech-to-Speech Language Model
Casanova, E., Chen, C., Hu, K., Pasad, A., Rastorgueva, E., Narasimhan, S. L., Deng, S., Hosseini-Asl, E., Żelasko, P., Mendelev, V., et al. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434669
Benchmarking Rotary Position Embeddings for Automatic Speech Recognition
Zhang, S., Parcollet, T., van Dalen, R. C., & Bhattacharya, S. (2025) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru65441.2025.11434788
DF-DETR: An Algorithm for Small Object Detection in UAV Aerial Images
Bu, L., Yang, H., & Yang, B. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-8417-8_42
Embracing Imperfections : Hardware-compatible Neural Networks for Neuromorphic Computing
Majumdar, A. (2023) · theses.fr (ABES) · doi:10.70675/4bf81ddbz25cfz48c6zaf34zcaedca216e17
TransLAP: Non‐Isometric Shape Matching Using Transformer Based Linear Assignment Problem
Amirfathiyan, A., & Ebrahimnezhad, H. (2026) · IET Image Processing · doi:10.1049/ipr2.70333
Methodology and tooling for energy-efficient neural networks computation and optimization
Waltsburger, H. (2024) · theses.fr (ABES) · doi:10.70675/6aadb508z47bfz4dc6z95b6zbd5a77514a0a
Optimizing Memory Usage when Training Deep Neural Networks
Zhao, X. (2024) · theses.fr (ABES) · doi:10.70675/14b989cezef3bz4263z8c05z34a16bd91adc
Developping and exploring the interest of deep learning approaches in the field of multi-omics data
Beaude, A. (2024) · theses.fr (ABES) · doi:10.70675/717fb1ffz8eaaz47d6z88c4zec38f96c9ae2
Representation learning of multimodal and longitudinal data for renal transplantation monitoring
Milecki, L. (2024) · theses.fr (ABES) · doi:10.70675/8478bb39za27fz4524z8e86z5df6c7d2fc00 · Cited by 1
Gómez-Luna, J. (2026) · Elsevier eBooks · doi:10.1016/b978-0-44-343900-1.00031-7
Mécanisme d’attention en apprentissage profond pour la mise en correspondance d’images
Vilain, M. (2024) · theses.fr (ABES) · doi:10.70675/002c4e0ezc61fz474az8120za63e66480b2e
Language models for document understanding
Douzon, T. (2023) · theses.fr (ABES) · doi:10.70675/66ce60f8zcccaz476eza4d6zf801a2256b86 · Cited by 9
Harnessing symmetries for modern deep learning challenges : a path-lifting perspective
Gonon, A. (2024) · theses.fr (ABES) · doi:10.70675/c0dd6b18z746ez4cfezae65zaba0b12d7641
Task Specification and Generalization in Multi-Task Reinforcement Learning
Cachet, T. (2025) · theses.fr (ABES) · doi:10.70675/08eadb30z8466z4763zbba6za6a4dab9a29e
Hardware Accelerators for a Sustainable AI Future
Garzón, E. (2025) · IEEE International Conference on Application of Information Technologies in Engineering, Management and Science (ICAI-TEMS) · doi:10.1109/icai-tems64510.2025.11467631
A collaborative optimization framework for efficient long-sequence Audio-Visual understanding
Ma, Y., Bai, H., Li, X., Shang, B., Zhang, L., & Sun, W. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132367
LLM Quantization: Enabling Edge Intelligence
Heidari, M., Zad, S., Boroojeni, K. G., & Katzman, M. (2026) · Lecture notes in electrical engineering · doi:10.1007/978-981-95-0429-9_9
Enhancing Transformer-Based Document Recognizer with Dynamic Attention Sink and Structured SkippingPreprint
Mao, L., Wang, W., Li, Q., & Wu, Z. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6560014
Hu, Y., Zhang, W., Huang, Y., Huang, Y., Ma, X., & Zhao, Z. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00723-5
Meng, F., Liu, H., He, C., Wang, X., Liu, S., Niu, C., Jia, Y., & Zan, H. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.133651
Development of a Near Real-Time PF Coil Temperature Prediction System Using Deep Learning
Kwon, G., & Lee, H. (2026) · IEEE Transactions on Plasma Science · doi:10.1109/tps.2026.3680879
Spatially-Restricted Single-Shot Token Merging for Efficient Vision TransformersPreprint
Keddous, F. E., & Nakib, A. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6582690
Basic Theory of Autonomous Driving
Huang, Y., & Yang, Z. (2026) · Springer eBooks · doi:10.1007/978-981-95-3725-9_2
Random Rolling Attention Augmentation for Efficient Agricultural Disease and Pest DetectionPreprint
Han, W., Ding, Y., Kang, J., Zhao, A., & Dong, X. (2026) · Research Square · doi:10.21203/rs.3.rs-9318824/v1
Jarsania, J., & Patel, V. (2025) · International Conference on Artificial Intelligence, Blockchain, Cloud Computing, and Data Analytics (ICoABCD) · doi:10.1109/icoabcd67551.2025.11470747
Zheng, L., Hu, Y., Shen, C., Zhao, Z., Shi, B., Zhang, Y., Zhao, L., Li, W., & Li, K. (2026) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2026.3683867
Yang, X., Xiao, Q., Xu, Y., Yang, J., Hou, Y., Long, W., Huang, M., & Zhang, Y. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag182
Qi, X., & Huang, Y. (2026) · International Symposium on Computer Applications and Information Systems (ISCAIS) · doi:10.1109/iscait69154.2026.11477210
Effective Multi-Scale Temporal Modeling for Small-Data Speech Rate Classification
Zheng, Q., & Abliz, W. (2025) · International Conference on Artificial Intelligence, Digital Media Technology and Interaction Design (ICADI) · doi:10.1145/3795926.3796024
MAR: Efficient Large Language Models Via Module-Aware Architecture Refinement
Cai, J., Wang, G., Zhao, K., Tang, J., Wang, X., Leng, L., Cheng, R., Ma, Y., & Guo, Q. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464338
Collaborative Compression for Large-Scale MoE Deployment on Edge
Chen, Y., Xie, Y., Yang, R., Jiang, W., Wang, W., He, Y., Chen, Y., Zhao, P., & Wang, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463800 · Cited by 3
Redundancy-Aware Feature Refinement for Lightweight Image Super-Resolution
Wu, Y., Liu, Y., Pang, Y., Pan, T., Yang, J., & Zeng, D. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464928
Learning to Cascade: A Pomdp Approach to Sequential Model Selection
Darjani, A. S., & Zois, D. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464865
Xu, P., Yan, F., Zhou, Y., Tao, Y., & Liu, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462489
ContaGPT: A Domain-Adapted LLM for Romanian Financial and Accounting Applications
Tudor, C., Achirei, Ș.-D., & Simionescu, C. (2025) · International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) · doi:10.1109/synasc69064.2025.00047
Geneses: Unified Generative Speech Enhancement and Separation
Asai, K., Nakata, W., Saito, Y., & Saruwatari, H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464549
Two-Stage Attention Triple Enhancement and U-KAN Diffusion for Few-Shot Knowledge Graph Completion
Gao, X., & Bingsong, T. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464940
MotionFusion: Fusing Motion and Saliency for Fast Video Large Language Model Inference
Du, C., Ren, J., & Zhang, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464991
Dai, H., Qu, T. L., Hu, Y., Si, S., Jiang, B., Chen, X., Liu, K., & Lv, S. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461664
Multi-Scale State Space Modeling for Cross-Modal Infrared and Visible Image Fusion
Xiong, L., Sun, C., & Huang, J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11460492
CORM: Coarse-To-Fine-Grained Offloading for Smoe LLM Inference on Consumer-Grade GPU
Tao, Z., Peng, J., Zhu, H., Zhang, W., Chen, T., & Zhang, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461364
LLM‐Based Scientific Assistants for Knowledge Extraction: Which Design Choices Matter?
Exler, D., Raimann, M., Münker, M. F., Rosin, M., Gómez, J. E. U., Niemeyer, C. M., Reischl, M., & Rettenberger, L. (2026) · Advanced Intelligent Discovery · doi:10.1002/aidi.202500197
SEDPA: SVD-Enhanced Dual Path Attention for Efficient Inference
Zhou, L., Sun, H., Hu, Y., Xue, Z., & Wang, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461401
Remote Sensing Semantic Change Detection With Multidimensional Spatiotemporal Scanning
Chen, Z., Chen, H., & Mu, H. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3686361
URoPE : Universal Relative Position Embedding across Geometric Spaces
Xie, Y., Meng, D., Peng, C., Hu, Y., Herau, Q., Tomizuka, M., & Zhan, W. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37324-3_11 · Cited by 3
Volume Transformer: Revisiting Vanilla Transformers for 3D Scene Understanding
Yilmaz, K., Kruse, A., Höfer, T., de Geus, D., & Leibe, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37044-0_12 · Cited by 1
SweetSpot: An Analytical Model for Predicting Energy Efficiency of LLM Inference
Cavagna, H. P., Proia, A., Madella, G., Esposito, G. B., Antici, F., Cesarini, D., Kiziltan, Z., & Bartolini, A. (2026) · ACM/SPEC International Conference on Performance Engineering (ICPE) · doi:10.1145/3777884.3797011
viNPU: Optimizing Vision Transformer Inference on Mobile NPUs
Lee, J., Kim, G., Jung, C. R., Kim, J., Park, S., & Cha, H. (2026) · European Conference on Computer Systems · doi:10.1145/3767295.3803619
Hierarchical Sparse Plus Low Rank Compression of LLM
Kumar, P., & Gupta, A. (2025) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3799830.3799837
Ploskonka, M. (2026) · Acta Informatica Pragensia · doi:10.18267/j.aip.313
Attention Mechanisms, Transformers and LLMs
Soria‐Olivas, E., Belda, E. A., Salvador, D. B., Cámara, P. H., Isla, M. A. S.-M., & Belenguer, P. R. (2026) · Studies in computational intelligence · doi:10.1007/978-3-032-15854-3_4
Evaluating CUDA Tile for AI Workloads on Hopper and Blackwell GPUs
Yadav, D. K., Zhao, T., & Kumar, D. (2026) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac69091.2026.00026
Li, Z., Yuan, J., & Zhang, Z. (2026) · Scientific Reports · doi:10.1038/s41598-026-50451-z
SHARD: A Compatibility Framework for Deploying Transformer Models on Edge NPUs
Mohan, A., Thompson, R., Keller, E. R., & Zhao, M. (2026) · Workshop on Machine Learning and Systems (EuroMLSys) · doi:10.1145/3805621.3807618
How foundation models will revolutionize robot swarms
Strobel, V., Dorigo, M., & Fritz, M. (2026) · Science Robotics · doi:10.1126/scirobotics.adz1543 · Cited by 1
ThreadWeaver: Plan-and-Restate Dialog over Ultra-Long Histories
Praneethreddy, B., & Baddipadiga, S. R. P. R. (2026) · International Conference on Emerging Smart Computing and Informatics (ESCI) · doi:10.1109/esci68015.2026.11493358
A Hybrid Coarse-to-Fine Method for Image-to-Point Cloud Registration with Probabilistic Supervision
Zhang, Z., Fan, B., Ai, J., & Wang, Z. (2025) · China Automation Congress (CAC) · doi:10.1109/cac67268.2025.11487198
Model Reveals What to Cache: Profiling-Based Feature Reuse for Video Diffusion Models
Ma, X., Liu, Y., Liu, Y., Wu, X., Zheng, M., Wang, Z., Lim, S.-N., & Yang, H. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01593 · Cited by 10
Lin, C.-Y., Emon, S. H., & Tseng, T. B. (2026) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2026.3689009 · Cited by 1
$\mathcal{D}$-Attn: Decomposed Attention for Large Vision-and-Language Models
Kuo, C.-W., Zhu, S., Chen, F., Shen, X., & Wen, L. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.02219
LiT: Delving into a Simple Linear Diffusion Transformer for Image Generation
Wang, J., Kang, N., Yao, L., Chen, M., Wu, C., Zhang, S., Xue, S., Liu, Y., Wu, T., Liu, X., et al. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01491
X-Prompt: Generalizable Auto-Regressive Visual Learning with In-Context Prompting
Sun, Z., Chu, Z., Zhang, P., Wu, T., Zang, Y., Dong, X., Xiong, Y., Lin, D., & Wang, J. (2025) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51701.2025.01604
Metadata
- Authors
- Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 16344-16359
- ISSN
- 1049-5258 (ISSN-L)
- arXiv
- 2205.14135
- OpenAlex
- W7133227460
- Cited by
- 1,214
- References
- 0
- Citations by year
- 2026: 555 · 2025: 316 · 2024: 256 · 2023: 86 · 2022: 1
- FWCI
- 132.50
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763315, 1563078, 1937301)
- U.S. Department of Defense (N000141712266)
- Accenture
- Nvidia
- VMware
- National Institutes of Health (N000141712266, CCF1563078, CCF1763315)
- National Defense Science and Engineering Graduate
- Office of Naval Research (N000141712266, ONR-N00014, N00014-20, N00014)
- National Institute of Biomedical Imaging and Bioengineering (U54EB020405)
- Topics
- Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- FlashAttention, memory efficiency, transformer attention, large language models, GPU memory optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
