CoChat
Sign Up
PreprintCited by 23Topic ModelingAdvanced Neural Network ApplicationsNatural Language Processing Techniques

The DOI matches its DataCite record; 2 of the 14 references it cites were found in a registry or an index, and 12 are not checked yet.

Gated Linear Attention Transformers with Hardware-Efficient Training

Open access
Yang, Songlin, Wang, Bailin, Shen, Yikang, Panda, Rameswar, Kim, Yoon·2023·arXiv
DOI 10.48550/arxiv.2312.06635arXiv 2312.06635
Open this paper in CoChat — free accountOpen PDF

Abstract

Transformers with linear attention allow for efficient parallel training but can simultaneously be formulated as an RNN with 2D (matrix-valued) hidden states, thus enjoying linear-time inference complexity. However, linear attention generally underperforms ordinary softmax attention. Moreover, current implementations of linear attention lack I/O-awareness and are thus slower than highly optimized implementations of softmax attention. This work describes a hardware-efficient algorithm for linear attention that trades off memory movement against parallelizability. The resulting implementation, dubbed FLASHLINEARATTENTION, is faster than FLASHATTENTION-2 (Dao, 2023) as a standalone layer even on short sequence lengths (e.g., 1K). We then generalize this algorithm to a more expressive variant of linear attention with data-dependent gates. When used as a replacement for the standard attention layer in Transformers, the resulting gated linear attention (GLA) Transformer is found to perform competitively against the LLaMA-architecture Transformer (Touvron et al., 2023) as well recent linear-time-inference baselines such as RetNet (Sun et al., 2023a) and Mamba (Gu & Dao, 2023) on moderate-scale language modeling experiments. GLA Transformer is especially effective at length generalization, enabling a model trained on 2K to generalize to sequences longer than 20K without significant perplexity degradations. For training speed, the GLA Transformer has higher throughput than a similarly-sized Mamba model.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2312.06635

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The finite range of the reciprocal rescaling factor (smaller than e^80) remains within BF16 dynamic range, allowing both diagonal and off-diagonal tiles to use dense Tensor Core matrix multiplications and eliminating the position-pair diagonal path.

    “The corresponding reciprocal rescaling factor is therefore smaller than e 80 and remains within the BF16 dynamic range.This finite range allows both diagonal and off-diagonal tiles to use dense Tensor Core matrix multiplications, eliminating the position-pair diagonal path.”

    From Kimi K3: Open Frontier Intelligence (2026) · cites this paper as [140]

    Split into 2 atoms — 0 confirmed, 0 partially supported, 1 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • DeltaNet and Gated DeltaNet use delta-rule key-value state updates.

    “DeltaNet and Gated DeltaNet use delta-rule key-value state updates [52, 50]”

    From Learning What to Remember: Test-Time Training via Context Distillation (2026) · cites this paper as [50]

    The Extended Related Work section says that Schlag et al. (2021) use the delta rule to edit memory dynamically. Our copy does not identify that method as DeltaNet or discuss Gated DeltaNet, so it does not establish that both named models use delta-rule key-value state updates. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

  • Recent variants including DeltaNet, Gated DeltaNet and KDA have narrowed the gap to softmax attention while showing favorable scaling.

    “Recent variants, including DeltaNet [38], Gated DeltaNet [50] and KDA [45], have narrowed the gap to softmax attention while showing favorable scaling.”

    From Learning What to Remember: Test-Time Training via Context Distillation (2026) · cites this paper as [50]

    The Introduction (page unavailable) says this paper’s gated linear attention is competitive with softmax attention and describes linear-time inference and subquadratic training. But the cited paper presents Gated Linear Attention, not Gated DeltaNet; the passages do not establish the claimed performance or scaling for Gated DeltaNet. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The idea of using linear models as hidden states has already been well studied in DeltaNet.

    “The idea of using linear models as hidden states has already been well studied in DeltaNet [62,83].”

    From Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024) · cites this paper as [83]

    Section 2.1 describes linear attention as a recurrent layer with a matrix-valued hidden state that maps queries to outputs. But the paper does not identify DeltaNet or say that this idea was well studied in DeltaNet; searches for “DeltaNet” and “delta rule” found no supporting passage. Page numbers were not available in our copy. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Many other modern RNN layers such as linear attention and DeltaNet are also inspired by the idea of FWPs.

    “Many other modern RNN layers such as linear attention [44,63] and DeltaNet [62,83] are also inspired by the idea of FWPs.”

    From Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024) · cites this paper as [83]

    Section 2.1 describes linear attention as a recurrent layer whose matrix-valued state is updated by an outer product, and the References list a work titled “Linear transformers are secretly fast weight programmers.” This supports a connection between linear attention and FWPs, but our copy does not say linear attention was inspired by FWPs. The citation is attached to linear attention, not DeltaNet. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 80

  1. Arora, S., Eyuboglu, S., Timalsina, A., Johnson, I., Poli, M., Zou, J., Rudra, A., and Ré, C. Zoology: Measuring and improving recall in efficient language models. CoRR, abs/2312.04927, 2023a. · arXiv
  2. Arora, S., Yang, B., Eyuboglu, S., Narayan, A., Hojel, A., Trummer, I., and Ré, C. Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes, April 2023b. URL http: //arxiv.org/abs/2304.09433. arXiv:2304.09433 [cs. · arXiv
  3. Arora, S., Eyuboglu, S., Zhang, M., Timalsina, A., Al- berti, S., Zinsley, D., Zou, J., Rudra, A., and R'e, C. Simple linear attention language models balance the recall-throughput tradeoff. ArXiv, abs/2402.18668, 2024. · arXiv
  4. Ba, J., Hinton, G. E., Mnih, V., Leibo, J. Z., and Ionescu, C. Using fast weights to attend to the recent past. Advances in neural information processing systems, 29, 2016.
  5. Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., and Hochreiter, S. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04517, 2024. · arXiv
  6. Beltagy, I., Peters, M. E., and Cohan, A. Long- former: The long-document transformer. arXiv preprint arXiv: Arxiv-2004.05150, 2020. URL https://arxiv.org/abs/2004.05150v2. · arXiv
  7. Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
  8. Brandon, W., Nrusimha, A., Qian, K., Ankner, Z., Jin, T., Song, Z., and Ragan-Kelley, J. Striped attention: Faster ring attention for causal transformers. ArXiv, abs/2311.09431, 2023. · arXiv
  9. Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. PREPRINT, 2019. URL https://arxiv.org/abs/1904.10509v1. · arXiv
  10. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. · arXiv
  11. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019. · arXiv
  12. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. · arXiv
  13. Dao, T., Chen, B., Sohoni, N. S., Desai, A. D., Poli, M., Grogan, J., Liu, A., Rao, A., Rudra, A., and Ré, C. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference on Machine Learning, 2022a.
  14. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS, 2022b.
  15. Fu, D. Y., Arora, S., Grogan, J., Johnson, I., Eyuboglu, S., Thomas, A. W., Spector, B., Poli, M., Rudra, A., and R'e, C. Monarch mixer: A simple sub-quadratic gemm-based architecture. ArXiv, abs/2310.12109, 2023a. · arXiv
  16. Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Ré, C. Hungry hungry hippos: Towards language model- ing with state space models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b.
  17. Fu, D. Y., Kumbong, H., Nguyen, E., and Ré, C. Flashfftconv: Efficient convolutions for long sequences with tensor cores. CoRR, abs/2311.05908, 2023d. · arXiv
  18. Gers, F. A., Schmidhuber, J., and Cummins, F. A. Learning to forget: Continual prediction with LSTM. Neural Comput., 12(10):2451-2471, 2000.
  19. Gu, A., Goel, K., and R'e, C. Efficiently modeling long sequences with structured state spaces. International Conference On Learning Representations, 2021a.
  20. Gu, A., Johnson, I., Goel, K., Saab, K. K., Dao, T., Rudra, A., and R'e, C. Combining recurrent, convolutional, and continuous-time models with linear state-space layers. Neural Information Processing Systems, 2021b. URL https://arxiv.org/abs/2110.13985v1. · arXiv
  21. Gu, A., Goel, K., and Ré, C. Efficiently modeling long sequences with structured state spaces. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
  22. Hasani, R., Lechner, M., Wang, T.-H., Chahine, M., Amini, A., and Rus, D. Liquid structural state-space models. arXiv preprint arXiv:2209.12951, 2022. · arXiv
  23. Hinton, G. E. and Plaut, D. C. Using fast weights to deblur old memories. In Proceedings of the ninth annual conference of the Cognitive Science Society, pp. 177-186, 1987.
  24. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9(8):1735-1780, 1997.
  25. Hooker, S. The hardware lottery. Communications of the ACM, 64:58 -65, 2020.
  26. Hua, W., Dai, Z., Liu, H., and Le, Q. V. Transformer quality in linear time. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvári, C., Niu, G., and Sabato, S. (eds.), Interna- tional Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9099-9117. PMLR, 2022.
  27. Irie, K., Schlag, I., Csordás, R., and Schmidhuber, J. Going beyond linear transformers with recurrent fast weight programmers. Advances in Neural Information Processing Systems, 34:7703-7717, 2021.
  28. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. ArXiv preprint, abs/2310.06825, 2023. · arXiv
  29. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156-5165. PMLR, 2020.
  30. Katsch, T. Gateloop: Fully data-controlled linear recurrence for sequence modeling. ArXiv, abs/2311.01927, 2023. · arXiv
  31. Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. International Confer- ence On Learning Representations, 2020. URL https://arxiv.org/abs/2001.04451v2. · arXiv
  32. Li, D., Shao, R., Xie, A., Xing, E. P., Gonzalez, J. E., Stoica, I., Ma, X., and Zhang, H. Lightseq: Sequence level parallelism for distributed training of long context transformers. ArXiv, abs/2310.03294, 2023a. · arXiv
  33. Li, S., Xue, F., Baranwal, C., Li, Y., and You, Y. Sequence parallelism: Long sequence training from system perspective. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, July 2023b. Association for Computational Linguistics.
  34. Li, S., You, C., Guruganesh, G., Ainslie, J., Ontanon, S., Zaheer, M., Sanghai, S., Yang, Y., Kumar, S., and Bhojanapalli, S. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310.04418, 2023c. · arXiv
  35. Li, Y., Cai, T., Zhang, Y., Chen, D., and Dey, D. What makes convolutional models great on long sequence modeling? In The Eleventh International Confer- ence on Learning Representations, 2023d. URL https://openreview.net/forum?id=TGJSPbRpJX-.
  36. Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context. ArXiv, abs/2310.01889, 2023. · arXiv
  37. Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024. · arXiv
  38. Ma, J., Li, F., and Wang, B. U-mamba: Enhancing long- range dependency for biomedical image segmentation. arXiv preprint arXiv:2401.04722, 2024. · arXiv
  39. Ma, X., Zhou, C., Kong, X., He, J., Gui, L., Neubig, G., May, J., and Zettlemoyer, L. Mega: Moving average equipped gated attention. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=qNLe3iq2El.
  40. Martin, E. and Cundy, C. Parallelizing linear recurrent neural nets over sequence length. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
  41. Massaroli, S., Poli, M., Fu, D. Y., Kumbong, H., Parnichkun, R. N., Timalsina, A., Romero, D. W., McIntyre, Q., Chen, B., Rudra, A., Zhang, C., Re, C., Ermon, S., and Bengio, Y. Laughing hyena distillery: Extracting compact recurrences from convolutions. NEURIPS, 2023. URL https://arxiv.org/abs/2310.18780v1. · arXiv
  42. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018. · arXiv
  43. Oren, M., Hassid, M., Adi, Y., and Schwartz, R. Transform- ers are multi-state rnns. ArXiv, abs/2401.06104, 2024. · arXiv
  44. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. · arXiv
  45. Peng, B., Goldstein, D., Anthony, Q., Albalak, A., Alcaide, E., Biderman, S., Cheah, E., Ferdinan, T., Hou, H., Kazienko, P., et al. Eagle and finch: Rwkv with matrix- valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892, 2024. · arXiv
  46. Peng, H., Pappas, N., Yogatama, D., Schwartz, R., Smith, N. A., and Kong, L. Random feature attention. arXiv preprint arXiv:2103.02143, 2021. · arXiv
  47. Peng, H., Kasai, J., Pappas, N., Yogatama, D., Wu, Z., Kong, L., Schwartz, R., and Smith, N. A. ABC: Attention with bounded-memory control. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, May 2022. Association for Computational Linguistics.
  48. Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Bac- cus, S., Bengio, Y., Ermon, S., and Ré, C. Hyena hierarchy: Towards larger convolutional language models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 28043-28078. PMLR, 2023.
  49. Pramanik, S., Elelimy, E., Machado, M. C., and White, A. Re- current linear transformers. CoRR, abs/2310.15719, 2023. · arXiv
  50. Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021. · arXiv
  51. Qin, Z., Han, X., Sun, W., Li, D., Kong, L., Barnes, N., and Zhong, Y. The devil in linear transformer. arXiv preprint arXiv:2210.10340, 2022. · arXiv
  52. Qin, Z., Han, X., Sun, W., He, B., Li, D., Li, D., Dai, Y., Kong, L., and Zhong, Y. Toeplitz neural network for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=IxmWsm4xrua.
  53. Qin, Z., Li, D., Sun, W., Sun, W., Shen, X., Han, X., Wei, Y., Lv, B., Yuan, F., Luo, X., et al. Scaling transnormer to 175 billion parameters. arXiv preprint arXiv:2307.14995, 2023b. · arXiv
  54. Qin, Z., Yang, S., Sun, W., Shen, X., Li, D., Sun, W., and Zhong, Y. Hgrn2: Gated linear rnns with state expansion. arXiv preprint arXiv:2404.07904, 2024b. · arXiv
  55. Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. P. Compressive transformers for long-range sequence modelling. arXiv preprint, 2019.
  56. Ren, L., Liu, Y., Wang, S., Xu, Y., Zhu, C., and Zhai, C. Sparse modular activation for efficient sequence modeling. In Thirty-seventh Conference on Neu- ral Information Processing Systems, 2023. URL https://openreview.net/forum?id=TfbzX6I14i.
  57. Roemmele, M., Bejan, C. A., and Gordon, A. S. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011. URL https://people.ict.usc.edu/ ∼ gordon/ publications/AAAI-SPRING11A.PDF.
  58. Romero, D. W., Kuzina, A., Bekkers, E. J., Tom- czak, J. M., and Hoogendoorn, M. Ckconv: Con- tinuous kernel convolution for sequential data. arXiv preprint arXiv: 2102.02611, 2021. URL https://arxiv.org/abs/2102.02611v3. · arXiv
  59. Roy, A., Saffar, M., Vaswani, A., and Grangier, D. Efficient content-based sparse attention with routing transformers. International Conference On Topology, Algebra And Categories In Logic, 2020. doi: 10.1162/tacl a 00353. URL https://arxiv.org/abs/2003.05997v5. · DOI
  60. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
  61. Saphra, N., Fleisig, E., Cho, K., and Lopez, A. First tragedy, then parse: History repeats itself in the new era of large language models. ArXiv, abs/2311.05020, 2023. · arXiv
  62. Schlag, I., Irie, K., and Schmidhuber, J. Linear transformers are secretly fast weight programmers. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Ma- chine Learning Research, pp. 9355-9366. PMLR, 2021.
  63. Schmidhuber, J. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139, 1992.
  64. Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. · arXiv
  65. Smith, J. T. H., Warrington, A., and Linderman, S. W. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
  66. Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. CoRR, abs/2104.09864, 2021. · arXiv
  67. Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023a. · arXiv
  68. van der Westhuizen, J. and Lasenby, J. The unreasonable effectiveness of the forget gate. CoRR, abs/1804.04849, 2018. · arXiv
  69. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  70. Wang, C., Tsepa, O., Ma, J., and Wang, B. Graph-mamba: To- wards long-range graph sequence modeling with selective state spaces. arXiv preprint arXiv:2402.00789, 2024a. · arXiv
  71. Wang, J., Yan, J. N., Gu, A., and Rush, A. M. Pretraining without attention. CoRR, abs/2212.10544, 2022. · arXiv
  72. Wang, J., Gangavarapu, T., Yan, J. N., and Rush, A. M. Mambabyte: Token-free selective state space model. arXiv preprint arXiv:2401.13660, 2024b. · arXiv
  73. Wu, F., Fan, A., Baevski, A., Dauphin, Y., and Auli, M. Pay less attention with lightweight and dynamic convolutions. International Conference on Learning Representations, 2019. URL https://arxiv.org/abs/1901.10430v2. · arXiv
  74. Xing, Z., Ye, T., Yang, Y., Liu, G., and Zhu, L. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv preprint arXiv:2401.13560, 2024. · arXiv
  75. Yang, Y., Xing, Z., and Zhu, L. Vivim: a video vision mamba for medical video object segmentation. arXiv preprint arXiv:2401.14168, 2024. · arXiv
  76. Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297, 2020. URL https://arxiv.org/abs/2007.14062v2. · arXiv
  77. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. · arXiv
  78. Zhang, B. and Sennrich, R. Root mean square layer nor- malization. Advances in Neural Information Processing Systems, 32, 2019.
  79. Zhang, Y. and Cai, D. Linearizing transformer with key-value memory. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  80. Zhu, L., Liao, B., Zhang, Q., Wang, X., Liu, W., and Wang, X. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024. · arXiv

Cited by

Sorted by citations

Related

Metadata

Authors
Yang, S., Wang, B., Shen, Y., Panda, R., & Kim, Y.
Affiliations
IBM (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2312.06635
arXiv
2312.06635
OpenAlex
W4389713725
Cited by
10
References
0
Citations by year
2025: 7 · 2024: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
linear attention, parallel training, linear-time inference, length generalization, language modeling, Mamba, RetNet

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review