CoChat
Sign Up
Cited by 5Mathematical Analysis and Transform MethodsNeural Networks and ApplicationsComputability, Logic, AI Algorithms

Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutions

Open access
Massaroli, Stefano, Poli, Michael, Fu, Dan, Kumbong, Hermann, Parnichkun, Rom, Romero, David, Timalsina, Aman, McIntyre, Quinn +6 more·2023·Advances in Neural Information Processing Systems 36
DOI 10.52202/075280-0747
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advances in attention-free sequence models rely on convolutions as alternatives to the attention operator at the core of Transformers. In particular, long convolution sequence models have achieved state-of-the-art performance in many domains, but incur a significant cost during auto-regressive inference workloads -- naively requiring a full pass (or caching of activations) over the input sequence for each generated token -- similarly to attention-based models. In this paper, we seek to enable $\mathcal O(1)$ compute and memory cost per token in any pre-trained long convolution architecture to reduce memory footprint and increase throughput during generation. Concretely, our methods consist in extracting low-dimensional linear state-space models from each convolution layer, building upon rational interpolation and model-order reduction techniques. We further introduce architectural improvements to convolution-based layers such as Hyena: by weight-tying the filters across channels into heads, we achieve higher pre-training quality and reduce the number of filters to be distilled. The resulting model achieves 10x higher throughput than Transformers and 1.5x higher than Hyena at 1.3B parameters, without any loss in quality after distillation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/075280-0747

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Evo gains lower latency and memory cost from the fast recurrent mode of its hyena layers.

    “Evo benefits from the fast recurrent mode of hyena layers, enabling lower latency and memory cost (24, 28).”

    From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [28]

    The Introduction (pp. 1–2) says recurrent inference can use constant per-step time and memory, and reports lower memory use and latency gains for LaughingHyena-distilled models. It does not establish that Evo itself achieves those gains; its results concern distilled long-convolution models, not Evo. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 56

  1. [1][1] Daniel Y Fu et al. 'Hungry Hungry Hippos: Towards Language Modeling with State Space Models'. In: 2023 (cit. on pp. 1-3, 7, 23). -
  2. [2][2] Michael Poli et al. 'Hyena Hierarchy: Towards Larger Convolutional Language Models'. In: (2023). arXiv: 2302.10866 (cit. on pp. 1, 3, 7, 8, 31, 35, 43). - · arXiv
  3. [3][3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 'Neural machine translation by jointly learning to align and translate'. In: (2014). arXiv: 1409.0473 (cit. on p. 1). - · arXiv
  4. [4][4] Ashish Vaswani et al. 'Attention is all you need'. In: Advances in neural information processing systems 30 (2017) (cit. on p. 1). -
  5. [5][5] Chi-Tsong Chen. Linear system theory and design . Saunders college publishing, 1984 (cit. on pp. 1, 6, 19, 21, 22). -
  6. [6][6] Albert Gu, Karan Goel, and Christopher Ré. 'Efficiently modeling long sequences with structured state spaces'. In: (2021). arXiv: 2111.00396 (cit. on pp. 1, 3, 4, 6, 20, 25). - · arXiv
  7. [7][7] Vadim Movsesovich Adamyan, Damir Zyamovich Arov, and Mark Grigor'evich Krein. 'Analytic properties of Schmidt pairs for a Hankel operator and the generalized Schur-Takagi problem'. In: Matematicheskii Sbornik 128.1 (1971), pp. 34-75 (cit. on pp. 2, 7, 29). -
  8. [8][8] Yuji Nakatsukasa, Olivier Sète, and Lloyd N Trefethen. 'The AAA algorithm for rational approximation'. In: SIAM Journal on Scientific Computing 40.3 (2018), A1494-A1522 (cit. on pp. 2, 6). -
  9. [9][9] GRB Prony. 'Essai experimental et analytique sur les lois de la dilatalrlite de fluids elastiques et sur cells de la vapeur de l'alcool, à différents tempoeratures'. In: Journal de l'Ecole Polytechnique (París) 1 (1795), pp. 24-76 (cit. on pp. 2, 6). -
  10. [11][11] Leo Gao et al. 'The pile: An 800gb dataset of diverse text for language modeling'. In: (2020). arXiv: 2101.00027 (cit. on pp. 2, 8, 35). - · arXiv
  11. [12][12] Albert Gu et al. 'Hippo: Recurrent memory with optimal polynomial projections'. In: Advances in Neural Information Processing Systems 33 (2020), pp. 1474-1487 (cit. on pp. 3, 25). -
  12. [13][13] David W Romero et al. 'Ckconv: Continuous kernel convolution for sequential data'. In: (2021). arXiv: 2102.02611 (cit. on p. 3). - · arXiv
  13. [14][14] David W Romero et al. 'Flexconv: Continuous kernel convolutions with differentiable kernel sizes'. In: (2021). arXiv: 2110.08059 (cit. on p. 3). - · arXiv
  14. [15][15] Vincent Sitzmann et al. 'Implicit neural representations with periodic activation functions'. In: Advances in neural information processing systems 33 (2020), pp. 7462-7473 (cit. on p. 3). -
  15. [16][16] Rizal Fathony et al. 'Multiplicative filter networks'. In: International Conference on Learning Representations . 2020 (cit. on p. 3). -
  16. [17][17] Daniel Y. Fu et al. 'Simple Hardware-Efficient Long Convolutions for Sequence Modeling'. In: International Conference on Machine Learning (2023) (cit. on pp. 3, 25). -
  17. [18][18] Kemin Zhou and John Comstock Doyle. Essentials of robust control . Vol. 104. Prentice hall Upper Saddle River, NJ, 1998 (cit. on p. 4). -
  18. [19][19] Athanasios C Antoulas. Approximation of large-scale dynamical systems . SIAM, 2005 (cit. on pp. 4, 7, 29). -
  19. [20][20] Wilhelmus HA Schilders, Henk A Van der Vorst, and Joost Rommes. Model order reduction: theory, research aspects and applications . Vol. 13. Springer, 2008 (cit. on p. 4). -
  20. [21][21] Sun-Yuan Kung. 'A new identification and model reduction algorithm via singular value decomposition'. In: Proc. 12th Asilomar Conf. on Circuits, Systems and Computer . 1978, pp. 705-714 (cit. on p. 4). -
  21. [22][22] D Friedman. 'On approximating an FIR filter using discrete orthonormal exponentials'. In: IEEE Transactions on Acoustics, Speech, and Signal Processing 29.4 (1981), pp. 923-926 (cit. on p. 4). -
  22. [23][23] J Bednar. 'On the approximation of FIR by IIR digital filters'. In: IEEE Transactions on Acoustics, Speech, and Signal Processing 31.1 (1983), pp. 28-34 (cit. on p. 4). -
  23. [24][24] Bartlomiej Beliczynski, Izzet Kale, and Gerald D Cain. 'Approximation of FIR by IIR digital filters: An algorithm based on balanced model reduction'. In: IEEE Transactions on Signal Processing 40.3 (1992), pp. 532-542 (cit. on pp. 4, 43, 44). -
  24. [25][25] William Henry Young. 'On the multiplication of successions of Fourier constants'. In: Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character 87.596 (1912), pp. 331-339 (cit. on p. 5). -
  25. [26][26] William Beckner. 'Inequalities in Fourier analysis on Rn'. In: Proceedings of the National Academy of Sciences 72.2 (1975), pp. 638-641 (cit. on p. 5). -
  26. [27][27] John Fournier. 'Sharpness in Young's inequality for convolution'. In: Pacific Journal of Mathematics 72.2 (1977), pp. 383-397 (cit. on p. 5). -
  27. [28][28] Tong S Quek and Leonard YH Yap. 'Sharpness of Young's inequality for convolution'. In: Mathematica Scandinavica 53.2 (1983), pp. 221-237 (cit. on p. 5). -
  28. [29][29] Jean-Paul Berrut and Lloyd N Trefethen. 'Barycentric lagrange interpolation'. In: SIAM review 46.3 (2004), pp. 501-517 (cit. on p. 6). -
  29. [30][30] Marcos Vicente Moreira and João Carlos Basilio. 'Fair and Square Computation of Inverse Z-Transforms of Rational Functions'. In: IEEE Transactions on Education 55.2 (2011), pp. 285-290 (cit. on p. 6). -
  30. [31][31] Henri Padé. 'Sur la représentation approchée d'une fonction par des fractions rationnelles'. In: Annales scientifiques de l'Ecole normale supérieure . Vol. 9. 1892, pp. 3-93 (cit. on pp. 6, 27). -
  31. [32][32] Ankit Gupta, Albert Gu, and Jonathan Berant. 'Diagonal state spaces are as effective as structured state spaces'. In: Advances in Neural Information Processing Systems 35 (2022), pp. 22982-22994 (cit. on pp. 6, 25). -
  32. [33][33] Antonio Orvieto et al. 'Resurrecting Recurrent Neural Networks for Long Sequences'. In: (2023). arXiv: 2303.06349 (cit. on pp. 6, 25, 26). - · arXiv
  33. [34][34] Jeffrey M Hokanson. 'A data-driven McMillan degree lower bound'. In: SIAM Journal on Scientific Computing 42.5 (2020), A3447-A3461 (cit. on p. 6). -
  34. [35][35] L Ho and Rudolf E Kalman. 'Effective construction of linear state-variable models from input/output functions'. In: at-Automatisierungstechnik 14.1-12 (1966), pp. 545-548 (cit. on p. 6). -
  35. [36][36] Carl Eckart and Gale Young. 'The approximation of one matrix by another of lower rank'. In: Psychometrika 1.3 (1936), pp. 211-218 (cit. on p. 7). -
  36. [38][38] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. 'Simplified state space layers for sequence modeling'. In: (2022). arXiv: 2208.04933 (cit. on p. 7). - · arXiv
  37. [39][39] Noam Shazeer. 'Fast transformer decoding: One write-head is all you need'. In: (2019). arXiv: 1911. 02150 (cit. on p. 8). -
  38. [40][40] Joshua Ainslie et al. 'GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints'. In: (2023). arXiv: 2305.13245 (cit. on p. 8). - · arXiv
  39. [41][41] Percy Liang et al. 'Holistic evaluation of language models'. In: (2022). arXiv: 2211.09110 (cit. on pp. 8, 9, 36). - · arXiv
  40. [43][43] Ari Holtzman et al. 'The curious case of neural text degeneration'. In: (2019). arXiv: 1904.09751 (cit. on p. 9). - · arXiv
  41. [44][44] Stella Biderman et al. 'Pythia: A suite for analyzing large language models across training and scaling'. In: (2023). arXiv: 2304.01373 (cit. on pp. 9, 36). - · arXiv
  42. [45][45] Yuntao Bai et al. 'Constitutional AI: Harmlessness from AI Feedback'. In: (2022). arXiv: 2212.08073 (cit. on p. 11). - · arXiv
  43. [46][46] Alan V Oppenheim. Discrete-time signal processing . Pearson Education India, 1999 (cit. on p. 21). -
  44. [47][47] Lennart Ljung. System identification . Springer, 1998 (cit. on p. 21). -
  45. [48][48] RP Guidorzi. 'Certain models from uncertain data: the algebraic case'. In: Systems & control letters 17.6 (1991), pp. 415-424 (cit. on p. 21). -
  46. [49][49] Irwin W Sandberg. 'On the theory of linear multi-loop feedback systems'. In: Bell System Technical Journal 42.2 (1963), pp. 355-382 (cit. on p. 23). -
  47. [50][50] Michael Zhang et al. 'Effectively Modeling Time Series with Simple Discrete State Spaces'. In: (2023). arXiv: 2303.09489 (cit. on p. 25). - · arXiv
  48. [51][51] Luca Perotti and Michał Wojtylak. 'Matrix methods for Padé approximation: Numerical calculation of poles, zeros and residues'. In: Linear Algebra and its Applications 548 (2018), pp. 95-122 (cit. on p. 27). -
  49. [52][52] Amer Abu-Omar and Fuad Kittaneh. 'Estimates for the numerical radius and the spectral radius of the Frobenius companion matrix and bounds for the zeros of polynomials'. In: Annals of Functional Analysis 5.1 (2014), pp. 56-62 (cit. on p. 27). -
  50. [53][53] Julia Eaton et al. 'Polynomial root radius optimization with affine constraints'. In: Mathematical Programming 165 (2017), pp. 509-528 (cit. on p. 27). -
  51. [54][54] Gerlind Plonka and Vlada Pototskaia. 'Application of the AAK theory for sparse approximation of exponential sums'. In: (2016). arXiv: 1609.09603 (cit. on p. 29). - · arXiv
  52. [55][55] Jimmy Ba et al. 'Using fast weights to attend to the recent past'. In: Advances in neural information processing systems 29 (2016) (cit. on p. 31). -
  53. [56][56] William B Johnson. 'Extensions of Lipschitz mappings into a Hilbert space'. In: Contemp. Math. 26 (1984), pp. 189-206 (cit. on p. 31). -
  54. [57][57] Devdatt P Dubhashi and Alessandro Panconesi. Concentration of measure for the analysis of randomized algorithms . Cambridge University Press, 2009 (cit. on p. 34). -
  55. [58][58] Ilya Loshchilov and Frank Hutter. 'Decoupled weight decay regularization'. In: (2017). arXiv: 1711. 05101 (cit. on p. 35). -
  56. [59][59] Dale F. Enns. 'Model reduction with balanced realizations: An error bound and a frequency weighted generalization'. In: The 23rd IEEE Conference on Decision and Control (1984), pp. 127-132 (cit. on p. 43).

Cited by

Sorted by citations

Related

Authors
Massaroli, S., Poli, M., Fu, D., Kumbong, H., Parnichkun, R., Romero, D., Timalsina, A., McIntyre, Q., Chen, B., Rudra, A., Zhang, C., Ré, C., Ermon, S., & Bengio, Y.
ORCID
Venue
Advances in Neural Information Processing Systems 36
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published
2023
Pages
17072-17116
ISSN
1049-5258 (ISSN-L)
DOI
10.52202/075280-0747
OpenAlex
W7133193680
Cited by
4
References
0
Citations by year
2025: 1 · 2024: 3
FWCI
4.46
Type
conference paper
Language
en
Open access
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • VU Research Portal (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Mathematics › Applied Mathematics › Mathematical Analysis and Transform Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Computability, Logic, AI Algorithms
Keywords
convolution, convolutional neural networks

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review