CoChat
Sign Up
PreprintCited by 1Neural Networks and ApplicationsTopic ModelingAdvanced Adaptive Filtering Techniques

Focus Your Attention (with Adaptive IIR Filters)

Open access
Lutati, Shahar, Zimerman, Itamar, Wolf, Lior·2023·arXiv
DOI 10.48550/arxiv.2305.14952arXiv 2305.14952
Open this paper in CoChat — free accountOpen PDF

Abstract

We present a new layer in which dynamic (i.e.,input-dependent) Infinite Impulse Response (IIR) filters of order two are used to process the input sequence prior to applying conventional attention. The input is split into chunks, and the coefficients of these filters are determined based on previous chunks to maintain causality. Despite their relatively low order, the causal adaptive filters are shown to focus attention on the relevant sequence elements. The new layer is grounded in control theory, and is shown to generalize diagonal state-space layers. The layer performs on-par with state-of-the-art networks, with a fraction of their parameters and with time complexity that is sub-quadratic with input size. The obtained layer is favorable to layers such as Heyna, GPT2, and Mega, both with respect to the number of parameters and the obtained level of performance on multiple long-range sequence problems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.14952

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Beyond selective SSMs and gated RNNs, other examples may include input-dependent convolutions and even attention.

    “Beyond selective SSMs and gated RNNs, other examples may include input-dependent convolutions (Kosma, Nikolentzos, and Vazirgiannis 2023; Lioutas and Guo 2020; Lutati, Zimerman, and Wolf 2023; Yang et al. 2019) and even attention.”

    From Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) · cites this paper as Lutati, Zimerman, and Wolf 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. Arij Al Adel. 2022. Global memory transformer for pro- cessing long documents. In Advances in Neural Com- putation, Machine Learning, and Cognitive Research VI: Selected Papers from the XXIV International Con- ference on Neuroinformatics, October 17-21, 2022, Moscow, Russia, pages 343-352. Springer.
  2. Nuha A. S. Alwan and Zahir M. Hussain. 2022. Deep learning for robust adaptive inverse control of nonlin- ear dynamic systems: Improved settling time with an autoencoder. Sensors, 22(16). · DOI
  3. Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. 2016. Learning feed-forward one-shot learners. In Advances in Neu- ral Information Processing Systems, pages 523-531.
  4. Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. 2023. Scaling transformer to 1m tokens and beyond with rmt. arXiv preprint arXiv:2304.11062. · arXiv
  5. Oscar Chang, Lampros Flokas, and Hod Lipson. 2020. Principled weight initialization for hypernetworks. In Int. Conf. on Learning Representations.
  6. Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long se- quences with sparse transformers. arXiv preprint arXiv:1904.10509. · arXiv
  7. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language mod- els beyond a fixed-length context. arXiv preprint arXiv:1901.02860. · arXiv
  8. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022a. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359.
  9. Tri Dao, Daniel Y Fu, Khaled K Saab, Armin W Thomas, Atri Rudra, and Christopher Ré. 2022b. Hungry hungry hippos: Towards language mod- eling with state space models. arXiv preprint arXiv:2212.14052. · arXiv
  10. Shmuel Bar David, Itamar Zimerman, Eliya Nach- mani, and Lior Wolf. 2023. Decision s4: Efficient sequence-based rl via state spaces layers. In The Eleventh International Conference on Learning Rep- resentations.
  11. Quentin Fournier, Gaétan Marceau Caron, and Daniel Aloise. 2021. A practical survey on faster and lighter transformers. ACM Computing Surveys.
  12. Daniel Y Fu, Elliot L Epstein, Eric Nguyen, Armin W Thomas, Michael Zhang, Tri Dao, Atri Rudra, and Christopher Ré. 2023. Simple hardware-efficient long convolutions for sequence modeling. arXiv preprint arXiv:2302.06646. · arXiv
  13. Xavier Glorot and Yoshua Bengio. 2010. Understand- ing the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth Interna- tional Conference on Artificial Intelligence and Statis- tics, volume 9 of Proceedings of Machine Learning Research, pages 249-256, Chia Laguna Resort, Sar- dinia, Italy. PMLR.
  14. Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. 2022. It's raw! audio generation with state-space models. In International Conference on Machine Learning, pages 7616-7633. PMLR.
  15. Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. 2022. On the parameterization and initialization of diagonal state space models. Advances in Neural Information Processing Systems, 35:35971-35983.
  16. Albert Gu, Karan Goel, and Christopher Ré. 2021a. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396. · arXiv
  17. Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. 2021b. Com- bining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572-585.
  18. Ankit Gupta and Jonathan Berant. 2020. Gmat: Global memory augmentation for transformers. arXiv preprint arXiv:2006.03274. · arXiv
  19. Ankit Gupta, Albert Gu, and Jonathan Berant. 2022a. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Pro- cessing Systems, 35:22982-22994.
  20. Ankit Gupta, Harsh Mehta, and Jonathan Berant. 2022b. Simplifying and understanding state space models with diagonal linear rnns. arXiv preprint arXiv:2212.00768. · arXiv
  21. David Ha, Andrew Dai, and Quoc V Le. 2016. Hyper- networks. arXiv preprint arXiv:1609.09106. · arXiv
  22. Yun He, Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Don- ald Metzler, et al. 2022. Hyperprompt: Prompt-based task-conditioning of transformers. In International Conference on Machine Learning, pages 8678-8690.
  23. PMLR.
  24. DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. 2022. Block-recurrent transformers. arXiv preprint arXiv:2203.07852. · arXiv
  25. Md Mohaiminul Islam, Mahmudul Hasan, Kis- han Shamsundar Athrey, Tony Braskich, and Gedas Bertasius. 2022. Efficient movie scene detection using state-space transformers. arXiv preprint arXiv:2212.14427. · arXiv
  26. Maor Ivgi, Uri Shaham, and Jonathan Berant. 2023. Ef- ficient long-text understanding with short-text mod- els. Transactions of the Association for Computa- tional Linguistics, 11:284-299.
  27. Boris Kuznetsov, Julian D Parker, and Fabián Esqueda. 2020. Differentiable iir filters for machine learning applications. In Proc. Int. Conf. Digital Audio Effects (eDAFx-20), pages 297-303.
  28. Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadeepta Dey. 2022. What makes convolutional models great on long sequence modeling? arXiv preprint arXiv:2210.09298. · arXiv
  29. Shahar Shlomo Lutati and Lior Wolf. 2023. Ocd: Learn- ing to overfit with conditional diffusion models. In ICML.
  30. Xuezhe Ma, Chunting Zhou, Xiang Kong, Junxian He, Liangke Gui, Graham Neubig, Jonathan May, and Luke Zettlemoyer. 2022. Mega: moving av- erage equipped gated attention. arXiv preprint arXiv:2209.10655. · arXiv
  31. Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. 2022. Long range language modeling via gated state spaces. arXiv preprint arXiv:2206.13947. · arXiv
  32. Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. 2023. Resurrecting recurrent neu- ral networks for long sequences. arXiv preprint arXiv:2303.06349. · arXiv
  33. Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Ste- fano Ermon, and Christopher Ré. 2023. Hyena hierar- chy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866. Pytorch. Adaptivemaxpool2d ¶. · arXiv
  34. Guy Revach, Nir Shlezinger, Xiaoyong Ni, Adria Lopez Escoriza, Ruud J. G. van Sloun, and Yonina C. Eldar. 2022. KalmanNet: Neural network aided kalman filtering for partially known dynamics. IEEE Trans- actions on Signal Processing, 70:1532-1547. · DOI
  35. David W Romero, Anna Kuzina, Erik J Bekkers, Jakub M Tomczak, and Mark Hoogendoorn. 2021. Ckconv: Continuous kernel convolution for sequen- tial data. arXiv preprint arXiv:2102.02611. · arXiv
  36. George Saon, Ankit Gupta, and Xiaodong Cui. 2023. Diagonal state space augmented transformers for speech recognition. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
  37. Filip Szatkowski, Karol J Piczak, Przemysław Spurek, Jacek Tabor, and Tomasz Trzciński. 2022. Hyper- sound: Generating implicit neural representations of audio signals with hypernetworks. arXiv preprint arXiv:2211.01839. · arXiv
  38. Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2020. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006. · arXiv
  39. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Met- zler. 2022. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-28.
  40. Ali Vardasbi, Telmo Pires, Robin M. Schmidt, and Stephan Peitz. 2023. State spaces aren't enough: Machine translation needs attention. ArXiv, abs/2304.12776. · arXiv
  41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  42. Johannes Von Oswald, Christian Henning, Benjamin F Grewe, and João Sacramento. 2019. Contin- ual learning with hypernetworks. arXiv preprint arXiv:1906.00695. · arXiv
  43. Jue Wang, Wentao Zhu, Pichao Wang, Xiang Yu, Linda Liu, Mohamed Omar, and Raffay Hamid. 2023. Se- lective structured state-spaces for long-form video understanding. arXiv preprint arXiv:2303.14526. · arXiv
  44. Junxiong Wang, Jing Nathan Yan, Albert Gu, and Alexander M Rush. 2022. Pretraining without at- tention. arXiv preprint arXiv:2212.10544. · arXiv
  45. Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768. · arXiv
  46. Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, and Pascale Fung. 2020. Lightweight and efficient end-to-end speech recognition us- ing low-rank transformer. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6144-6148. IEEE.
  47. Wenhan Xiong, Barlas Oguz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Wen-tau Yih, and Yashar Mehdad. 2021. Simple local attentions remain competitive for long-context tasks. arXiv preprint arXiv:2112.07210. · arXiv
  48. Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. Megabyte: Predicting million-byte se- quences with multiscale transformers. arXiv preprint arXiv:2305.07185. · arXiv
  49. Chris Zhang, Mengye Ren, and Raquel Urtasun. 2019. Graph hypernetworks for neural architecture search. In 7th International Conference on Learning Repre- sentations, ICLR 2019.
  50. Hao Zhang and DeLiang Wang. 2021. Deep anc: A deep learning approach to active noise control. Neu- ral Networks, 141:1-10. · DOI
  51. Michael Zhang, Khaled K Saab, Michael Poli, Tri Dao, Karan Goel, and Christopher Ré. 2023. Ef- fectively modeling time series with simple discrete state spaces. arXiv preprint arXiv:2303.09489. · arXiv
  52. Qingyi Zhou, Fan Zhang, and Chuanchuan Yang. 2020. AdaNN: Adaptive neural network-based equalizer via online semi-supervised learning. Journal of Light- wave Technology, 38(16):4315-4324. · DOI
  53. Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. Fedformer: Fre- quency enhanced decomposed transformer for long- term series forecasting. In International Conference on Machine Learning, pages 27268-27286.
  54. PMLR.
  55. Simiao Zuo, Xiaodong Liu, Jian Jiao, Denis Charles, Eren Manavoglu, Tuo Zhao, and Jianfeng Gao. 2022. Efficient long sequence modeling via state space augmented transformer. arXiv preprint arXiv:2212.08136. · arXiv

Cited by

Sorted by citations

Authors
Lutati, S., Zimerman, I., & Wolf, L.
Affiliations
Tel Aviv University (IL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
12538-12549
DOI
10.48550/arxiv.2305.14952
arXiv
2305.14952
OpenAlex
W4389520351
Cited by
0
References
1
FWCI
0.00
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Tel Aviv University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Engineering › Computational Mechanics › Advanced Adaptive Filtering Techniques
Keywords
adaptive IIR filters, attention mechanism, causal filters, control theory, parameter efficiency, time complexity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review