CoChat
Sign Up
PreprintCited by 1Time Series Analysis and ForecastingForecasting Techniques and ApplicationsStock Market Forecasting Methods

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization

Open access
Masserano, Luca, Ansari, Abdul Fatir, Han, Boran, Zhang, Xiyuan, Faloutsos, Christos, Mahoney, Michael W., Wilson, Andrew Gordon, Park, Youngsuk, Rangapuram, Syama, Maddix, Danielle C., Wang, Yuyang·2024·arXiv
DOI 10.48550/arxiv.2412.05244arXiv 2412.05244
Open this paper in CoChat — free accountOpen PDF

Abstract

How to best develop foundational models for time series forecasting remains an important open question. Tokenization is a crucial consideration in this effort: what is an effective discrete vocabulary for a real-valued sequential input? To address this question, we develop WaveToken, a wavelet-based tokenizer that allows models to learn complex representations directly in the space of time-localized frequencies. Our method first scales and decomposes the input time series, then thresholds and quantizes the wavelet coefficients, and finally pre-trains an autoregressive model to forecast coefficients for the forecast horizon. By decomposing coarse and fine structures in the inputs, wavelets provide an eloquent and compact language for time series forecasting that simplifies learning. Empirical results on a comprehensive benchmark, including 42 datasets for both in-domain and zero-shot settings, show that WaveToken: i) provides better accuracy than recently proposed foundation models for forecasting while using a much smaller vocabulary (1024 tokens), and performs on par or better than modern deep learning models trained specifically on each dataset; and ii) exhibits superior generalization capabilities, achieving the best average rank across all datasets for three complementary metrics. In addition, we show that our method can easily capture complex temporal patterns of practical relevance that are challenging for other recent pre-trained models, including trends, sparse spikes, and non-stationary time series with varying frequencies evolving over time.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2412.05244

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. Felix Abramovich and Yoav Benjamini. Adaptive thresholding of wavelet coefficients. Computational Statistics & Data Analysis , 22(4):351-361, 1996. 5, 16
  2. Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al. Gluonts: Probabilistic and neural time series modeling in python. Journal of Machine Learning Research , 21(116):1-6, 2020. 21
  3. Zaid Alyafeai, Maged S Al-shaibani, Mustafa Ghaleb, and Irfan Ahmad. Evaluating various tokenizers for arabic text classification. Neural Processing Letters , 55(3):2911-2933, 2023. 3
  4. Abdul Fatir Ansari, Lorenzo Stella, Caner Turkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda Arango, Shubham Kapoor, et al. Chronos: Learning the language of time series. arXiv preprint arXiv:2403.07815 , 2024. 1, 2, 3, 5, 6, 7, 10, 21, 25 · arXiv
  5. Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Bohlke-Schneider, David Salinas, Lorenzo Stella, et al. Deep learning for time series forecasting: Tutorial and literature survey. ACM Computing Surveys , 55(6):1-36, 2022. 1, 5
  6. S Grace Chang, Bin Yu, and Martin Vetterli. Adaptive wavelet thresholding for image denoising and compression. IEEE transactions on image processing , 9(9):1532-1546, 2000. 5
  7. Seokmin Choi, Sajad Mousavi, Phillip Si, Haben G Yhdego, Fatemeh Khadem, and Fatemeh Afghah. Ecgbert: Understanding hidden language of ecgs with self-supervised representation learning. arXiv preprint arXiv:2306.06340 , 2023. 3 · arXiv
  8. Charilaos Christopoulos, Athanassios Skodras, and Touradj Ebrahimi. The jpeg2000 still image coding system: an overview. IEEE transactions on consumer electronics , 46(4):1103-1127, 2000. 5, 9
  9. Gautier Dagan, Gabriel Synnaeve, and Baptiste Roziere. Getting the most out of your tokenizer for pre-training and domain adaptation. In Proceedings of the 41st International Conference on Machine Learning , volume 235, pp. 9784-9805. PMLR, 2024. 1
  10. Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688 , 2023. 1, 2, 3, 7, 21 · arXiv
  11. David L Donoho. De-noising by soft-thresholding. IEEE transactions on information theory , 41(3): 613-627, 1995. 5, 10
  12. David Freedman and Persi Diaconis. On the histogram as a density estimator: L 2 theory. Zeitschrift f¨ ur Wahrscheinlichkeitstheorie und verwandte Gebiete , 57(4):453-476, 1981. 6
  13. Philip Gage. A new algorithm for data compression. The C Users Journal , 12(2):23-38, 1994. 3
  14. Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, et al. xval: A continuous number encoding for large language models. arXiv preprint arXiv:2310.02989 , 2023. 3 · arXiv
  15. Tobias Golling, Lukas Heinrich, Michael Kagan, Samuel Klein, Matthew Leigh, Margarita Osadchy, and John Andrew Raine. Masked particle modeling on sets: Towards self-supervised high energy physics foundation models. Machine Learning: Science and Technology , 5(3):035074, 2024. 1
  16. Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03885 , 2024. 3 · arXiv
  17. Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew G Wilson. Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems , 36, 2024. 3
  18. Florentin Guth, Simon Coste, Valentin De Bortoli, and Stephane Mallat. Wavelet score-based generative modeling. Advances in neural information processing systems , 35:478-491, 2022. 3
  19. Alfred Haar. Zur theorie der orthogonalen funktionensysteme. Mathematische Annalen , 71(1): 38-53, 1911. 15
  20. Rob J Hyndman and Anne B Koehler. Another look at measures of forecast accuracy. International journal of forecasting , 22(4):679-688, 2006. 7, 17
  21. Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728 , 2023. 3 · arXiv
  22. Roger Koenker and Kevin F Hallock. Quantile regression. Journal of economic perspectives , 15(4): 143-156, 2001. 17
  23. Pedro Lara-Ben´ ıtez, Manuel Carranza-Garc´ ıa, and Jos´ e C Riquelme. An experimental review on deep learning architectures for time series forecasting. International journal of neural systems , 31(03):2130001, 2021. 1
  24. Vincent Le Guen and Nicolas Thome. Shape and time distortion loss for training deep time series forecasting models. Advances in neural information processing systems , 32, 2019. 6
  25. Bryan Lim, Sercan ¨ O Arık, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting , 37(4): 1748-1764, 2021. 7
  26. St´ ephane Mallat. A Wavelet Tour of Signal Processing: The Sparse Way . Academic Press, 3rd edition, 2009. 2, 4, 9, 15, 16
  27. Stephane G Mallat. Multiresolution approximations and wavelet orthonormal bases of L 2 ( R ) . Transactions of the American mathematical society , 315(1):69-87, 1989. 16
  28. Wael Mattar, Idan Levy, Nir Sharon, and Shai Dekel. Wavelets are all you need for autoregressive image generation. arXiv preprint arXiv:2406.19997 , 2024. 3 · arXiv
  29. Yves Meyer. Wavelets and operators: volume 1 . Cambridge university press, 1992. 16
  30. Suvir Mirchandani, Fei Xia, Pete Florence, Brian Ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. arXiv preprint arXiv:2307.04721 , 2023. 1 · arXiv
  31. Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 , 2022. 7 · arXiv
  32. Alan V. Oppenheim and Ronald W. Schafer. Discrete-Time Signal Processing . Pearson, 2010. 15
  33. Spiros Papadimitriou, Anthony Brockwell, and Christos Faloutsos. Adaptive, hands-off stream mining. In Proceedings 2003 VLDB Conference , pp. 560-571. Elsevier, 2003. 3
  34. Lalithsai Posam, Shubhranshu Shekhar, Meng-Chieh Lee, and Christos Faloutsos. Difffind: Discovering differential equations from time series. In Pacific-Asia Conference on Knowledge Discovery and Data Mining , pp. 175-187. Springer, 2024. 7, 17
  35. Hendrik Purwins, Bo Li, Tuomas Virtanen, Jan Schl¨ uter, Shuo-Yiin Chang, and Tara Sainath. Deep learning for audio signal processing. IEEE Journal of Selected Topics in Signal Processing , 13 (2):206-219, 2019. 3
  36. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020. 2, 6
  37. Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloˇ s, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, et al. Lag-llama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.08278 , 2023. 3, 7, 21 · arXiv
  38. David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting , 36(3):11811191, 2020. 7
  39. Lena Sasal, Tanujit Chakraborty, and Abdenour Hadid. W-transformers: a wavelet-based transformer framework for univariate time series forecasting. In 2022 21st IEEE international conference on machine learning and applications (ICMLA) , pp. 671-676. IEEE, 2022. 3
  40. Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. Advances in Neural Information Processing Systems , 36, 2024. 1
  41. Sabera Talukder, Yisong Yue, and Georgia Gkioxari. Totem: Tokenized time series embeddings for general time series analysis. arXiv preprint arXiv:2402.16412 , 2024. 3 · arXiv
  42. Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arXiv:2404.02905 , 2024. 3 · arXiv
  43. Arseny Tolmachev, Daisuke Kawahara, and Sadao Kurohashi. Juman++: A morphological analysis toolkit for scriptio continua. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pp. 54-59, 2018. 3
  44. Christopher Torrence and Gilbert P Compo. A practical guide to wavelet analysis. Bulletin of the American Meteorological society , 79(1):61-78, 1998. 5
  45. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023. 3 · arXiv
  46. Bryan E Usevitch. A tutorial on modern lossy wavelet image compression: foundations of jpeg 2000. IEEE signal processing magazine , 18(5):22-35, 2001. 9
  47. Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017. 3
  48. Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. arXiv preprint arXiv:2402.02592 , 2024. 3, 7, 21 · arXiv
  49. Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering , 2023. 3
  50. Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 30:495-507, 2021. 3
  51. Xiyuan Zhang, Xiaoyong Jin, Karthick Gopalswamy, Gaurav Gupta, Youngsuk Park, Xingjian Shi, Hao Wang, Danielle C Maddix, and Yuyang Wang. First de-trend then attend: Rethinking attention for time-series forecasting. arXiv preprint arXiv:2212.08151 , 2022. 3 · arXiv
  52. Xiyuan Zhang, Ranak Roy Chowdhury, Rajesh K Gupta, and Jingbo Shang. Large language models for time series: A survey. arXiv preprint arXiv:2402.01801 , 2024. 3 · arXiv
  53. Wenqing Zheng, SP Sharan, Ajay Kumar Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, and Zhangyang Wang. Outline, then details: Syntactically guided coarse-to-fine code generation. In International Conference on Machine Learning , pp. 42403-42419. PMLR, 2023. 3
  54. Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning , pp. 27268-27286. PMLR, 2022. 3
  55. Zhenhai Zhu and Radu Soricut. Wavelet-based image tokenizer for vision transformers. arXiv preprint arXiv:2405.18616 , 2024. 3 · arXiv

Cited by

Sorted by citations

Metadata

Authors
Masserano, L., Ansari, A. F., Han, B., Zhang, X., Faloutsos, C., Mahoney, M. W., Wilson, A. G., Park, Y., Rangapuram, S., Maddix, D. C., & Wang, Y.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2412.05244
arXiv
2412.05244
OpenAlex
W4405172852
Cited by
1
References
2
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
Keywords
time series forecasting, foundation models, wavelet coefficient quantization, zero-shot forecasting, non-stationary time series, temporal patterns

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review