CoChat
Sign Up
Cited by 1,260Time Series Analysis and ForecastingStock Market Forecasting MethodsNeural Networks and Applications

Transformers in Time Series: A Survey

Open access
Wen, Qingsong, Zhou, Tian, Zhang, Chaoli, Chen, Weiqi, Ma, Ziqing, Yan, Junchi, Sun, Liang·2023·Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence
DOI 10.24963/ijcai.2023/759
Open this paper in CoChat — free accountOpen PDF

Abstract

Transformers have achieved superior performances in many tasks in natural language processing and computer vision, which also triggered great interest in the time series community. Among multiple advantages of Transformers, the ability to capture long-range dependencies and interactions is especially attractive for time series modeling, leading to exciting progress in various time series applications. In this paper, we systematically review Transformer schemes for time series modeling by highlighting their strengths as well as limitations. In particular, we examine the development of time series Transformers in two perspectives. From the perspective of network structure, we summarize the adaptations and modifications that have been made to Transformers in order to accommodate the challenges in time series analysis. From the perspective of applications, we categorize time series Transformers based on common tasks including forecasting, anomaly detection, and classification. Empirically, we perform robust analysis, model size analysis, and seasonal-trend decomposition analysis to study how Transformers perform in time series. Finally, we discuss and suggest future directions to provide useful research guidance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.24963/ijcai.2023/759

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 62

  1. Ankur Bapna, Mia Xu Chen, Orhan Firat, Yuan Cao, and Yonghui Wu. Training deeper neural machine translation models with transparent attention. In EMNLP, 2018.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, et al. Lan- guage models are few-shot learners. NeurIPS, 2020.
  3. Ling Cai, Krzysztof Janowicz, Gengchen Mai, Bo Yan, and Rui Zhu. Traffic transformer: Capturing the conti- nuity and periodicity of time series for traffic forecasting. Trans- actions in GIS, 24(3):736-755, 2020.
  4. Chen et al., 2021a] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, et al. Pre-trained image processing transformer. In CVPR, 2021.
  5. Chen et al., 2021c] Zekai Chen, Dingshuo Chen, Xiao Zhang, Zix- uan Yuan, and Xiuzhen Cheng. Learning graph structures with transformer for multivariate time series anomaly detection in IoT. IEEE Internet of Things Journal, 2021.
  6. Weiqi Chen, Wenwei Wang, Bingqing Peng, Qingsong Wen, Tian Zhou, and Liang Sun. Learning to rotate: Quaternion transformer for complicated periodical time series forecasting. In KDD, 2022.
  7. Choi et al., 2021] Kukjin Choi, Jihun Yi, Changhwa Park, and Sungroh Yoon. Deep learning for anomaly detection in time- series data: Review, analysis, and guidelines. IEEE Access, 2021.
  8. Chowdhury et al., 2022] Ranak Roy Chowdhury, Xiyuan Zhang, Jingbo Shang, Rajesh K Gupta, and Dezhi Hong. TARNet: Task- aware reconstruction for time-series transformer. In KDD, 2022.
  9. Razvan-Gabriel Cirstea, Chenjuan Guo, Bin Yang, Tung Kieu, Xuanyi Dong, and Shirui Pan. Triformer: Tri- angular, variable-specific attentions for long sequence multivari- ate time series forecasting. In IJCAI, 2022.
  10. Robert Cleveland, William Cleveland, Jean McRae, et al. STL: A seasonal-trend decomposition procedure based on loess. Journal of Official Statistics, 6(1):3-73, 1990.
  11. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, et al. Transformer-XL: Attentive lan- guage models beyond a fixed-length context. In ACL, 2019.
  12. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal trans- formers. In ICLR, 2019.
  13. Elsken, Thomas, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. Journal of Machine Learning Research, 2019.
  14. Gao et al., 2022] Zhihan Gao, Xingjian Shi, Hao Wang, Yi Zhu, Bernie Wang, Mu Li, et al. Earthformer: Exploring space-time transformers for earth system forecasting. In NeurIPS, 2022.
  15. Jonas Gehring, Michael Auli, David Grang- ier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In ICML, 2017.
  16. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, et al. Generative adversarial nets. NeurIPS, 2014.
  17. Han et al., 2022] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, et al. A survey on vision transformer. IEEE TPAMI, 45(1):87-110, 2022.
  18. Rob J Hyndman and Yeasmin Khandakar. Automatic time series forecasting: the forecast pack- age for r. Journal of statistical software, 27:1-22, 2008.
  19. Guolin Ke, Di He, and Tie-Yan Liu. Rethinking positional encoding in language pre-training. In ICLR, 2021.
  20. Jacob Devlin Ming-Wei Chang Kenton et al. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
  21. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
  22. Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS, 2019.
  23. Longyuan Li, Jian Yao, Li Wenliang, Tong He, Tianjun Xiao, Junchi Yan, David Wipf, and Zheng Zhang. Grin: Generative relation and intention network for multi-agent trajec- tory prediction. In NeurIPS, 2021.
  24. Yuxuan Liang, Yutong Xia, Songyu Ke, Yiwei Wang, Qingsong Wen, Junbo Zhang, Yu Zheng, and Roger Zim- mermann. AirFormer: Predicting nationwide air quality in china with transformers. In AAAI, 2023.
  25. Bryan Lim and Stefan Zohren. Time- series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society, 2021.
  26. Yang Lin, Irena Koprinska, and Mashud Rana. SSDNet: State space decomposition neural network for time se- ries forecasting. In ICDM, 2021.
  27. Liu et al., 2022a] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X. Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In ICLR, 2022.
  28. Liu et al., 2022b] Yong Liu, Haixu Wu, Jianmin Wang, and Ming- sheng Long. Non-stationary transformers: Exploring the station- arity in time series forecasting. In NeurIPS, 2022.
  29. Sachin Mehta, Marjan Ghazvininejad, Srini Iyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Delight: Deep and light-weight transformer. In ICLR, 2021.
  30. Hongyuan Mei, Chenghao Yang, and Jason Eis- ner. Transformer embeddings of irregularly spaced events and their participants. In ICLR, 2022.
  31. Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long- term forecasting with transformers. In ICLR, 2023.
  32. Amin Shabani, Amir Abdi, Lili Meng, and Tristan Sylvain. Scaleformer: iterative multi-scale refining trans- formers for time series forecasting. In ICLR, 2023.
  33. Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In NAACL, 2018.
  34. Oleksandr Shchur, Ali Caner Türkmen, Tim Januschowski, and Stephan Günnemann. Neural temporal point processes: A review. In IJCAI, 2021.
  35. David So, Quoc Le, and Chen Liang. The evolved transformer. In ICML, 2019.
  36. Binh Tang and David Matteson. Proba- bilistic transformer for time series analysis. In NeurIPS, 2021.
  37. Yi Tay, Mostafa Dehghani, Dara Bahri, and Don- ald Metzler. Efficient transformers: A survey. ACM Computing Surveys, 55(6):1-28, 2022.
  38. Tuli et al., 2022] Shreshth Tuli, Giuliano Casale, and Nicholas R Jennings. TranAD: Deep transformer networks for anomaly de- tection in multivariate time series data. In VLDB, 2022.
  39. Ashish Vaswani, Noam Shazeer, Niki Par- mar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, et al. Attention is all you need. In NeurIPS, 2017.
  40. Xiaoxing Wang, Chao Xue, Junchi Yan, Xi- aokang Yang, Yonggang Hu, et al. MergeNAS: Merge operations into one for differentiable architecture search. In IJCAI, 2020.
  41. Qingsong Wen, Jingkun Gao, Xiaomin Song, Liang Sun, Huan Xu, et al. RobustSTL: A robust seasonal-trend decomposition algorithm for long time series. In AAAI, 2019.
  42. Qingsong Wen, Zhe Zhang, Yan Li, and Liang Sun. Fast RobustSTL: Efficient and robust seasonal-trend decom- position for time series with complex patterns. In KDD, 2020.
  43. Wen et al., 2021a] Qingsong Wen, Kai He, Liang Sun, Yingying Zhang, Min Ke, et al. RobustPeriod: Time-frequency mining for robust multiple periodicities detection. In SIGMOD, 2021.
  44. Wen et al., 2021b] Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey. In IJCAI, 2021.
  45. Qingsong Wen, Linxiao Yang, Tian Zhou, and Liang Sun. Robust time series analysis and applications: An in- dustrial perspective. In KDD, 2022.
  46. Wu et al., 2020a] Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse trans- former for time series forecasting. In NeurIPS, 2020.
  47. Wu et al., 2020b] Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. In ICLR, 2020.
  48. Wu et al., 2021] Haixu Wu, Jiehui Xu, Jianmin Wang, and Ming- sheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurIPS, 2021.
  49. Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy J. Lin. DeeBERT: Dynamic early exiting for acceler- ating bert inference. In ACL, 2020.
  50. Xu et al., 2022] Jiehui Xu, Haixu Wu, Jianmin Wang, and Ming- sheng Long. Anomaly Transformer: Time series anomaly detec- tion with association discrepancy. In ICLR, 2022.
  51. Yan et al., 2019] Junchi Yan, Hongteng Xu, and Liangda Li. Mod- eling and applications for temporal point processes. In KDD, 2019.
  52. Chao-Han Huck Yang, Yun-Yun Tsai, and Pin- Yu Chen. Voice2series: Reprogramming acoustic models for time series classification. In ICML, 2021.
  53. Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedes- trian trajectory prediction. In ECCV, 2020.
  54. Yuan Yuan and Lei Lin. Self-supervised pre- training of transformers for satellite image time series classifica- tion. IEEE J-STARS, 14:474-487, 2020.
  55. Yun et al., 2020] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, et al. Are transformers universal ap- proximators of sequence-to-sequence functions? In ICLR, 2020.
  56. Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI, 2023.
  57. George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series repre- sentation learning. In KDD, 2021.
  58. Yunhao Zhang and Junchi Yan. Cross- former: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In ICLR, 2023.
  59. Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive Hawkes process. In ICML, 2020.
  60. Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. In- former: Beyond efficient transformer for long sequence time- series forecasting. In AAAI, 2021.
  61. Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency en- hanced decomposed transformer for long-term series forecasting. In ICML, 2022.
  62. Zuo et al., 2020] Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer Hawkes process. In ICML, 2020.

Cited by

Sorted by citations

Authors
Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., & Sun, L.
Affiliations
  • Alibaba Group (United States) (US)
  • Alibaba DAMO Academy (China) (CN)
  • Alibaba Group (Cayman Islands) (KY)
  • Shanghai Jiao Tong University (CN)
  • Alibaba Group (China) (CN)
ORCID
Venue
Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence
Publisher
International Joint Conferences on Artificial Intelligence Organization
Published
2023-08
Pages
6778-6786
DOI
10.24963/ijcai.2023/759
OpenAlex
W4385763767
Cited by
1,260
References
8
Citations by year
2026: 297 · 2025: 431 · 2024: 337 · 2023: 171 · 2022: 23
FWCI
256.69
Type
conference paper
Language
en
Open access
Locations
  • International Joint Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
transformer models, time series modeling, long-range dependencies, time series forecasting, anomaly detection, time series classification, seasonal-trend decomposition, time series transformer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review