The DOI matches its Crossref record; 83 of the 83 references it cites were found in a registry or an index.
Evaluation of Deep Learning Models for Multi-Step Ahead Time Series Prediction
Open accessAbstract
Time series prediction with neural networks has been the focus of much research in the past few decades. Given the recent deep learning revolution, there has been much attention in using deep learning models for time series prediction, and hence it is important to evaluate their strengths and weaknesses. In this paper, we present an evaluation study that compares the performance of deep learning models for multi-step ahead time series prediction. The deep learning methods comprise simple recurrent neural networks, long short-term memory (LSTM) networks, bidirectional LSTM networks, encoder-decoder LSTM networks, and convolutional neural networks. We provide a further comparison with simple neural networks that use stochastic gradient descent and adaptive moment estimation (Adam) for training. We focus on univariate time series for multi-step-ahead prediction from benchmark time-series datasets and provide a further comparison of the results with related methods from the literature. The results show that the bidirectional and encoder-decoder LSTM network provides the best performance in accuracy for the given time series problems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 1×
“CNNs introduced by LeCun [91], [92] are prominent deep learning architecture inspired by the natural visual system of mammals.”
Background(2014) · Cited by 33,419
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Deterministic Nonperiodic Flow
Lorenz, E. N. (1963) · Journal of the Atmospheric Sciences · doi:10.1175/1520-0469(1963)020<0130:dnf>2.0.co;2 · Cited by 19,398
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Detecting strange attractors in turbulence
Takens, F. (1981) · Lecture Notes in Mathematics · doi:10.1007/bfb0091924 · Cited by 10,459 · cited 3×
“Taken’s embedding theorem expresses that the reconstruction can reproduce important features of the original time series [83].”
BackgroundMethodInfluentialBidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · cited 2×
“Bidirectional RNNs connect two hidden layers of opposite directions to the same output, where the output layer can get information from past and future states simultaneously [35].”
BackgroundPlaying Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
(2010) · Cited by 8,261
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,839
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
Oscillation and Chaos in Physiological Control Systems
Mackey, M. C., & Glass, L. (1977) · Science · doi:10.1126/science.267326 · Cited by 4,135
An equation for continuous chaos
Rössler, O. E. (1976) · Physics Letters A · doi:10.1016/0375-9601(76)90101-8 · Cited by 4,012
(1989) · neural information processing systems · Cited by 3,736
A two-dimensional mapping with a strange attractor
Hénon, M. (1976) · Communications in Mathematical Physics · doi:10.1007/bf01608556 · Cited by 2,987
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
Hochreiter, S. (1998) · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems · doi:10.1142/s0218488598000094 · Cited by 2,939
DeepAR: Probabilistic forecasting with autoregressive recurrent networks
Salinas, D., Flunkert, V., Gasthaus, J., & Januschowski, T. (2020) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.07.001 · Cited by 2,661
Theory of the backpropagation neural network
Hecht-Nielsen (1989) · International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1989.118638 · Cited by 2,222
Named Entity Recognition with Bidirectional LSTM-CNNs
Chiu, J. P. C., & Nichols, E. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00104 · Cited by 1,909
Hybrid speech recognition with Deep Bidirectional LSTM
Graves, A., Jaitly, N., & Mohamed, A.-R. (2013) · 2013 IEEE Workshop on Automatic Speech Recognition and Understanding · doi:10.1109/asru.2013.6707742 · Cited by 1,736
25 years of time series forecasting
De Gooijer, J. G., & Hyndman, R. J. (2006) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2006.01.001 · Cited by 1,587
Recurrent neural networks and robust time series prediction
Connor, J. T., Martin, R. D., & Atlas, L. E. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279188 · Cited by 1,316
Time series forecasting of COVID-19 transmission in Canada using LSTM networks
Chimmula, V. K. R., & Zhang, L. (2020) · Chaos, Solitons & Fractals · doi:10.1016/j.chaos.2020.109864 · Cited by 1,075
Stock market's price movement prediction with LSTM neural networks
Nelson, D. M. Q., Pereira, A. C. M., & de Oliveira, R. A. (2017) · 2017 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2017.7966019 · Cited by 951 · cited 1×
“[81] used LSTM networks to forecast stock market trends.”
MethodAn Empirical Comparison of Machine Learning Models for Time Series Forecasting
Ahmed, N. K., Atiya, A. F., Gayar, N. E., & El-Shishiny, H. (2010) · Econometric Reviews · doi:10.1080/07474938.2010.481556 · Cited by 789
A comparison of direct and iterated multistep AR methods for forecasting macroeconomic time series
Marcellino, M., Stock, J. H., & Watson, M. W. (2006) · Journal of Econometrics · doi:10.1016/j.jeconom.2005.07.020 · Cited by 789
Deep learning based ensemble approach for probabilistic wind power forecasting
Wang, H.-Z., Li, G.-Q., Wang, G.-B., Peng, J.-C., Jiang, H., & Liu, Y.-T. (2017) · Applied Energy · doi:10.1016/j.apenergy.2016.11.111 · Cited by 753
Ben Taieb, S., Bontempi, G., Atiya, A. F., & Sorjamaa, A. (2012) · Expert Systems with Applications · doi:10.1016/j.eswa.2012.01.039 · Cited by 734
A Deep CNN-LSTM Model for Particulate Matter (PM2.5) Forecasting in Smart Cities
Huang, C.-J., & Kuo, P.-H. (2018) · Sensors · doi:10.3390/s18072220 · Cited by 705
Deep learning for event-driven stock prediction
Ding, X., Zhang, Y., Liu, T., & Duan, J. (2015) · International Conference on Artificial Intelligence · Cited by 685
Recent Developments in Econometric Modeling and Forecasting
Li, G., Song, H., & Witt, S. F. (2005) · Journal of Travel Research · doi:10.1177/0047287505276594 · Cited by 523
Time series forecasting using artificial neural networks methodologies: A systematic review
Tealab, A. (2018) · Future Computing and Informatics Journal · doi:10.1016/j.fcij.2018.10.003 · Cited by 489
Methodology for long-term prediction of time series
Sorjamaa, A., Hao, J., Reyhani, N., Ji, Y., & Lendasse, A. (2007) · Neurocomputing · doi:10.1016/j.neucom.2006.06.015 · Cited by 438
TTS synthesis with bidirectional LSTM based recurrent neural networks
Fan, Y., Qian, Y., Xie, F.-L., & Soong, F. K. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-443 · Cited by 426
Girard, A., Rasmussen, C. E., Candela, J. Q., & Murray‐Smith, R. (2002) · Cited by 384
Recurrent neural networks for time series classification
Hüsken, M., & Stagge, P. (2003) · Neurocomputing · doi:10.1016/s0925-2312(01)00706-8 · Cited by 352
Deep neural networks for energy load forecasting
Amarasinghe, K., Marino, D. L., & Manic, M. (2017) · 2017 IEEE 26th International Symposium on Industrial Electronics (ISIE) · doi:10.1109/isie.2017.8001465 · Cited by 336
Time Series Prediction and Neural Networks
Frank, R. J., Davey, N., & Hunt, S. P. (2001) · Journal of Intelligent and Robotic Systems · doi:10.1023/a:1012074215150 · Cited by 323
Time series forecasting for nonlinear and non-stationary processes: a review and comparative study
Cheng, C., Sa-Ngasoongsong, A., Beyca, O., Le, T., Yang, H., Kong, Z. (., & Bukkapatnam, S. T. S. (2015) · IIE Transactions · doi:10.1080/0740817x.2014.999180 · Cited by 287
Learning the hidden structure of speech
Elman, J. L., & Zipser, D. (1988) · The Journal of the Acoustical Society of America · doi:10.1121/1.395916 · Cited by 268
Multiple-output modeling for multi-step-ahead time series forecasting
Ben Taieb, S., Sorjamaa, A., & Bontempi, G. (2010) · Neurocomputing · doi:10.1016/j.neucom.2009.11.030 · Cited by 251
Chang, F.-J., Chen, P.-A., Lu, Y.-R., Huang, E., & Chang, K.-Y. (2014) · Journal of Hydrology · doi:10.1016/j.jhydrol.2014.06.013 · Cited by 250
Cooperative coevolution of Elman recurrent neural networks for chaotic time series prediction
Chandra, R., & Zhang, M. (2012) · Neurocomputing · doi:10.1016/j.neucom.2012.01.014 · Cited by 243
Self-citationMulti-step-ahead time series prediction using multiple-output support vector regression
Bao, Y., Xiong, T., & Hu, Z. (2014) · Neurocomputing · doi:10.1016/j.neucom.2013.09.010 · Cited by 239
Predictive State Representations: A New Theory for Modeling Dynamical Systems
Singh, S. P., James, M., & Rudary, M. R. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4167 · Cited by 201
A Bias and Variance Analysis for Multistep-Ahead Time Series Forecasting
Taieb, S. B., & Atiya, A. F. (2016) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2411629 · Cited by 186
Time series prediction with recurrent neural networks trained by a hybrid PSO–EA algorithm
Cai, X., Zhang, N., Venayagamoorthy, G. K., & Wunsch, D. C. (2007) · Neurocomputing · doi:10.1016/j.neucom.2005.12.138 · Cited by 170
Chandra, R. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2404823 · Cited by 162
Self-citationExplore an evolutionary recurrent ANFIS for modelling multi-step-ahead flood forecasts
Zhou, Y., Guo, S., & Chang, F. (2019) · Journal of Hydrology · doi:10.1016/j.jhydrol.2018.12.040 · Cited by 154
Wang, J., & Li, Y. (2018) · Applied Energy · doi:10.1016/j.apenergy.2018.08.114 · Cited by 143
Recursive Bayesian Recurrent Neural Networks for Time-Series Modeling
Mirikitani, D. T., & Nikolaev, N. (2010) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2009.2036174 · Cited by 137
Reinforced recurrent neural networks for multi-step-ahead flood forecasts
Chen, P.-A., Chang, L., & Chang, F. (2013) · Journal of Hydrology · doi:10.1016/j.jhydrol.2013.05.038 · Cited by 133
Time Series Prediction with Multilayer Perception, FIR and Elman Neural Networks
Koskela, T., Lehtokangas, M. I., Saarinen, J. P. P., & Kaski, K. K. (1996) · Cited by 121
Solar cycle forecasting: A nonlinear dynamics approach
Sello, S. (2001) · Astronomy and Astrophysics · doi:10.1051/0004-6361:20011068 · Cited by 114
Giorgi, M. G. D., Malvoni, M., & Congedo, P. M. (2016) · Energy · doi:10.1016/j.energy.2016.04.020 · Cited by 114
Long-term time series prediction using OP-ELM
Grigorievskiy, A., Miché, Y., Ventelä, A., Séverin, É., & Lendasse, A. (2013) · Neural Networks · doi:10.1016/j.neunet.2013.12.002 · Cited by 109
Chaos Theory and Transportation Systems: Instructive Example
Frazier, C. F., & Kockelman, K. M. (2004) · Transportation Research Record Journal of the Transportation Research Board · doi:10.3141/1897-02 · Cited by 91
Chandra, R., Ong, Y.-S., & Goh, C.-K. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.02.065 · Cited by 84
Self-citationNiu, G., & Yang, B. (2008) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2008.08.004 · Cited by 76
Giles, C. L., Omlin, C. W. P., & Thornber, K. K. (1999) · Proceedings of the IEEE · doi:10.1109/5.784244 · Cited by 75
Sudriani, Y., Ridwansyah, I., & Rustini, H. A. (2019) · IOP Conference Series Earth and Environmental Science · doi:10.1088/1755-1315/299/1/012037 · Cited by 75
Chandra, R., & Chand, S. (2016) · Applied Soft Computing · doi:10.1016/j.asoc.2016.08.029 · Cited by 62
Self-citationChang, L., Chen, P.-A., & Chang, F. (2012) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2200695 · Cited by 58
Co-evolutionary multi-task learning for dynamic time series prediction
Chandra, R., Ong, Y.-S., & Goh, C.-K. (2018) · Applied Soft Computing · doi:10.1016/j.asoc.2018.05.041 · Cited by 54
Self-citationLangevin-gradient parallel tempering for Bayesian neural learning
Chandra, R., Jain, K., Deo, R., & Cripps, S. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.05.082 · Cited by 53
Self-citationEvolving Deep LSTM-based Memory Networks using an Information Maximization Objective
Rawal, A., & Miikkulainen, R. P. (2016) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/2908812.2908941 · Cited by 48
Multi-Step Ahead Predictions for Critical Levels in Physiological Time Series
ElMoaqet, H., Tilbury, D. M., & Ramachandran, S. K. (2016) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2561974 · Cited by 37
Predictive State Recurrent Neural NetworksPreprint
Downey, C., Hefny, A., Li, B., Boots, B., & Gordon, G. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.09353 · Cited by 37
MultiTL-KELM: A multi-task learning algorithm for multi-step-ahead time series prediction
Ye, R., & Dai, Q. (2019) · Applied Soft Computing · doi:10.1016/j.asoc.2019.03.039 · Cited by 32
A pattern fusion model for multi-step-ahead CPU load prediction
Yang, D., Cao, J., Fu, J., Wang, J., & Guo, J. (2012) · Journal of Systems and Software · doi:10.1016/j.jss.2012.12.023 · Cited by 22
Direct and iterated multistep AR methods for difference stationary processes
Proietti, T. (2010) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2010.05.014 · Cited by 17
Multistep forecasting in the presence of location shifts
Chevillon, G. (2015) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2015.04.004 · Cited by 12
Coevolutionary multi-task learning for feature-based modular pattern classification
Chandra, R., & Cripps, S. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.08.011 · Cited by 10
Self-citation
Cited by
Time Series Prediction Based on LSTM-Attention-LSTM Model
Wen, X., & Li, W. (2023) · IEEE Access · doi:10.1109/access.2023.3276628 · Cited by 335
Metadata
- Authors
- Chandra, R., Goyal, S., & Gupta, R.
- Venue
- IEEE Access
- Published
- 2021
- Volume
- 9
- Pages
- 83105-83123
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3139885963
- Cited by
- 265
- References
- 83
- FWCI
- 17.44
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.