Deep Direct Reinforcement Learning for Financial Signal Representation and Trading
The DOI matches its Crossref record; 50 of the 51 references it cites were found in a registry or an index.
Abstract
Can we train the computer to beat experienced traders for financial assert trading? In this paper, we try to address this challenge by introducing a recurrent deep neural network (NN) for real-time financial signal representation and trading. Our model is inspired by two biological-related learning concepts of deep learning (DL) and reinforcement learning (RL). In the framework, the DL part automatically senses the dynamic market condition for informative feature learning. Then, the RL module interacts with deep representations and makes trading decisions to accumulate the ultimate rewards in an unknown environment. The learning system is implemented in a complex NN that exhibits both the deep and recurrent structures. Hence, we propose a task-aware backpropagation through time method to cope with the gradient vanishing issue in deep training. The robustness of the neural system is verified on both the stock and the commodity future markets under broad testing conditions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In machinelearning systems, humans shape and reshape algorithms through their daily actions and interactions (Deng, Bao, Kong, Ren, & Dai, 2017).”
From Artificial Intelligence and Management: The Automation–Augmentation Paradox (2021) · cites this paper as Deng, Bao, Kong, Ren, & Dai, 2017
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
(1999) · Cited by 12,812
(1995) · Cited by 10,558
A neural substrate of prediction and reward
Schultz, W., Dayan, P., & Montague, P. R. (1997) · Science · doi:10.1126/science.275.5306.1593 · Cited by 9,388
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
Markov decision processes: Discrete stochastic dynamic programming
Hazeghi, K., & Puterman, M. L. (1995) · Journal of the American Statistical Association · doi:10.2307/2291177 · Cited by 8,037
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
Learning to forget: Continual prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 4,823
(2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953039 · Cited by 4,051
Fuzzy sets, uncertainty, and information
(1989) · European Journal of Operational Research · doi:10.1016/0377-2217(89)90454-2 · Cited by 3,175
Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011 · cited 1×
“While admitting the general effectiveness of sparse coding for feature learning [23]–[25], [36], it is essentially a shallow data representation strategy whose performance is not comparable with the state-of-the-art DL in a wide range tests [26], [27].”
BackgroundLinear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206757 · Cited by 2,828
Sharpe, W. F. (1994) · The Journal of Portfolio Management · doi:10.3905/jpm.1994.409501 · Cited by 2,534
Markov Decision Processes: Discrete Stochastic Dynamic Programming
Baxter, L. A. (1995) · Technometrics · doi:10.1080/00401706.1995.10484354 · Cited by 2,415
Lee, H., Grosse, R., Ranganath, R., & Ng, A. Y. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553453 · Cited by 2,412
Mean Reversion in Stock Prices: Evidence and Implications
Poterba, J., & Summers, L. (1987) · National Bureau of Economic Research · doi:10.3386/w2343 · Cited by 1,875
Mean reversion in stock prices
Poterba, J. M., & Summers, L. H. (1988) · Journal of Financial Economics · doi:10.1016/0304-405x(88)90021-9 · Cited by 1,858
Neural-network-based fuzzy logic control and decision system
Lin, C.-T., & Lee, C. S. G. (1991) · IEEE Transactions on Computers · doi:10.1109/12.106218 · Cited by 1,480
Mean Reversion in Stock Prices: Archival Version
Poterba, J., & Summers, L. (1996) · ICPSR Data Holdings · doi:10.3886/icpsr01029 · Cited by 1,083
Forecasting the equity risk premium: the role of technical indicators
Neely, C. J., Rapach, D. E., Tu, J., & Zhou, G. (2014) · Management Science · doi:10.1287/mnsc.2013.1838 · Cited by 1,003
Robust performance hypothesis testing with the Sharpe ratio
Ledoit, O., & Wolf, M. (2008) · Journal of Empirical Finance · doi:10.1016/j.jempfin.2008.03.002 · Cited by 948 · cited 1×
“, SR, which has been widely used in many trading related works [42], [43].”
BackgroundTD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play
Tesauro, G. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.215 · Cited by 787
(1999) · Choice Reviews Online · doi:10.5860/choice.36-4016 · Cited by 771
Saad, E. W., Prokhorov, D. V., & Wunsch, D. C. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/72.728395 · Cited by 537
Learning to trade via direct reinforcement
Moody, J., & Saffell, M. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.935097 · Cited by 535
Linear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvprw.2009.5206757 · Cited by 496
How effective are neural networks at forecasting and prediction? A review and evaluation
Adya, M., & Collopy, F. (1998) · Journal of Forecasting · doi:10.1002/(sici)1099-131x(1998090)17:5/6<481::aid-for709>3.0.co;2-q · Cited by 451
Pal, N. R., & Bezdek, J. C. (1994) · IEEE Transactions on Fuzzy Systems · doi:10.1109/91.277960 · Cited by 326
A sensor-based navigation for a mobile robot using fuzzy logic and reinforcement learning
Beom, H. R., & Cho, H. S. (1995) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.364859 · Cited by 291
Performance functions and reinforcement learning for trading systems and portfolios
Moody, J., Wu, L., Liao, Y., & Saffell, M. (1998) · Journal of Forecasting · doi:10.1002/(sici)1099-131x(1998090)17:5/6<441::aid-for707>3.0.co;2-# · Cited by 291
Autonomous Helicopter Flight via Reinforcement Learning
Kim, H. J., Jordan, M. I., Sastry, S., & Ng, A. Y. (2003) · neural information processing systems · Cited by 287
Stock price pattern recognition-a recurrent neural network approach
Kamijo, K., & Tanigawa, T. (1990) · 1990 IJCNN International Joint Conference on Neural Networks · doi:10.1109/ijcnn.1990.137572 · Cited by 252
Self-constructing fuzzy neural network speed controller for permanent-magnet synchronous motor drive
Lin, F.-J., Lin, C.-H., & Shen, P.-H. (2001) · IEEE Transactions on Fuzzy Systems · doi:10.1109/91.963761 · Cited by 250
Support-vector-based fuzzy neural network for pattern classification
Lin, C.-T., Yeh, C.-M., Liang, S.-F., Chung, J.-F., & Kumar, N. (2006) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2005.861604 · Cited by 211
An automated FX trading system using adaptive reinforcement learning
Dempster, M. A. H., & Leemans, V. (2006) · Expert Systems with Applications · doi:10.1016/j.eswa.2005.10.012 · Cited by 181
Learning-Induced Plasticity in Deep Cerebellar Nucleus
Ohyama, T., Nores, W. L., Medina, J. F., Riusech, F., & Mauk, M. D. (2006) · Journal of Neuroscience · doi:10.1523/jneurosci.4023-06.2006 · Cited by 143
Low-Rank Structure Learning via Nonconvex Heuristic Recovery
Deng, Y. C., Dai, Q., Liu, R., Zhang, Z., & Hu, S. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2012.2235082 · Cited by 127
Self-citationStock Trading Using RSPOP: A Novel Rough Set-Based Neuro-Fuzzy Approach
Ang, K. K., & Quek, C. H. (2006) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2006.875996 · Cited by 125
Song, Y., Song, Q., & Cai, W. (2014) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2013.2290310 · Cited by 109
Graph Laplace for Occluded Face Completion and Recognition
Deng, Y. C., Dai, Q., & Zhang, Z. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2109729 · Cited by 84
Self-citationMEC—A Near-Optimal Online Reinforcement Learning Algorithm for Continuous Deterministic Systems
Zhao, D., & Zhu, Y. (2014) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2371046 · Cited by 79
Sparse Coding-Inspired Optimal Trading System for HFT Industry
Deng, Y. C., Kong, Y., Bao, F., & Dai, Q. (2015) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2015.2404299 · Cited by 50
Self-citationNoisy Depth Maps Fusion for Multiview Stereo Via Matrix Completion
Deng, Y. C., Liu, Y., Dai, Q., Zhang, Z., & Wang, Y. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2195472 · Cited by 43
Self-citationVisual Words Assignment Via Information-Theoretic Manifold Embedding
Deng, Y., Li, Y., Qian, Y., Ji, X., & Dai, Q. (2014) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2014.2300192 · Cited by 30
Self-citation
Research works printed but not linked: 1
- prokhorov, 2001, Dynamical neural networks for control, A Field Guide to Dynamical Recurrent Networks
Cited by
State-of-the-art in artificial neural network applications: A survey
Abiodun, O. I., Jantan, A., Omolara, A. E., Dada, K. V., Mohamed, N. A., & Arshad, H. (2018) · Heliyon · doi:10.1016/j.heliyon.2018.e00938 · Cited by 3,054
“The comparison was made based on (i) author(s)/year of publication (ii) ANN modeling (iii) ANN area of application (iii) studied contribution to human challenges (with references to Supplementary Table 2) [11, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140,…”
MethodArtificial Intelligence and Management: The Automation–Augmentation Paradox
Raisch, S., & Krakowski, S. (2021) · Academy of Management Review · doi:10.5465/amr.2018.0072 · Cited by 2,755
Villarraga Ossa, E. F. (2021) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 1,692
Rahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364
“More sophisticated agents may compute strategies based on adaptive heuristics or explicit maximization of expected utility 86 .”
MethodTransfer Learning in Deep Reinforcement Learning: A Survey
Zhu, Z., Lin, K., Jain, A. K., & Zhou, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3292075 · Cited by 861
Metadata
- Authors
- Deng, Y., Bao, F., Kong, Y., Ren, Z., & Dai, Q.
- Affiliations
- University of California, San Francisco (US)
- Tsinghua University (CN)
- Southeast University (CN)
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2017-03
- Volume
- 28
- Issue
- 3
- Pages
- 653-664
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W2344786740
- Cited by
- 822
- References
- 51
- Citations by year
- 2026: 90 · 2025: 90 · 2024: 95 · 2023: 110 · 2022: 109 · 2021: 100 · 2020: 98 · 2019: 76 · 2018: 40 · 2017: 10
- FWCI
- 47.14
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Project of the National Natural Science Foundation of China (61327902, 61120106003)
- National Science Foundation of Jiangsu Province, China (BK20150650)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Stock Market Forecasting Methods
- Social Sciences › Economics, Econometrics and Finance › Finance › Financial Markets and Investment Strategies
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- deep reinforcement learning, financial trading, recurrent neural networks, backpropagation through time, gradient vanishing, stock market, commodity futures, trading decision-making, representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.