The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Learning to trade via direct reinforcement
Abstract
We present methods for optimizing portfolios, asset allocations, and trading systems based on direct reinforcement (DR). In this approach, investment decision-making is viewed as a stochastic control problem, and strategies are discovered directly. We present an adaptive algorithm called recurrent reinforcement learning (RRL) for discovering investment policies. The need to build forecasting models is eliminated, and better trading performance is obtained. The direct reinforcement approach differs from dynamic programming and reinforcement algorithms such as TD-learning and Q-learning, which attempt to estimate a value function for the control problem. We find that the RRL direct reinforcement framework enables a simpler problem representation, avoids Bellman's curse of dimensionality and offers compelling advantages in efficiency. We demonstrate how direct reinforcement can be used to optimize risk-adjusted investment returns (including the differential Sharpe ratio), while accounting for the effects of transaction costs. In extensive simulation work using real financial data, we find that our approach based on RRL produces better trading strategies than systems utilizing Q-learning (a value function method). Real-world applications include an intra-daily currency trader and a monthly asset allocation system for the S&P 500 Stock Index and T-Bills.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,172
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Identification and control of dynamical systems using neural networks
Narendra, K. S., & Parthasarathy, K. (1990) · IEEE Transactions on Neural Networks · doi:10.1109/72.80202 · Cited by 8,014
Option pricing: A simplified approach
Cox, J. C., Ross, S. A., & Rubinstein, M. (1979) · Journal of Financial Economics · doi:10.1016/0304-405x(79)90015-1 · Cited by 6,083
Optimum consumption and portfolio rules in a continuous-time model
Merton, R. C. (1971) · Journal of Economic Theory · doi:10.1016/0022-0531(71)90038-x · Cited by 5,958
Portfolio Selection: Efficient Diversification of Investments
Stuart, A., & Markowitz, H. M. (1959) · OR · doi:10.2307/3006625 · Cited by 5,534
Watkins, C. (1989) · OpenGrey (Institut de l'Information Scientifique et Technique) · doi:10.17863/cam.132164 · Cited by 5,297
Lifetime Portfolio Selection under Uncertainty: The Continuous-Time Case
Merton, R. C. (1969) · The Review of Economics and Statistics · doi:10.2307/1926560 · Cited by 5,187
Learning to predict by the methods of temporal differences
Sutton, R. S. (1988) · Machine Learning · doi:10.1007/bf00115009 · Cited by 5,148
(1999) · Cited by 5,147
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
Some Studies in Machine Learning Using the Game of Checkers
Samuel, A. L. (1959) · IBM Journal of Research and Development · doi:10.1147/rd.33.0210 · Cited by 4,443
A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
Williams, R. J., & Zipser, D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.2.270 · Cited by 4,369
WIDROW, B., & HOFF, M. E. (1960) · doi:10.21236/ad0241531 · Cited by 4,009
Sharpe, W. F. (1966) · The Journal of Business · doi:10.1086/294846 · Cited by 3,937
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1023/a:1022676722315 · Cited by 3,907
Portfolio Selection: Efficient Diversification of Investments.
Houthakker, H. S., & Markowitz, H. M. (1962) · Journal of the American Statistical Association · doi:10.2307/2282415 · Cited by 3,621
(2012) · Encyclopedia of the Sciences of Learning · doi:10.1007/978-1-4419-1428-6_5086 · Cited by 3,515
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 3,416
Neuronlike adaptive elements that can solve difficult learning control problems
Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1983.6313077 · Cited by 3,233
Valuing American Options by Simulation: A Simple Least-Squares Approach
Longstaff, F. A., & Schwartz, E. S. (2001) · Review of Financial Studies · doi:10.1093/rfs/14.1.113 · Cited by 3,216
Ingersoll, J. E., & Duffie, D. (1993) · The Journal of Finance · doi:10.2307/2329081 · Cited by 2,891
Konda, V. R., & Tsitsiklis, J. N. (2002) · Cited by 2,202
Ross, S. A., & Merton, R. C. (1991) · The Journal of Finance · doi:10.2307/2328875 · Cited by 1,788
Temporal difference learning and TD-Gammon
Tesauro, G. (1995) · Communications of the ACM · doi:10.1145/203330.203343 · Cited by 1,618
Some studies in machine learning using the game of checkers
Samuel, A. L. (2000) · IBM Journal of Research and Development · doi:10.1147/rd.441.0206 · Cited by 1,439
Approximate dynamic programming for real-time control and neural modeling
Werbos, P. J. (1992) · Medical Entomology and Zoology · Cited by 971
(2011) · Encyclopedia of Machine Learning · doi:10.1007/978-0-387-30164-8_588 · Cited by 950
Prioritized sweeping: Reinforcement learning with less data and less time
Moore, A. W., & Atkeson, C. G. (1993) · Machine Learning · doi:10.1007/bf00993104 · Cited by 823
Simple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J. (1992) · Machine Learning · doi:10.1023/a:1022672621406 · Cited by 817
TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play
Tesauro, G. (1994) · Neural Computation · doi:10.1162/neco.1994.6.2.215 · Cited by 787
Temporal Difference Learning and TD-Gammon
Tesauro, G. (1995) · ICGA Journal · doi:10.3233/icg-1995-18207 · Cited by 764
Brennan, M. J., Schwartz, E. S., & Lagnado, R. (1997) · Journal of Economic Dynamics and Control · doi:10.1016/s0165-1889(97)00031-6 · Cited by 762
Temporal credit assignment in reinforcement learning
Sutton, R. S. (1984) · ScholarWorks@UMassAmherst (University of Massachusetts Amherst) · Cited by 749
Some Studies in Machine Learning Using the Game of Checkers. II—Recent Progress
Samuel, A. L. (1967) · IBM Journal of Research and Development · doi:10.1147/rd.116.0601 · Cited by 695
Regression methods for pricing complex American-style options
Tsitsiklis, J. N., & Van Roy, B. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.935083 · Cited by 612
Sortino, F. A., & van der Meer, R. (1991) · The Journal of Portfolio Management · doi:10.3905/jpm.1991.409343 · Cited by 605
Portfolio Selection: Efficient Diversification of Investments.
Tintner, G., & Markowitz, H. M. (1960) · The Journal of Finance · doi:10.2307/2326204 · Cited by 582
Improving Elevator Performance Using Reinforcement Learning
Crites, R. H., & Barto, A. G. (1995) · Cited by 499
Tsitsiklis, J. N., & van Roy, B. (1999) · IEEE Transactions on Automatic Control · doi:10.1109/9.793723 · Cited by 326
Simulation-based optimization of Markov reward processes
Marbach, P., & Tsitsiklis, J. N. (2001) · IEEE Transactions on Automatic Control · doi:10.1109/9.905687 · Cited by 306
Performance functions and reinforcement learning for trading systems and portfolios
Moody, J., Wu, L., Liao, Y., & Saffell, M. (1998) · Journal of Forecasting · doi:10.1002/(sici)1099-131x(1998090)17:5/6<441::aid-for707>3.0.co;2-# · Cited by 291
Self-citationSimulation of self-organizing systems by digital computer
Farley, B., & Clark, W. (1954) · Transactions of the IRE Professional Group on Information Theory · doi:10.1109/tit.1954.1057468 · Cited by 270
Risk-Sensitive Reinforcement Learning
Mihatsch, O., & Neuneier, R. (2002) · Machine Learning · doi:10.1023/a:1017940631555 · Cited by 258
Actor-Critic--Type Learning Algorithms for Markov Decision Processes
Konda, V. R., & Borkar, V. S. (1999) · SIAM Journal on Control and Optimization · doi:10.1137/s036301299731669x · Cited by 234
Gradient Descent for General Reinforcement Learning
Baird, L. C., & Moore, A. (1998) · Cited by 232
Risk-Sensitive Reinforcement Learning
Shen, Y., Tobia, M. J., Sommer, T., & Obermayer, K. (2014) · Neural Computation · doi:10.1162/neco_a_00600 · Cited by 211
Efficient Learning and Planning Within the Dyna Framework
Peng, J., & Williams, R. J. (1993) · Adaptive Behavior · doi:10.1177/105971239300100403 · Cited by 199
Duffie, D. (1996) · Cited by 141
On the Use and Misuse of Downside Risk
Sortino, F. A., & Forsey, H. (1996) · The Journal of Portfolio Management · doi:10.3905/jpm.1996.35 · Cited by 134
High-Performance Job-Shop Scheduling With A Time-Delay TD(λ) Network
Zhang, W., & Dietterich, T. G. (1995) · neural information processing systems · Cited by 97
Direct gradient-based reinforcement learning
Baxter, J., & Bartlett, P. L. (2002) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2000.856049 · Cited by 94
Optimal Asset Allocation using Adaptive Dynamic Programming
Neuneier, R. (1995) · Cited by 76
Optimization of trading systems and portfolios
Moody, J. E., & Wu, L. (2002) · IEEE Conference on Computational Intelligence for Financial Engineering & Economics · doi:10.1109/cifer.1997.618952 · Cited by 69
Self-citationBaird, I., & Leemon, C. (1993) · Defense Technical Information Center (DTIC) · doi:10.21236/ada280862 · Cited by 61
Reinforcement learning for trading
Moody, J. E., & Saffell, M. (1999) · Cited by 54
Self-citationRisk-sensitive reinforcement learning: a martingale approach to reward uncertainty
Vadori, N., Ganesh, S., Reddy, P., & Veloso, M. (2020) · Proceedings of the First ACM International Conference on AI in Finance · doi:10.1145/3383455.3422519 · Cited by 25
Intertemporal Portfolio Theory and Asset Pricing
(1989) · doi:10.1007/978-1-349-20213-3_20 · Cited by 8
Cited by
Deep Direct Reinforcement Learning for Financial Signal Representation and Trading
Deng, Y., Bao, F., Kong, Y., Ren, Z., & Dai, Q. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2522401 · Cited by 822
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Reinforcement learning algorithms: A brief survey
Shakya, A. K., Pillai, G., & Chakrabarty, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120495 · Cited by 656
A Hybrid Reinforcement Learning Approach to Autonomic Resource Allocation
Tesauro, G., Jong, N. K., Das, R., & Bennani, M. N. (2006) · 2006 IEEE International Conference on Autonomic Computing · doi:10.1109/icac.2006.1662383 · Cited by 365
A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem
Jiang, Z., Xu, D., & Liang, J. (2017) · arXiv · doi:10.48550/arxiv.1706.10059 · Cited by 321
Metadata
- Authors
- Moody, J., & Saffell, M.
- Venue
- IEEE Transactions on Neural Networks
- Published
- 2001-07
- Volume
- 12
- Issue
- 4
- Pages
- 875-889
- ISSN
- 1045-9227 (ISSN-L)
- OpenAlex
- W2169015875
- Cited by
- 535
- References
- 62
- FWCI
- 5.54
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.